---
title: "频谱分配：为何Muon优于Adam及如何改进Muon"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.25990v1"
permalink: "https://feed.astralor.com/items/itm_e5fa96afdec4f09bd5193b24"
published: "2026-08-26T16:41:12.000Z"
score: 78
channel: "ai"
tags: ["Muon","SAMuon","优化器","大模型预训练","频谱分析"]
---

# 频谱分配：为何Muon优于Adam及如何改进Muon

## AI 点评

> 这篇论文值得关注，因为它不仅解释了Muon为何有效，还指出了其缺陷并给出可落地的改进方案。对做模型预训练和优化器研究的人尤其重要，省token就是省成本。

## 速览

本文通过离样本频谱探针分析，揭示了Muon等正交优化器相对Adam加速大模型预训练的机制。研究发现损失景观的频谱轮廓呈各向异性且稳定：头部处于失稳边缘，需要更小步长；主体部分则允许更大步长。基于此，作者提出频谱感知Muon（SAMuon），通过静态频谱先验放大主体步长，在124M至1B参数模型上比调优的AdamW和Muon节省13.3%-24.0%的训练token，且几乎不增加额外开销。

## 关键点

- 提出频谱分配统一解释：Muon优于Adam、Adam优于SGD的原因在于对损失景观头部和主体采用了不同的有效步长。
- 发现Muon的均匀缩放存在局限：主体部分仍未被充分利用，这启发了SAMuon的设计。
- SAMuon使用低秩随机SVD或幂迭代两级近似，不增加持久优化器状态和显著FLOPs，保持Muon的渐近收敛率。
- 在124M到1B参数的modded-nanogpt模型中，SAMuon比Muon节省13.3%-24.0%训练token，SAMuon-lite几乎零墙钟开销。

## 全文

Orthogonal optimisers such as Muon can substantially accelerate large language model pretraining relative to Adam, yet the mechanism remains incompletely understood. We investigate this through an out-of-sample spectral probing analysis of Transformer loss landscapes. At checkpoints along real training trajectories, we decompose each momentum buffer into its singular directions and estimate the loss-optimal step size along each direction on held-out data. The resulting spectral profile is anisotropic yet stable across batches and training stages, and consistent across the optimisers and model scales: a volatile head operating at the Edge-of-Stability supports a much smaller step size than the tolerant bulk, which permits substantially larger steps. This profile provides a unified spectral allocation account of why Muon outperforms Adam, which outperforms SGD. It also exposes a limitation of Muon's uniform scaling: it still underutilises the bulk. Guided by this finding, we introduce Spectral-Aware Muon (SAMuon), which holds the head at the Muon scale and amplifies the bulk using a static spectral prior. We provide two variants: the complete SAMuon follows the measured profile using a low-rank randomised SVD and the simplified SAMuon-lite uses a two-level approximation via rank-one power iteration. Neither method adds persistent optimiser state or notable extra FLOPs beyond Muon at scale, and the idealised exact-whitening versions of both retain Muon's asymptotic convergence rate under standard assumptions. Across "modded-nanogpt" models from 124M to 1B parameters, both variants outperform tuned AdamW and Muon (Scion implementation) baselines in all evaluated model-scale and batch-size configurations. SAMuon requires 13.3% to 24.0% fewer training tokens to reach the same validation loss as Muon, while SAMuon-lite retains most of this gain with near-zero wall-clock overhead.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_e5fa96afdec4f09bd5193b24