---
title: "Spectral Allocation: Why Muon Outperforms Adam, and How to Improve Muon"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.25990v1"
permalink: "https://feed.astralor.com/items/itm_e5fa96afdec4f09bd5193b24"
published: "2026-08-26T16:41:12.000Z"
score: 78
channel: "ai"
tags: ["Muon","SAMuon","Optimizer","LLM Pretraining","Spectral Analysis"]
---

# Spectral Allocation: Why Muon Outperforms Adam, and How to Improve Muon

## AI commentary

> This paper is worth attention as it not only explains why Muon works but also identifies its flaw and provides a practical fix. It matters especially for those working on LLM pretraining and optimizer research, since saving tokens means saving cost.

## At a glance

This paper investigates why orthogonal optimizers like Muon accelerate large language model pretraining over Adam via out-of-sample spectral probing of Transformer loss landscapes. The resulting spectral profile is anisotropic yet stable: a volatile head at the Edge-of-Stability supports smaller step sizes while the tolerant bulk permits much larger ones. Guided by this, the authors introduce Spectral-Aware Muon (SAMuon), which amplifies bulk step sizes using a static spectral prior. Across 124M to 1B parameter models, both variants outperform tuned AdamW and Muon, requiring 13.3%-24.0% fewer training tokens to reach the same validation loss, with near-zero wall-clock overhead.

## Key points

- Provides a unified spectral allocation account: Muon outperforms Adam and Adam outperforms SGD due to different effective step sizes on the head and bulk of the loss landscape.
- Reveals Muon's uniform scaling limitation: the bulk is still underutilized, motivating the SAMuon design.
- SAMuon uses low-rank randomized SVD or power-iteration two-level approximation, adding no persistent optimizer state or notable FLOPs, retaining Muon's asymptotic convergence rate.
- On modded-nanogpt models from 124M to 1B, SAMuon requires 13.3%-24.0% fewer training tokens than Muon; SAMuon-lite achieves most gains with near-zero wall-clock overhead.

## Full text

Orthogonal optimisers such as Muon can substantially accelerate large language model pretraining relative to Adam, yet the mechanism remains incompletely understood. We investigate this through an out-of-sample spectral probing analysis of Transformer loss landscapes. At checkpoints along real training trajectories, we decompose each momentum buffer into its singular directions and estimate the loss-optimal step size along each direction on held-out data. The resulting spectral profile is anisotropic yet stable across batches and training stages, and consistent across the optimisers and model scales: a volatile head operating at the Edge-of-Stability supports a much smaller step size than the tolerant bulk, which permits substantially larger steps. This profile provides a unified spectral allocation account of why Muon outperforms Adam, which outperforms SGD. It also exposes a limitation of Muon's uniform scaling: it still underutilises the bulk. Guided by this finding, we introduce Spectral-Aware Muon (SAMuon), which holds the head at the Muon scale and amplifies the bulk using a static spectral prior. We provide two variants: the complete SAMuon follows the measured profile using a low-rank randomised SVD and the simplified SAMuon-lite uses a two-level approximation via rank-one power iteration. Neither method adds persistent optimiser state or notable extra FLOPs beyond Muon at scale, and the idealised exact-whitening versions of both retain Muon's asymptotic convergence rate under standard assumptions. Across "modded-nanogpt" models from 124M to 1B parameters, both variants outperform tuned AdamW and Muon (Scion implementation) baselines in all evaluated model-scale and batch-size configurations. SAMuon requires 13.3% to 24.0% fewer training tokens to reach the same validation loss as Muon, while SAMuon-lite retains most of this gain with near-zero wall-clock overhead.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_e5fa96afdec4f09bd5193b24