---
title: "Muon Learns Balanced Solutions in Matrix Factorization without Slow Saddle-to-Saddle Dynamics"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2606.30509v1"
permalink: "https://feed.astralor.com/items/itm_2a4d044739f70c37a2253f29"
published: "2026-06-29T16:17:38.000Z"
score: 57
channel: "ai"
tags: ["Muon optimizer","Learning Rate Scheduling","Matrix Factorization","Saddle Point Dynamics"]
---

# Muon Learns Balanced Solutions in Matrix Factorization without Slow Saddle-to-Saddle Dynamics

## At a glance

This paper investigates parameter trajectories under the Muon optimizer in matrix factorization, identifying three key dynamical differences from gradient descent: avoiding slow saddle-to-saddle dynamics from small initialization, learning all top modes at the same rate; stability beyond local sharpness thresholds enabling exponential annealing; and a different conserved quantity that still leads to balanced solutions. It derives alignment rates for spontaneous early weight alignment and constructs a two-step learning rate schedule achieving near-perfect alignment.

## Key points

- Muon learns all top modes simultaneously from small initialization, avoiding slow saddle-to-saddle dynamics of gradient descent.
- Learning rate can exceed local sharpness threshold, freeing from condition number and enabling exponential annealing.
- Muon flow conserves √(P^T P)-√(Q^T Q) instead of gradient flow's P^T P - Q^T Q.
- A two-step learning schedule exploiting Muon's structure achieves near-perfect alignment.

## Full text

Matrix factorization (i.e., problems of the form $\min_{\mathbf{P},\mathbf{Q}} \|\mathbf{M}^\star - \mathbf{P}^\top\mathbf{Q}\|_\mathrm{F}^2$) is a minimal learning problem that exhibits both nonlinear parameter dynamics and representation learning. In this setting, we study how parameter trajectories under the Muon optimizer differ from those of gradient descent. We identify three main dynamical differences: 1) Muon avoids the slow saddle-to-saddle dynamics from small initialization. Muon instead learns all the top modes of $\mathbf{M}^\star$ at the same rate, with the smaller modes converging first. 2) Muon remains stable even when the learning rate exceeds the critical threshold set by the local loss sharpness. This frees the learning rate from the condition number of the problem, enabling rapid convergence via exponential learning rate annealing. 3) Once the weights are aligned with each other and the target, Muon flow conserves the matrix quantity $\sqrt{\mathbf{P}^\top \mathbf{P}}-\sqrt{\mathbf{Q}^\top \mathbf{Q}}$, while gradient flow is known to conserve the matrix $\mathbf{P}^\top\mathbf{P} - \mathbf{Q}^\top\mathbf{Q}$. Despite having distinct conserved quantities, both optimizers find the so-called \textit{balanced} solution from vanishing initialization. When training from small random initialization, the weights spontaneously align early in training. We derive the alignment rates in simple settings and show that they predict the empirical alignment rates in general. Finally, we exploit structural properties of Muon to construct a learning rate schedule that achieves near-perfect alignment in only two optimization steps.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_2a4d044739f70c37a2253f29