---
title: "The Loss Does Not See the Basis, but Adam Does: Gauge Equivariance Determines Optimizers' Low-Rank Bias"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.05136v1"
permalink: "https://feed.astralor.com/items/itm_212d74d857fe1c562374eaef"
published: "2026-08-05T17:56:26.000Z"
score: 77
channel: "ai"
tags: ["Adam","Muon","optimizer design","low-rank bias","gauge equivariance"]
---

# The Loss Does Not See the Basis, but Adam Does: Gauge Equivariance Determines Optimizers' Low-Rank Bias

## AI commentary

> I think the most elegant part of this paper is turning optimizer coordinate choice into a provable mathematical condition rather than an empirical black box. For engineering teams who want to replace Adam with Muon or Shampoo, this is especially important: if the goal is low-rank compression or more compact interpolants, equivariant optimizers are the right direction.

## At a glance

This paper shows that in factored models W = UV^T, gradient descent is implicitly biased toward low-rank solutions, while Adam is not. The difference is traced to gauge symmetry: the loss is invariant under (U,V)->(UQ,VQ), but only gauge-equivariant optimizers can inherit gradient flow's low-rank mechanism. Gradient descent, momentum, shared-scalar Adam, Muon, and Shampoo satisfy this condition; Adam, RMSProp, and other coordinate-wise methods do not. The authors prove a structure theorem identifying memoryless equivariant rules with Gram-determined left preconditioners, and a transfer theorem for common-scalar flows. Experiments on matrix sensing, Transformers, and hyperspectral datasets confirm that basis choice determines which interpolant the optimizer selects, with gradient descent cutting held-out error 43-44% at the lowest sampling density.

## Key points

- Gauge equivariance determines whether an optimizer inherits gradient flow's low-rank bias: GD, momentum, shared-scalar Adam, Muon, and Shampoo satisfy it, while Adam, RMSProp, and coordinate-wise methods do not.
- A structure theorem shows memoryless equivariant update rules are exactly Gram-determined left preconditioners, and a transfer theorem extends gradient-flow path properties to common-scalar flows.
- In Transformers, Adam separates gauge-equivalent initializations at the first step; on hyperspectral data, gradient descent reduces held-out error by 43-44% at the lowest sampling density.

## Full text

Gradient descent on a factored model $W = UV^\top$ is implicitly biased toward low-rank solutions, while Adam, starting from the same small initialization, is not. We trace the difference to the gauge symmetry of the loss, its invariance under $(U, V) \mapsto (UQ, VQ)$. Gradient flow's low-rank mechanism is available to an optimizer only if that optimizer is gauge-equivariant, a condition necessary for the transfer but not sufficient for low-rank recovery. Gradient descent, momentum, "shared-scalar" Adam, Muon, and Shampoo satisfy it. Adam, RMSProp, and the other coordinate-wise methods do not. A structure theorem characterizes the memoryless equivariant rules as exactly the Gram-determined left preconditioners, and a transfer theorem carries gradient flow's pathwise properties to common-scalar flows. We then sort nine update rules on underdetermined matrix sensing by recovery error against the planted ground truth. A one-parameter family from coordinate-wise to shared-scalar preconditioning restores the bias monotonically, isolating anisotropy as the cause. A "spectral schedule" reconciles two opposing reports about Muon: equal-rate updates recover exactly low-rank targets but lose their edge as the spectral tail grows. In transformers, Adam separates two gauge-equivalent initializations at the first step, where the equivariant optimizers stay at float precision, and ends with the per-head invariants $W_Q^\top W_K$ 56% apart in relative Frobenius distance, a gap no per-head rotation can close. On two hyperspectral datasets at matched training loss, gradient descent cuts held-out error by 43-44% at the lowest sampling density, and at lower effective rank. Basis choice is therefore not a tuning detail but a decision about which interpolant the optimizer selects.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_212d74d857fe1c562374eaef