---
title: "损失函数看不见基，但Adam看得见：规范等变性决定优化器的低秩偏置"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.05136v1"
permalink: "https://feed.astralor.com/items/itm_212d74d857fe1c562374eaef"
published: "2026-08-05T17:56:26.000Z"
score: 77
channel: "ai"
tags: ["Adam","Muon","优化器设计","低秩偏置","规范等变性"]
---

# 损失函数看不见基，但Adam看得见：规范等变性决定优化器的低秩偏置

## AI 点评

> 我觉得这篇论文最漂亮的地方，是把“优化器用哪个坐标基”从经验黑箱变成了可证明的数学条件。对想用Muon或Shampoo替换Adam的工程团队尤其重要：如果目标是低秩压缩或更紧凑的插值解，等变优化器才是正确方向。

## 速览

这篇论文发现，在矩阵分解模型 W=UV^T 中，梯度下降因损失函数的规范对称性而隐式偏向低秩解，Adam却不会。作者将差异追溯到规范等变性：GD、动量、shared-scalar Adam、Muon 和 Shampoo 满足该条件，而 Adam、RMSProp 等逐坐标方法不满足。结构定理刻画了无记忆等变更新规则等价于 Gram 决定的左预条件器；实验证明，基选择不是调参细节，而是决定优化器选择哪种插值函数的关键。

## 关键点

- 规范等变性决定优化器能否继承梯度流的低秩偏置：GD、动量、shared-scalar Adam、Muon、Shampoo满足，Adam、RMSProp等逐坐标方法不满足。
- 结构定理证明无记忆等变更新规则恰好是Gram决定的左预条件器，迁移定理将梯度流的路径性质推广到共享标量流。
- Transformer实验中Adam第一步就分离规范等价初始化；超光谱数据上梯度下降在最低采样密度下将留出误差降低43-44%。

## 全文

Gradient descent on a factored model $W = UV^\top$ is implicitly biased toward low-rank solutions, while Adam, starting from the same small initialization, is not. We trace the difference to the gauge symmetry of the loss, its invariance under $(U, V) \mapsto (UQ, VQ)$. Gradient flow's low-rank mechanism is available to an optimizer only if that optimizer is gauge-equivariant, a condition necessary for the transfer but not sufficient for low-rank recovery. Gradient descent, momentum, "shared-scalar" Adam, Muon, and Shampoo satisfy it. Adam, RMSProp, and the other coordinate-wise methods do not. A structure theorem characterizes the memoryless equivariant rules as exactly the Gram-determined left preconditioners, and a transfer theorem carries gradient flow's pathwise properties to common-scalar flows. We then sort nine update rules on underdetermined matrix sensing by recovery error against the planted ground truth. A one-parameter family from coordinate-wise to shared-scalar preconditioning restores the bias monotonically, isolating anisotropy as the cause. A "spectral schedule" reconciles two opposing reports about Muon: equal-rate updates recover exactly low-rank targets but lose their edge as the spectral tail grows. In transformers, Adam separates two gauge-equivalent initializations at the first step, where the equivariant optimizers stay at float precision, and ends with the per-head invariants $W_Q^\top W_K$ 56% apart in relative Frobenius distance, a gap no per-head rotation can close. On two hyperspectral datasets at matched training loss, gradient descent cuts held-out error by 43-44% at the lowest sampling density, and at lower effective rank. Basis choice is therefore not a tuning detail but a decision about which interpolant the optimizer selects.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_212d74d857fe1c562374eaef