---
title: "逐步缩放：大规模混合专家模型的高计算效率超参数迁移"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.20061v1"
permalink: "https://feed.astralor.com/items/itm_8b0167c1162be6f5797dcb62"
published: "2026-08-20T13:57:43.000Z"
score: 77
channel: "ai"
tags: ["Muon优化器","多头潜在注意力","最大更新参数化","混合专家模型","缩放定律","超参数迁移"]
---

# 逐步缩放：大规模混合专家模型的高计算效率超参数迁移

## AI 点评

> 这篇最值得看的地方是把超参数迁移从'经验技巧'变成了'可外推的定量规律'，尤其是R²=0.95的token维度预测，对大模型团队算力规划很有参考意义。对正在训练MoE或准备做超大规模预训练的研究者来说，这可能是省下大量试错成本的关键工具。

## 速览

混合专家（MoE）架构能以较低计算成本扩展模型容量，但在极大模型规模和海量token预算下，网格搜索最优超参数（尤其是学习率）成本过高。本文提出两步式高计算效率超参数迁移框架：首先基于最大更新参数化（μP）适配含多头潜在注意力（MLA）和Muon优化器的MoE架构，证明最优学习率可跨模型宽度迁移；其次沿token维度建立预测缩放规律，通过小型代理模型回归，以R²=0.95的拟合度将最优学习率外推到10万亿token规模。作者用该法从零预训练155B总参（17B激活）模型，验证了最小消融成本即可准确预测全规模目标超参数。

## 关键点

- 提出针对MoE的μP适配方案，覆盖MLA与Muon优化器，验证最优学习率可跨模型宽度迁移。
- 建立token维度预测缩放律，用线性回归以R²=0.95将学习率外推至10万亿token训练规模。
- 在155B总参/17B激活的模型上从零预训练，实证表明小模型代理训练可准确预测全规模最优配置。

## 全文

Mixture-of-Experts (MoE) architectures significantly expand model capacity without a proportional increase in computational cost. However, optimizing their hyperparameters---particularly the learning rate---at extreme scales of both model size and token budget via sweeping remains computationally prohibitive. In this paper, we propose a compute-efficient, two-step hyperparameter transfer framework that estimates optimal learning rates for training large MoE models by transferring them across scaling model widths, and subsequently extrapolating to trillion-token horizons. First, we formulate a Maximal Update Parameterization ($μ$P) adaptation for MoE architectures utilizing Multi-head Latent Attention (MLA) and the Muon optimizer, demonstrating that optimal learning rates transfer consistently across width-scaled models. Second, we extend this transferability along the token dimension by establishing a predictive scaling law. By applying linear regression to the optimal values derived from small proxy models on limited budgets, we successfully extrapolate the ideal learning rate to massive training horizons (e.g., 10 trillion tokens) with high fidelity ($R^2=0.95$). Consequently, this indicates that proxy training on small models is sufficient to determine the optimal learning rate for the extensive training of large-scale MoEs. We apply the proposed methodology to pretrain our foundation model (155B total, 17B active parameters) from scratch, and the stable training and evaluation results validate that optimal configurations for full-scale target models can be accurately predicted with minimal ablation costs.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_8b0167c1162be6f5797dcb62