---
title: "Let's Scale Step by Step: Compute-Efficient Hyperparameter Transfer for Large-Scale Mixture-of-Experts"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.20061v1"
permalink: "https://feed.astralor.com/items/itm_8b0167c1162be6f5797dcb62"
published: "2026-08-20T13:57:43.000Z"
score: 77
channel: "ai"
tags: ["Muon optimizer","Multi-head Latent Attention","Maximal Update Parameterization","Mixture-of-Experts","Scaling Law","Hyperparameter Transfer"]
---

# Let's Scale Step by Step: Compute-Efficient Hyperparameter Transfer for Large-Scale Mixture-of-Experts

## AI commentary

> The most valuable takeaway is turning hyperparameter transfer from an empirical trick into a quantifiable extrapolation law, especially the token-dimension prediction at R²=0.95, which is highly relevant for compute planning. For researchers training MoE or preparing ultra-large-scale pretraining, this could be a key tool to save massive trial-and-error costs.

## At a glance

Mixture-of-Experts (MoE) architectures expand model capacity at sublinear computational cost, yet sweeping hyperparameters—especially learning rate—at extreme model sizes and trillion-token budgets remains prohibitive. This paper proposes a compute-efficient two-step transfer framework: first, adapt Maximal Update Parameterization (μP) to MoE models with Multi-head Latent Attention (MLA) and the Muon optimizer, showing that optimal learning rates transfer consistently across width-scaled models; second, extend transferability along the token dimension via a predictive scaling law. Linear regression on small proxy runs extrapolates the ideal learning rate to massive horizons (e.g., 10 trillion tokens) with high fidelity (R²=0.95). The authors pretrain a 155B-total / 17B-active foundation model from scratch, validating that full-scale optimal configurations can be predicted with minimal ablation cost.

## Key points

- Proposes a μP adaptation for MoE architectures with MLA and the Muon optimizer, demonstrating width-wise transfer of optimal learning rates.
- Establishes a token-dimension predictive scaling law, extrapolating learning rates to 10-trillion-token horizons with R²=0.95.
- Validates on a 155B-total/17B-active foundation model pretrained from scratch, showing small proxy runs can predict full-scale optimal configurations.

## Full text

Mixture-of-Experts (MoE) architectures significantly expand model capacity without a proportional increase in computational cost. However, optimizing their hyperparameters---particularly the learning rate---at extreme scales of both model size and token budget via sweeping remains computationally prohibitive. In this paper, we propose a compute-efficient, two-step hyperparameter transfer framework that estimates optimal learning rates for training large MoE models by transferring them across scaling model widths, and subsequently extrapolating to trillion-token horizons. First, we formulate a Maximal Update Parameterization ($μ$P) adaptation for MoE architectures utilizing Multi-head Latent Attention (MLA) and the Muon optimizer, demonstrating that optimal learning rates transfer consistently across width-scaled models. Second, we extend this transferability along the token dimension by establishing a predictive scaling law. By applying linear regression to the optimal values derived from small proxy models on limited budgets, we successfully extrapolate the ideal learning rate to massive training horizons (e.g., 10 trillion tokens) with high fidelity ($R^2=0.95$). Consequently, this indicates that proxy training on small models is sufficient to determine the optimal learning rate for the extensive training of large-scale MoEs. We apply the proposed methodology to pretrain our foundation model (155B total, 17B active parameters) from scratch, and the stable training and evaluation results validate that optimal configurations for full-scale target models can be accurately predicted with minimal ablation costs.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_8b0167c1162be6f5797dcb62