---
title: "层次化缪子：用于高效缪子优化的分块牛顿-舒尔茨更新方法"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2606.27216v1"
permalink: "https://feed.astralor.com/items/itm_c590a37cac827c632d540827"
published: "2026-06-25T16:05:55.000Z"
score: 48
channel: "tech"
tags: ["HiMuon","GPU","Muon-type","Newton-Schulz","For"]
---

# 层次化缪子：用于高效缪子优化的分块牛顿-舒尔茨更新方法

## 速览

本研究提出层次化缪子（HiMuon），一种用于缪子型优化的分块牛顿-舒尔茨更新方案。HiMuon将动量梯度矩阵划分为T×T块，对每个块独立应用牛顿-舒尔茨映射，然后将结果重组。固定T时，计算量降至O(H W T K)，且可分解为独立的小矩阵运算，支持按块大小的GPU核、跨层批处理和运行时块大小调度。实验表明，HiMuon在保持训练行为接近全矩阵缪子的同时，提高了优化器步效率。

## 关键点

- Muon-type optimizers construct update directions for dense neural-network weights by applying a finite Newton-Schulz map to momentum-gradient matrices.
- For an $H \times W$ matrix, with $r=\min\{H,W\}$ and $s=\max\{H,W\}$, $K$ steps of the full-matrix Newton-Schulz update require $O(r^2 s K)$ work and couple all rows and columns through repeated Gram matrix products.
- We introduce Hierarchical Muon (HiMuon), a tiled Newton-Schulz scheme for Muon-type optimization.

## 全文

Muon-type optimizers construct update directions for dense neural-network weights by applying a finite Newton-Schulz map to momentum-gradient matrices. For an $H \times W$ matrix, with $r=\min\{H,W\}$ and $s=\max\{H,W\}$, $K$ steps of the full-matrix Newton-Schulz update require $O(r^2 s K)$ work and couple all rows and columns through repeated Gram matrix products. We introduce Hierarchical Muon (HiMuon), a tiled Newton-Schulz scheme for Muon-type optimization. HiMuon partitions each momentum-gradient matrix into $T \times T$ tiles, applies the same finite Newton-Schulz map independently to each tile, and reassembles the results. For finite $T$ below the matrix dimensions, HiMuon defines a local matrix-function map rather than a convergent approximation to the full-matrix update: spectral interactions are preserved within tiles and discarded across tile boundaries. For fixed finite $T$, the leading Newton-Schulz work decreases to $O(H W T K)$, and the computation decomposes into independent small dense matrix operations. This structure enables tile-size-dependent GPU kernels, cross-layer batching, memory-bounded chunking, and runtime tile-size schedules. Experiments on transformer training and controlled matrix-function diagnostics show that HiMuon improves optimizer-step efficiency while keeping training behavior close to full-matrix Muon in the tested regimes.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_c590a37cac827c632d540827