层次化缪子：用于高效缪子优化的分块牛顿-舒尔茨更新方法

Mathematics > Numerical Analysis

arXiv:2606.27216v1 (math) [Submitted on 25 Jun 2026]

Title:Hierarchical Muon: Tiled Newton-Schulz Updates for Efficient Muon Optimization

Authors:Ziyuan Tang, Tianshi Xu, Yousef Saad, Yuanzhe Xi View a PDF of the paper titled Hierarchical Muon: Tiled Newton-Schulz Updates for Efficient Muon Optimization, by Ziyuan Tang and 3 other authors View PDF HTML (experimental)

Abstract:Muon-type optimizers construct update directions for dense neural-network weights by applying a finite Newton-Schulz map to momentum-gradient matrices. For an $H \times W$ matrix, with $r=\min\{H,W\}$ and $s=\max\{H,W\}$, $K$ steps of the full-matrix Newton-Schulz update require $O(r^2 s K)$ work and couple all rows and columns through repeated Gram matrix products. We introduce Hierarchical Muon (HiMuon), a tiled Newton-Schulz scheme for Muon-type optimization. HiMuon partitions each momentum-gradient matrix into $T \times T$ tiles, applies the same finite Newton-Schulz map independently to each tile, and reassembles the results. For finite $T$ below the matrix dimensions, HiMuon defines a local matrix-function map rather than a convergent approximation to the full-matrix update: spectral interactions are preserved within tiles and discarded across tile boundaries. For fixed finite $T$, the leading Newton-Schulz work decreases to $O(H W T K)$, and the computation decomposes into independent small dense matrix operations. This structure enables tile-size-dependent GPU kernels, cross-layer batching, memory-bounded chunking, and runtime tile-size schedules. Experiments on transformer training and controlled matrix-function diagnostics show that HiMuon improves optimizer-step efficiency while keeping training behavior close to full-matrix Muon in the tested regimes.

Comments:	23 pages, 10 figures, 3 tables
Subjects:	Numerical Analysis (math.NA); Machine Learning (cs.LG)
MSC classes:	65F30, 90C06, 68T07
Cite as:	arXiv:2606.27216 [math.NA]
	(or arXiv:2606.27216v1 [math.NA] for this version)
	https://doi.org/10.48550/arXiv.2606.27216 Focus to learn more arXiv-issued DOI via DataCite

Submission history

From: Ziyuan Tang [view email]
[v1] Thu, 25 Jun 2026 16:05:55 UTC (1,825 KB)
Full-text links:

Access Paper:

View a PDF of the paper titled Hierarchical Muon: Tiled Newton-Schulz Updates for Efficient Muon Optimization, by Ziyuan Tang and 3 other authors
• View PDF
• HTML (experimental)
• TeX Source

view license

Current browse context:

math.NA < prev   |   next >
new | recent | 2026-06 Change to browse by: cs
cs.LG
cs.NA
math

References & Citations

• NASA ADS
• Google Scholar
• Semantic Scholar

export BibTeX citation Loading...

BibTeX formatted citation

× Data provided by:

Bookmark

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle Bibliographic Explorer (What is the Explorer?) Connected Papers Toggle Connected Papers (What is Connected Papers?) Litmaps Toggle Litmaps (What is Litmaps?) scite.ai Toggle scite Smart Citations (What are Smart Citations?) Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle alphaXiv (What is alphaXiv?) Links to Code Toggle CatalyzeX Code Finder for Papers (What is CatalyzeX?) DagsHub Toggle DagsHub (What is DagsHub?) GotitPub Toggle Gotit.pub (What is GotitPub?) Huggingface Toggle Hugging Face (What is Huggingface?) ScienceCast Toggle ScienceCast (What is ScienceCast?) Demos

Demos

Replicate Toggle Replicate (What is Replicate?) Spaces Toggle Hugging Face Spaces (What is Spaces?) Spaces Toggle TXYZ.AI (What is TXYZ.AI?) Related Papers

Recommenders and Search Tools

Link to Influence Flower Influence Flower (What are Influence Flowers?) Core recommender toggle CORE Recommender (What is CORE?)

• Author
• Venue
• Institution
• Topic

About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)