---
title: "MMOE：通过高效专家设计现代化扩散Transformer"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.24665v1"
permalink: "https://feed.astralor.com/items/itm_b75064f8612c0a2c3a5d0c3f"
published: "2026-07-27T17:05:04.000Z"
score: 81
channel: "ai"
tags: ["AIGC基础模型","MMOE","专家混合","扩散Transformer","高效扩展"]
---

# MMOE：通过高效专家设计现代化扩散Transformer

## AI 点评

> 这篇论文值得所有关注AIGC效率优化和扩散模型架构的研究者关注，它用清晰的实验证明了LLM中的MoE设计思路可以有效地移植到扩散Transformer中，并且仅用单机8卡就能取得显著提升，非常实用。对于希望在不增加海量算力的情况下提升生成模型质量的研究团队来说，MMOE提供了直接可行的方案。

## 速览

MMOE是一种现代化扩散Transformer架构，系统性地将路由专家、共享专家、轻量级专家、门控残差路由和注意力残差信息复用引入AIGC生成。在单机8×H100、批量256、训练40万步的预算下，MMOE在每个记录检查点均达到比密集基线和中间稀疏专家基线更低的FID，收敛更快，并在稀疏变体中实现了最佳质量-成本平衡。路由分析显示专家跨层稳定专业化、轻量级路由广泛使用以及去噪过程中逐步骤路由变化适中。研究表明AIGC基础模型可以借鉴LLM的高效扩展路径，而不是简单地增加总参数量和稀疏率。

## 关键点

- MMOE将路由专家、共享专家、轻量级专家、门控残差路由和注意力残差信息复用系统性地整合到扩散Transformer中。
- 在单机8×H100的有限计算预算下，MMOE在收敛速度和生成质量（FID）上均优于密集基线和现有稀疏专家基线。
- 路由分析揭示了专家跨层稳定专业化、轻量级路由的广泛使用以及去噪过程中路由变化的适度性。
- 该工作表明AIGC基础模型可以借鉴LLM的高效扩展设计，而非仅通过增加参数和稀疏率来提升性能。

## 全文

Modern large language models scale successfully by pairing capacity growth with efficiency, keeping per-token and deployment costs under control as capacity grows. AIGC Foundation Models (AFMs), especially diffusion-transformer backbones, have begun to adopt sparse experts, but recent efforts mostly enlarge total parameter counts and sparsity ratios without importing the efficiency mechanisms that made LLM scaling practical, so generation quality is seldom balanced against training and deployment cost. This raises a natural question: can the architectural principles behind efficient LLM scaling be adapted to AFMs in a more balanced way? We introduce ModernMOE (MMOE), a modernization of SiT-style diffusion transformers that systematically adapts routed experts, shared and lightweight experts, gate-residual routing, and attention-residual information reuse to AIGC generation. Rather than treating MoE as a single plug-in replacement, MMOE studies how different modern expert components affect convergence, efficiency, and generation quality when composed inside a diffusion transformer. Every experiment in this paper is trained on a single eight-GPU H100 node with batch size 256 for 400k steps, an accessible single-machine budget. Under matched training and sampling protocols and at this budget, MMOE reaches lower FID at every recorded checkpoint, that is, it converges faster per training step, than dense and intermediate sparse-expert baselines, and among the sparse variants it attains the best quality-cost balance. Routing analysis further shows stable expert specialization across depth, substantial use of lightweight routes, and modest step-to-step routing changes during denoising. These results suggest that AFMs can follow the balanced scaling path of LLMs by importing proven efficiency designs, rather than by simply increasing total parameters and sparsity ratios.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_b75064f8612c0a2c3a5d0c3f