---
title: "MMOE: Modernizing Diffusion Transformers with Efficient Expert Design"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.24665v1"
permalink: "https://feed.astralor.com/items/itm_b75064f8612c0a2c3a5d0c3f"
published: "2026-07-27T17:05:04.000Z"
score: 81
channel: "ai"
tags: ["AIGC Foundation Models","MMOE","Mixture of Experts","Diffusion Transformer","Efficient Scaling"]
---

# MMOE: Modernizing Diffusion Transformers with Efficient Expert Design

## AI commentary

> This paper deserves attention from researchers focused on AIGC efficiency optimization and diffusion model architecture. It demonstrates with clear experiments that MoE design principles from LLMs can be effectively transferred to diffusion transformers, achieving significant improvements with only a single 8-GPU node, making it highly practical. For teams aiming to enhance generation model quality without massive compute scaling, MMOE offers a directly actionable solution.

## At a glance

MMOE is a modernized diffusion transformer architecture that systematically introduces routed experts, shared and lightweight experts, gate-residual routing, and attention-residual information reuse into AIGC generation. Under a budget of a single 8×H100 node with batch size 256 for 400k steps, MMOE achieves lower FID at every recorded checkpoint than dense and intermediate sparse-expert baselines, converges faster, and attains the best quality-cost balance among sparse variants. Routing analysis shows stable expert specialization across depth, substantial use of lightweight routes, and modest step-to-step routing changes during denoising. The study suggests that AIGC foundation models can follow the balanced scaling path of LLMs by importing proven efficiency designs rather than simply increasing total parameters and sparsity ratios.

## Key points

- MMOE systematically integrates routed experts, shared experts, lightweight experts, gate-residual routing, and attention-residual information reuse into diffusion transformers.
- Under a limited budget of a single 8×H100 node, MMOE outperforms dense and existing sparse-expert baselines in both convergence speed and generation quality (FID).
- Routing analysis reveals stable expert specialization across layers, substantial use of lightweight routes, and moderate step-to-step routing changes during denoising.
- The work demonstrates that AIGC foundation models can follow the efficient scaling path of LLMs by importing proven efficiency designs rather than merely increasing parameters and sparsity ratios.

## Full text

Modern large language models scale successfully by pairing capacity growth with efficiency, keeping per-token and deployment costs under control as capacity grows. AIGC Foundation Models (AFMs), especially diffusion-transformer backbones, have begun to adopt sparse experts, but recent efforts mostly enlarge total parameter counts and sparsity ratios without importing the efficiency mechanisms that made LLM scaling practical, so generation quality is seldom balanced against training and deployment cost. This raises a natural question: can the architectural principles behind efficient LLM scaling be adapted to AFMs in a more balanced way? We introduce ModernMOE (MMOE), a modernization of SiT-style diffusion transformers that systematically adapts routed experts, shared and lightweight experts, gate-residual routing, and attention-residual information reuse to AIGC generation. Rather than treating MoE as a single plug-in replacement, MMOE studies how different modern expert components affect convergence, efficiency, and generation quality when composed inside a diffusion transformer. Every experiment in this paper is trained on a single eight-GPU H100 node with batch size 256 for 400k steps, an accessible single-machine budget. Under matched training and sampling protocols and at this budget, MMOE reaches lower FID at every recorded checkpoint, that is, it converges faster per training step, than dense and intermediate sparse-expert baselines, and among the sparse variants it attains the best quality-cost balance. Routing analysis further shows stable expert specialization across depth, substantial use of lightweight routes, and modest step-to-step routing changes during denoising. These results suggest that AFMs can follow the balanced scaling path of LLMs by importing proven efficiency designs, rather than by simply increasing total parameters and sparsity ratios.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_b75064f8612c0a2c3a5d0c3f