---
title: "Open-MOPD：诊断与修复多教师同策略蒸馏中的能力失衡"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.19098v1"
permalink: "https://feed.astralor.com/items/itm_89e4ef1ee59b33d7a05e1bf4"
published: "2026-08-19T16:50:39.000Z"
score: 77
channel: "ai"
tags: ["arXiv","Open-MOPD","SmolLM3-3B-Base","多教师蒸馏","强化学习"]
---

# Open-MOPD：诊断与修复多教师同策略蒸馏中的能力失衡

## AI 点评

> 如果你正在做多专家模型融合或通用Agent训练，这篇论文的‘预算错配’观点会颠覆你以前认为的梯度冲突叙事。它给出的三个因素和修复手段都很具体，开源代码让复现门槛极低，值得所有做后训练的团队认真读一遍。

## 速览

多教师同策略蒸馏（M-OPD）通过密集的令牌级奖励将多个领域专家整合为通用学生模型，但优化动力学仍不清晰。论文在SmolLM3-3B-Base上构建可控基准，发现标准M-OPD仅捕获35.6%的理论性能空间，问题源于令牌级优化预算错配而非梯度冲突，并受序列长度差异、非均匀学习率和异步奖励陈旧性三因素驱动。作者提出Open-MOPD框架，通过令牌份额平衡、动态预算分配和学生奖励刷新，将性能恢复提升至83.4%，并开源全部训练配方与评估套件。

## 关键点

- 标准M-OPD仅恢复35.6%的域路由集成性能，暴露明显能力整合缺口。
- 失败根源是令牌级优化预算错配，而非梯度冲突。
- Open-MOPD通过令牌份额平衡、动态预算分配和奖励刷新，将恢复率提升至83.4%。
- 全流程配方与评估套件已开源，可在学术可负担硬件上复现。

## 全文

Multi-teacher on-policy distillation (M-OPD) has emerged as a promising paradigm for consolidating domain-specialized reinforcement learning (RL) experts into a single generalist student via dense, token-level reward supervision. Despite its practical success, the optimization dynamics governing multi-teacher capability integration remain poorly understood, and open, rigorously reproducible recipes are conspicuously lacking. In this work, we establish a controlled M-OPD benchmark on SmolLM3-3B-Base with oracle routing, isolating capability integration from routing ambiguity. Our investigation reveals a pronounced capability integration gap: standard M-OPD captures only 35.6% of the available headroom relative to a domain-routed oracle ensemble, with concise tasks such as instruction following suffering severe degradation and premature stagnation. Crucially, we show that this failure stems not from gradient conflict, but from a severe misallocation of the token-level optimization budget. This pathology is driven by three orthogonal factors: structural sequence-length disparities across domains, dynamic convergence drift due to non-uniform learning rates, and multi-step reward staleness from asynchronous policy updates. To resolve these imbalances, we introduce Open-MOPD, a principled framework incorporating token-share balancing, gap-aware dynamic budget allocation, and student reward refresh. Together, these mechanisms systematically restore cross-domain balance, elevating headroom recovery from 35.6% to 83.4% in a single deployable student. We fully open-source our end-to-end post-training recipe, training trajectories, and evaluation suites on an academically accessible hardware budget.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_89e4ef1ee59b33d7a05e1bf4