---
title: "Open-MOPD: Diagnosing and Fixing Capability Imbalance in Multi-Teacher On-Policy Distillation"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.19098v1"
permalink: "https://feed.astralor.com/items/itm_89e4ef1ee59b33d7a05e1bf4"
published: "2026-08-19T16:50:39.000Z"
score: 77
channel: "ai"
tags: ["arXiv","Open-MOPD","SmolLM3-3B-Base","Multi-Teacher Distillation","Reinforcement Learning"]
---

# Open-MOPD: Diagnosing and Fixing Capability Imbalance in Multi-Teacher On-Policy Distillation

## AI commentary

> If you are working on multi-expert merging or generalist agent training, the 'budget misallocation' perspective here may challenge your previous gradient-conflict narrative. The three factors and fixes are concrete, and the open-source code lowers replication barriers significantly—worth a close read for any post-training team.

## At a glance

Multi-teacher on-policy distillation (M-OPD) consolidates domain-specialized RL experts into a single generalist student via dense token-level rewards, yet its optimization dynamics remain poorly understood. This paper builds a controlled benchmark on SmolLM3-3B-Base and finds that standard M-OPD captures only 35.6% of the headroom relative to an oracle ensemble. The failure stems from misallocation of the token-level optimization budget, not gradient conflict, driven by sequence-length disparities, non-uniform learning rates, and multi-step reward staleness. The proposed Open-MOPD framework introduces token-share balancing, gap-aware dynamic budget allocation, and student reward refresh, lifting headroom recovery to 83.4%. All post-training recipes, trajectories, and evaluation suites are open-sourced.

## Key points

- Standard M-OPD captures only 35.6% of the oracle-routed ensemble headroom, revealing a severe capability integration gap.
- The failure is caused by token-level optimization budget misallocation, not gradient conflict.
- Open-MOPD uses token-share balancing, gap-aware dynamic budget allocation, and reward refresh to achieve 83.4% headroom recovery.
- End-to-end recipes and evaluation suites are open-sourced for reproducible research on accessible hardware.

## Full text

Multi-teacher on-policy distillation (M-OPD) has emerged as a promising paradigm for consolidating domain-specialized reinforcement learning (RL) experts into a single generalist student via dense, token-level reward supervision. Despite its practical success, the optimization dynamics governing multi-teacher capability integration remain poorly understood, and open, rigorously reproducible recipes are conspicuously lacking. In this work, we establish a controlled M-OPD benchmark on SmolLM3-3B-Base with oracle routing, isolating capability integration from routing ambiguity. Our investigation reveals a pronounced capability integration gap: standard M-OPD captures only 35.6% of the available headroom relative to a domain-routed oracle ensemble, with concise tasks such as instruction following suffering severe degradation and premature stagnation. Crucially, we show that this failure stems not from gradient conflict, but from a severe misallocation of the token-level optimization budget. This pathology is driven by three orthogonal factors: structural sequence-length disparities across domains, dynamic convergence drift due to non-uniform learning rates, and multi-step reward staleness from asynchronous policy updates. To resolve these imbalances, we introduce Open-MOPD, a principled framework incorporating token-share balancing, gap-aware dynamic budget allocation, and student reward refresh. Together, these mechanisms systematically restore cross-domain balance, elevating headroom recovery from 35.6% to 83.4% in a single deployable student. We fully open-source our end-to-end post-training recipe, training trajectories, and evaluation suites on an academically accessible hardware budget.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_89e4ef1ee59b33d7a05e1bf4