---
title: "超越教师似然：用于长上下文推理的组校准在线策略蒸馏"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.19181v1"
permalink: "https://feed.astralor.com/items/itm_15f045bd4dd0b79fcb8caa2f"
published: "2026-08-19T17:54:58.000Z"
score: 75
channel: "tech"
tags: ["GC-OPD","On-policy Distillation","Qwen3","信用分配","长上下文推理"]
---

# 超越教师似然：用于长上下文推理的组校准在线策略蒸馏

## AI 点评

> 这项研究对从事模型蒸馏和长上下文处理的研究者很有参考价值。它没有简单地在蒸馏中加入验证器信号，而是通过组内归一化构造了可解释的残差项，思路值得借鉴。开源代码也便于复现和进一步实验。

## 速览

在线策略蒸馏（OPD）在长上下文任务中会出现教师模型与验证器评分错配的问题：token级教师指导可能忽略分布在长输入中的证据或违反全局约束，而任务级验证器提供整体奖励。本文系统诊断了这一错配，并提出组校准在线策略蒸馏（GC-OPD），在每组内分别归一化验证器奖励和轨迹级OPD分数，用其差值作为带符号的教师-验证器分歧残差，再通过相对优势信用分配（RACA）将残差分配到各token。在五个长上下文基准上，Qwen3-4B和Qwen3-8B平均分分别从29.08提升至40.47、从35.12提升至44.65，优于普通OPD（39.31和43.56）。消融实验表明带符号残差和RACA均有效，代码已开源。

## 关键点

- 诊断了长上下文任务中轨迹级OPD分数与验证器奖励随输入长度增加而逐渐失配的问题。
- 提出GC-OPD：分组归一化后利用带符号的教师-验证器分歧残差，并结合相对优势信用分配逐token分配。
- 在五个长上下文基准上显著提升Qwen3-4B/8B的蒸馏效果，优于普通OPD、额外OPD项和直接加验证器奖励等变体。

## 全文

On-policy distillation (OPD) trains a student on its own responses using dense token-level guidance from a stronger teacher. In long-context tasks, however, token-level teacher support can favor locally plausible responses that omit evidence distributed across the input or violate global task constraints. Task-specific verifiers, in contrast, evaluate task completion at the response level and may return graded rewards that reflect partial success. We diagnose this mismatch on fixed responses from two representative long-context evidence-aggregation tasks. Across longer input ranges, trajectory-level OPD scores become progressively less aligned with verifier rewards, indicating teacher-verifier disagreement. Motivated by this observation, we introduce Group-Calibrated On-Policy Distillation (GC-OPD). GC-OPD separately normalizes verifier rewards and trajectory-level OPD scores within each rollout group and uses their difference as a signed teacher-verifier disagreement residual. Relative-advantage-based credit assignment (RACA) distributes this trajectory-level residual across tokens according to their relative OPD advantages while preserving the original OPD signal. Across five long-context benchmarks, post-training with GC-OPD raises the five-benchmark averages of the official Qwen3-4B and Qwen3-8B checkpoints from 29.08 to 40.47 and from 35.12 to 44.65, respectively. Vanilla OPD reaches 39.31 and 43.56 under the same setup. Controlled ablations show that the signed residual is more effective than either an additional OPD-derived term or direct group-normalized verifier reward addition, while RACA further improves over uniform token allocation. Together, these results demonstrate that group-relative residual calibration can incorporate verifier outcomes without discarding dense token-level guidance. Code is available at https://github.com/SolereZhang/GC-OPD.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_15f045bd4dd0b79fcb8caa2f