---
title: "Beyond Teacher Likelihood: Group-Calibrated On-Policy Distillation for Long-Context Reasoning"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.19181v1"
permalink: "https://feed.astralor.com/items/itm_15f045bd4dd0b79fcb8caa2f"
published: "2026-08-19T17:54:58.000Z"
score: 75
channel: "tech"
tags: ["GC-OPD","On-policy Distillation","Qwen3","Credit Assignment","Long-context Reasoning"]
---

# Beyond Teacher Likelihood: Group-Calibrated On-Policy Distillation for Long-Context Reasoning

## AI commentary

> This research is valuable for researchers working on distillation and long-context handling. Rather than simply injecting verifier signals into distillation, it constructs an interpretable residual via group normalization—an idea worth borrowing. The open-source code also facilitates reproduction and further experiments.

## At a glance

On-policy distillation (OPD) trains a student model on its own responses with dense token-level guidance from a stronger teacher. However, in long-context tasks, such guidance often favors locally plausible responses that miss distributed evidence or violate global constraints, while task-specific verifiers provide response-level graded rewards. This paper systematically diagnoses this teacher-verifier mismatch on two representative evidence-aggregation tasks, showing that trajectory-level OPD scores become less aligned with verifier rewards over longer inputs. To address it, the authors propose Group-Calibrated On-Policy Distillation (GC-OPD), which normalizes verifier rewards and OPD scores within each rollout group, uses their difference as a signed disagreement residual, and redistributes it via relative-advantage-based credit assignment. Across five long-context benchmarks, GC-OPD improves Qwen3-4B and 8B averages from 29.08 to 40.47 and 35.12 to 44.65, outperforming vanilla OPD. Code is available.

## Key points

- Diagnoses the progressive mismatch between trajectory-level OPD scores and verifier rewards as input length increases in long-context tasks.
- Proposes GC-OPD, which uses a signed teacher-verifier disagreement residual after group normalization and distributes it token-wise via relative-advantage-based credit assignment.
- Significantly improves distilled Qwen3-4B/8B on five long-context benchmarks, outperforming vanilla OPD, extra OPD terms, and direct verifier reward addition.

## Full text

On-policy distillation (OPD) trains a student on its own responses using dense token-level guidance from a stronger teacher. In long-context tasks, however, token-level teacher support can favor locally plausible responses that omit evidence distributed across the input or violate global task constraints. Task-specific verifiers, in contrast, evaluate task completion at the response level and may return graded rewards that reflect partial success. We diagnose this mismatch on fixed responses from two representative long-context evidence-aggregation tasks. Across longer input ranges, trajectory-level OPD scores become progressively less aligned with verifier rewards, indicating teacher-verifier disagreement. Motivated by this observation, we introduce Group-Calibrated On-Policy Distillation (GC-OPD). GC-OPD separately normalizes verifier rewards and trajectory-level OPD scores within each rollout group and uses their difference as a signed teacher-verifier disagreement residual. Relative-advantage-based credit assignment (RACA) distributes this trajectory-level residual across tokens according to their relative OPD advantages while preserving the original OPD signal. Across five long-context benchmarks, post-training with GC-OPD raises the five-benchmark averages of the official Qwen3-4B and Qwen3-8B checkpoints from 29.08 to 40.47 and from 35.12 to 44.65, respectively. Vanilla OPD reaches 39.31 and 43.56 under the same setup. Controlled ablations show that the signed residual is more effective than either an additional OPD-derived term or direct group-normalized verifier reward addition, while RACA further improves over uniform token allocation. Together, these results demonstrate that group-relative residual calibration can incorporate verifier outcomes without discarding dense token-level guidance. Code is available at https://github.com/SolereZhang/GC-OPD.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_15f045bd4dd0b79fcb8caa2f