---
title: "One Symptom, Three Levers: A Critical Review of On-Policy Self-Distillation"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.25936v1"
permalink: "https://feed.astralor.com/items/itm_b9c96b990b8cc95e6105d2f4"
published: "2026-08-26T15:52:19.000Z"
score: 73
channel: "ai"
tags: ["OPSD","Mathematical Reasoning","Self-Distillation","Language Model"]
---

# One Symptom, Three Levers: A Critical Review of On-Policy Self-Distillation

## AI commentary

> This review elevates OPSD collapse from scattered symptoms to a system of controllable levers, especially valuable for researchers using self-distillation for reasoning models. It reports no new experiments, but provides a shared vocabulary to quickly decode the 'same disease, different names' across papers. If you want to improve your distillation pipeline, start by asking where the signal is applied, what the teacher sees, and when guidance decays.

## At a glance

This review critically examines on-policy self-distillation (OPSD). OPSD trains a language model on its own generations, using the model itself as teacher with privileged information for token-level scoring, eliminating the need for a separate larger teacher. Early results showed accuracy comparable to reinforcement learning with far fewer generated tokens. However, the dominant failure mode is collapse: the progressive narrowing of reasoning paths, aggravated by privileged information. The paper treats collapse as a symptom governed by three levers: where the signal is applied, what the teacher is shown, and when the signal changes. Focusing on mathematical reasoning, the review reports no new experiments. Its main contribution is a shared vocabulary and a clear line between settled and disputed findings.

## Key points

- OPSD uses the model itself as teacher, leveraging privileged information for token-level supervision, eliminating the cost of an extra large model.
- Early results show OPSD can achieve RL-comparable accuracy on mathematical reasoning with far fewer generated tokens.
- The dominant failure mode is collapse: the progressive narrowing of reasoning path diversity, aggravated by privileged information.
- This paper proposes three levers to understand collapse: signal weighting, the nature of privileged information, and the dynamics of teacher guidance decay.

## Full text

On-policy distillation trains a language model on its own generations while a teacher scores them token by token. It combines the dense supervision of imitation learning with the on-policy sampling of reinforcement learning. But it requires a second, larger model to act as teacher. On-Policy Self-Distillation (OPSD) removes that cost. The teacher is the model itself, conditioned on privileged information the student will not have at test time, such as a reference solution, a plan, or environment feedback. The teacher is no stronger than the student, only better informed. Early results were promising, with accuracy comparable to reinforcement learning at a fraction of the generated tokens. But the same asymmetry that produces the signal also biases it. One failure mode now dominates the field: collapse, the progressive narrowing of the set of reasoning paths the model can produce. Collapse is not specific to OPSD, though privileged information aggravates it. This review treats collapse as a symptom governed by three levers: (i) where the signal is applied, that is, how tokens are weighted; (ii) what the teacher is shown, that is, the nature of the privileged information; and (iii) when the signal changes, that is, the teacher's dynamics and the decay of guidance. We restrict our scope to mathematical reasoning, where the method originated and where its failure modes are best documented. We report no new experiments. The contribution is structural: a shared vocabulary for phenomena named differently across papers, and a clear line between what is settled and what is still disputed.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_b9c96b990b8cc95e6105d2f4