---
title: "一个症状，三个杠杆：同策略自蒸馏的批判性综述"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.25936v1"
permalink: "https://feed.astralor.com/items/itm_b9c96b990b8cc95e6105d2f4"
published: "2026-08-26T15:52:19.000Z"
score: 73
channel: "ai"
tags: ["OPSD","数学推理","自蒸馏","语言模型"]
---

# 一个症状，三个杠杆：同策略自蒸馏的批判性综述

## AI 点评

> 这篇综述把OPSD崩溃从零散现象提升为可调控的系统性杠杆问题，对正在用自蒸馏做推理模型的研究者尤其值得一读。它没有给出新实验，但用一个统一词汇表帮你快速看懂这个领域里的“同病异名”。如果你想改进自己的蒸馏流程，可以从“信号在哪、教师看到什么、指导何时衰减”三个问题入手。

## 速览

这篇综述批判性审视同策略自蒸馏（OPSD）。OPSD让语言模型用自己的生成内容训练，由模型自身借助特权信息逐token打分，省去额外教师模型。早期结果准确率接近强化学习，且生成token更少。但主导失败模式是崩溃：推理路径多样性逐渐收窄，特权信息会加剧。文章将崩溃视为症状，由三个杠杆控制：信号应用位置、教师所见特权信息、信号变化时机。综述聚焦数学推理，未报告新实验，主要贡献是统一术语并划分共识与争议。

## 关键点

- OPSD利用模型自身作为教师，借助特权信息提供逐token监督，省去额外大模型成本。
- 早期研究表明OPSD在数学推理上能以更少的生成token达到接近RL的准确率。
- 当前主导失败模式是“崩溃”，即推理路径多样性不断收窄，特权信息会加剧该问题。
- 本文提出三个杠杆来理解崩溃：信号权重、特权信息性质、教师指导的衰减动态。

## 全文

On-policy distillation trains a language model on its own generations while a teacher scores them token by token. It combines the dense supervision of imitation learning with the on-policy sampling of reinforcement learning. But it requires a second, larger model to act as teacher. On-Policy Self-Distillation (OPSD) removes that cost. The teacher is the model itself, conditioned on privileged information the student will not have at test time, such as a reference solution, a plan, or environment feedback. The teacher is no stronger than the student, only better informed. Early results were promising, with accuracy comparable to reinforcement learning at a fraction of the generated tokens. But the same asymmetry that produces the signal also biases it. One failure mode now dominates the field: collapse, the progressive narrowing of the set of reasoning paths the model can produce. Collapse is not specific to OPSD, though privileged information aggravates it. This review treats collapse as a symptom governed by three levers: (i) where the signal is applied, that is, how tokens are weighted; (ii) what the teacher is shown, that is, the nature of the privileged information; and (iii) when the signal changes, that is, the teacher's dynamics and the decay of guidance. We restrict our scope to mathematical reasoning, where the method originated and where its failure modes are best documented. We report no new experiments. The contribution is structural: a shared vocabulary for phenomena named differently across papers, and a clear line between what is settled and what is still disputed.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_b9c96b990b8cc95e6105d2f4