---
title: "无需任何监督的在线策略自蒸馏方法 U-OPSD"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.06296v1"
permalink: "https://feed.astralor.com/items/itm_40159f8f97e3f0fcd02847a0"
published: "2026-08-06T17:18:23.000Z"
score: 76
channel: "ai"
tags: ["Qwen3","大语言模型","强化学习","数学推理","自蒸馏"]
---

# 无需任何监督的在线策略自蒸馏方法 U-OPSD

## AI 点评

> 我非常看好这个方向，因为后训练常被视为“炼丹”，而 U-OPSD 用纯自监督就把数学推理拉到监督方法水平，这意味着很多没有高质量标注的团队也能跟进。对做 LLM 后训练和推理优化的人来说，这篇值得精读，尤其是它只用最长错误补全做纠偏的细节，很有启发。

## 速览

论文提出 U-OPSD，一种不依赖任何外部监督的在线策略自蒸馏方法。它通过模型自身多次采样生成候选，利用多数投票和自一致性阈值构造伪解决方案，并基于最短伪解构造教师分布，将其蒸馏到模型最长错误补全的前缀部分，实现模型的自我纠错。实验表明，U-OPSD 在 AIME、MATH500 等基准上持续超过基座模型，并匹配或超越使用真实标签的监督方法（如 OPSD、GRPO），在 Qwen3 4B 和 8B 规模上均取得显著提升。

## 关键点

- U-OPSD 仅使用模型自身生成内容，通过多数投票与自一致性阈值构建伪标签，完全摆脱外部监督。
- 通过将教师分布集中于最短伪解，并将其蒸馏到最长错误补全的前缀，实现精准自我纠错。
- 在 Qwen3 非思考模式 4B/8B 上，相比基座模型平均提升 8.5% 和 10.7%，并优于 OPSD 平均 3.2% 和 2.3%。
- 在思考模式下，U-OPSD 在 4B 上超过 OPSD 0.9%，在 8B 上与 OPSD 持平，并超越 GRPO。

## 全文

On-policy (Self-)Distillation (OPD / OPSD) has shown strong potential for post-training large language models (LLMs). However, existing methods still rely heavily on external supervision, including ground-truth signals, environmental feedback, or guidance from larger models, and therefore fall short of genuine "self"-distillation. In this study, we show that on-policy self-distillation can be achieved using only a model's own generations via internal consistency. We propose Unsupervised On-Policy Self-Distillation (U-OPSD). U-OPSD first samples multiple rollouts and constructs a pseudo-solution by majority vote under a self-consistency threshold. It then conditions a teacher distribution on the shortest pseudo-solution and distills it into prefixes of the model's longest incorrect completion, allowing the model to correct itself precisely where it is confidently wrong. Across diverse benchmarks, base models, and training settings, U-OPSD consistently improves over the base models and matches or surpasses supervised methods with ground truth (GT), such as OPSD and GRPO. On AIME24, AIME25, HMMT25, MATH500, and AMC23, U-OPSD improves over the base model by 8.5% and 10.7% on Qwen3 non-thinking mode at the 4B and 8B scales, respectively, and outperforms OPSD by an average of 3.2% and 2.3%. In thinking mode, U-OPSD remains on par with OPSD, outperforming it by 0.9% at 4B and matching it at 8B, while surpassing GRPO by 0.7% and 1.1%, respectively.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_40159f8f97e3f0fcd02847a0