---
title: "U-OPSD: Unsupervised On-Policy Self-Distillation without Any Supervision"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.06296v1"
permalink: "https://feed.astralor.com/items/itm_40159f8f97e3f0fcd02847a0"
published: "2026-08-06T17:18:23.000Z"
score: 76
channel: "ai"
tags: ["Qwen3","Large Language Models","Reinforcement Learning","Mathematical Reasoning","Self-Distillation"]
---

# U-OPSD: Unsupervised On-Policy Self-Distillation without Any Supervision

## AI commentary

> I am very optimistic about this direction because post-training is often seen as 'alchemy', yet U-OPSD brings math reasoning to supervised levels with pure self-supervision, meaning teams without high-quality labels can follow suit. For those working on LLM post-training and inference optimization, this paper is worth a close read, especially the detail of using only the longest incorrect completion for correction, which is quite insightful.

## At a glance

This paper introduces U-OPSD, an on-policy self-distillation method that requires no external supervision. It samples multiple rollouts from the model, constructs a pseudo-solution via majority voting under a self-consistency threshold, then conditions a teacher distribution on the shortest pseudo-solution and distills it into prefixes of the model's longest incorrect completion, enabling precise self-correction. Across benchmarks such as AIME and MATH500, U-OPSD consistently improves over base models and matches or surpasses supervised methods with ground truth (e.g., OPSD and GRPO), achieving notable gains on Qwen3 4B and 8B scales.

## Key points

- U-OPSD uses only the model's own generations, building pseudo-labels via majority voting and a self-consistency threshold, eliminating external supervision.
- It conditions a teacher distribution on the shortest pseudo-solution and distills it into prefixes of the longest incorrect completion, enabling precise self-correction.
- On Qwen3 non-thinking mode at 4B/8B, it improves over the base model by 8.5% and 10.7%, and outperforms OPSD by 3.2% and 2.3% on average.
- In thinking mode, U-OPSD surpasses OPSD by 0.9% at 4B, matches it at 8B, and outperforms GRPO by 0.7% and 1.1%, respectively.

## Full text

On-policy (Self-)Distillation (OPD / OPSD) has shown strong potential for post-training large language models (LLMs). However, existing methods still rely heavily on external supervision, including ground-truth signals, environmental feedback, or guidance from larger models, and therefore fall short of genuine "self"-distillation. In this study, we show that on-policy self-distillation can be achieved using only a model's own generations via internal consistency. We propose Unsupervised On-Policy Self-Distillation (U-OPSD). U-OPSD first samples multiple rollouts and constructs a pseudo-solution by majority vote under a self-consistency threshold. It then conditions a teacher distribution on the shortest pseudo-solution and distills it into prefixes of the model's longest incorrect completion, allowing the model to correct itself precisely where it is confidently wrong. Across diverse benchmarks, base models, and training settings, U-OPSD consistently improves over the base models and matches or surpasses supervised methods with ground truth (GT), such as OPSD and GRPO. On AIME24, AIME25, HMMT25, MATH500, and AMC23, U-OPSD improves over the base model by 8.5% and 10.7% on Qwen3 non-thinking mode at the 4B and 8B scales, respectively, and outperforms OPSD by an average of 3.2% and 2.3%. In thinking mode, U-OPSD remains on par with OPSD, outperforming it by 0.9% at 4B and matching it at 8B, while surpassing GRPO by 0.7% and 1.1%, respectively.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_40159f8f97e3f0fcd02847a0