---
title: "On-Policy Self-Distillation with Teacher Evolution"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2610.38637v1"
permalink: "https://feed.astralor.com/items/itm_28c23d0edfd5ae7ad7b98f43"
published: "2026-10-28T10:51:52.000Z"
score: 54
channel: "ai"
tags: ["多模态","On-Policy Self-Distillation","Teacher Evolution On-Policy","Self-Distillation","OPSD","However"]
---

# On-Policy Self-Distillation with Teacher Evolution

## 速览

On-Policy Self-Distillation (OPSD), which aligns a student model's token-level distribution with that of its self-teacher prompted by privileged context, has emerged as a promising direction to mitigate reward sparsity …

## 关键点

- On-Policy Self-Distillation (OPSD), which aligns a student model's token-level distribution with that of its self-teacher prompted by privileged context, has emerged as a promising direction to mitigate reward sparsity …
- However, OPSD often fails to achieve continuous performance gains during training.
- In this paper, we find that the model's limited self-teaching capability underlies this phenomenon by generating unreliable guidance on certain samples.

## 全文

On-Policy Self-Distillation (OPSD), which aligns a student model's token-level distribution with that of its self-teacher prompted by privileged context, has emerged as a promising direction to mitigate reward sparsity in sequence-level post-training paradigms. However, OPSD often fails to achieve continuous performance gains during training. In this paper, we find that the model's limited self-teaching capability underlies this phenomenon by generating unreliable guidance on certain samples. To address this problem, we propose Teacher-Evolution Self-Distillation (TESD), which enables the self-teacher to generate rollouts and receive environmental feedback based on task correctness and context independence. First, using these rewards as training objectives, we perform teacher-oriented optimization to continuously enhance the model's self-teaching capability during distillation. Second, by estimating teacher guidance reliability from these rewards, we apply adaptive student-oriented optimization to down-weight samples with unreliable guidance. Extensive experiments across multiple datasets and model scales show that TESD consistently outperforms representative token-level and sequence-level methods, highlighting the necessity of improving supervision quality in OPSD.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_28c23d0edfd5ae7ad7b98f43