---
title: "Visual Contrastive Self-Distillation: On-Policy Self-Distillation Without External Teacher, Privileged Answers, or Visual Evidence"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.21556v1"
permalink: "https://feed.astralor.com/items/itm_dd2199f2d68391496fa993af"
published: "2026-07-23T17:37:37.000Z"
score: 78
channel: "ai"
tags: ["Qwen3-VL","ViRL39K","MLLMs","Self-Distillation","Visual Contrastive Learning"]
---

# Visual Contrastive Self-Distillation: On-Policy Self-Distillation Without External Teacher, Privileged Answers, or Visual Evidence

## AI commentary

> This paper cleverly turns input conditioning differences into a distillation signal, completely removing the need for external teachers and privileged answers in traditional self-distillation, making it 'purer'. For engineers and researchers training multimodal models, VCSD offers a plug-and-play, effective training trick—especially suitable for teams with limited compute who want to improve visual understanding quality. I recommend focusing on the practical impact of its contrastive sharpening mechanism.

## At a glance

This paper proposes Visual Contrastive Self-Distillation (VCSD), a simplified on-policy self-distillation method that generates asymmetric signals purely through input conditioning. An EMA teacher produces two next-token distributions under the same prompt and prefix—one conditioned on the original image and the other on a content-erased control. The token-wise log-probability difference highlights candidates specifically boosted by visual content, used to sharpen the teacher's original-image distribution for student distillation. On the ViRL39K dataset, VCSD consistently outperforms matched OPSD baselines across Qwen3-VL and Qwen3.5 models; e.g., on Qwen3-VL 2B, the seven-benchmark aggregate rises from 62.27% to 67.04%. VCSD requires no external teacher, privileged answers, visual evidence signals, reasoning traces, or additional inference cost.

## Key points

- Proposes VCSD that creates asymmetric signals via input conditioning (original image vs. content-erased control), eliminating the need for external teachers or privileged answers.
- Uses an EMA teacher to generate two distributions and computes token-level log-probability differences to identify visual-specific gains, then sharpens the teacher's original image distribution.
- On the ViRL39K dataset, VCSD substantially improves Qwen3-VL and Qwen3.5 models across seven benchmarks, with gains up to 4.75 percentage points.
- Requires no additional inference cost, reasoning traces, or visual evidence signals, resulting in a simpler training pipeline.

## Full text

On-policy self-distillation (OPSD) is promising as it removes the external teacher required by on-policy distillation (OPD), yet it still needs asymmetric information between teacher and student to ensure that the self-teacher provides a stronger learning signal than the student. Existing methods create this asymmetry either through privileged answers or visual evidence. We ask whether both can be removed, yielding a simpler form of OPSD driven purely by input conditioning. For this purpose, we propose Visual Contrastive Self-Distillation, namely VCSD, which converts image-content removal into an on-policy self-distillation signal. At each student-generated response prefix, the EMA teacher produces two next-token distributions under the same prompt and prefix -- one conditioned on the original image and the other on a content-erased control. Their token-wise log-probability difference highlights candidates whose likelihood is specifically increased by the instance-level visual content. We use this contrast to sharpen the teacher's original-image distribution within its plausible support, and distill the resulting full-distribution target into the student. Using ViRL39K dataset, VCSD consistently outperforms matched OPSD across Qwen3-VL and Qwen3.5 models. For example, on Qwen3-VL, it improves the seven-benchmark aggregate from $62.27\% \rightarrow 67.04\%$ at 2B, $71.30\% \rightarrow 73.16\%$ at 4B, and $72.51\% \rightarrow 76.26\%$ at 8B. Furthermore, VCSD requires no external teacher, privileged answers, visual evidence signals, reasoning traces, or additional inference-time cost.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_dd2199f2d68391496fa993af