---
title: "Direct On-Policy Distillation: Boosting Strong Model Reasoning via Weak Model RL"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.05394v1"
permalink: "https://feed.astralor.com/items/itm_d7fc2c3c037841f3c6dd9500"
published: "2026-07-06T17:59:58.000Z"
score: 81
channel: "ai"
tags: ["Direct-OPD","Qwen","Weak-to-strong generalization","Reinforcement Learning","Knowledge Distillation"]
---

# Direct On-Policy Distillation: Boosting Strong Model Reasoning via Weak Model RL

## AI commentary

> This paper caught my eye—it cleverly uses RL policy shift as a signal rather than distilling the final model, achieving low cost and high performance. It's a must-read for teams optimizing post-training pipelines, especially those with limited resources aiming to quickly boost reasoning capabilities.

## At a glance

This paper proposes Direct-OPD to address the high cost of post-training large language models. By running RL on a smaller model and transferring its policy shift as an implicit reward to a larger model, it avoids explicit reward models or sparse-reward RL on the target model. Experiments show it boosts Qwen3-1.7B from 48.3% to 62.4% on AIME 2024 in just 4 hours on 8 A100 GPUs, outperforming direct distillation and direct RL.

## Key points

- Proposes Direct-OPD that reuses weak teacher's RL policy shift as implicit reward.
- Eliminates expensive RL on strong target model, requiring only two forward passes of weak model.
- Achieves significant improvement on Qwen3-1.7B on AIME 2024, enables sequential composition of multiple policy shifts.

## Full text

Reinforcement learning with verifiable rewards (RLVR) is a powerful recipe for improving language-model reasoning, but it is expensive to repeat on every new strong model because the target model must generate many rollouts during training. As models scale, post-training itself becomes a bottleneck. We study a weak-to-strong alternative: run RL on a smaller model where rollouts are cheaper, then reuse what that RL run learned to improve a stronger target model. Directly distilling the post-RL weak teacher is not enough, because the teacher's final policy mixes useful RL gains with the limitations of the smaller model. We propose Direct On-Policy Distillation (Direct-OPD), which transfers the teacher's RL-induced policy shift instead. Direct-OPD compares the post-RL teacher with its own pre-RL reference and treats their log-ratio as a dense implicit reward for the student. In plain terms, the checkpoint pair tells us which actions RL made the weak model more or less likely to take, and Direct-OPD applies that signal on the stronger student's own on-policy states. This directly reuses the weak model's RL supervision signal without training an explicit reward model or running sparse-reward RL on the target model. Empirically, Direct-OPD consistently leverages weaker teachers to improve stronger target models; notably, it boosts Qwen3-1.7B from 48.3% to 62.4% on AIME 2024 in just 4 hours on 8 A100 GPUs. It outperforms step-matched direct RL and enables the sequential composition of multiple policy shifts. Our results show that RL outcomes can be reused across model scales as implicit reward signals, not merely as final models to imitate.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_d7fc2c3c037841f3c6dd9500