---
title: "直接在线策略蒸馏：通过弱模型强化学习提升强模型推理能力"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.05394v1"
permalink: "https://feed.astralor.com/items/itm_d7fc2c3c037841f3c6dd9500"
published: "2026-07-06T17:59:58.000Z"
score: 81
channel: "ai"
tags: ["Direct-OPD","Qwen","弱到强泛化","强化学习","知识蒸馏"]
---

# 直接在线策略蒸馏：通过弱模型强化学习提升强模型推理能力

## AI 点评

> 这篇论文让我眼前一亮——它巧妙地将强化学习的策略变化当作信号而非模型蒸馏，成本低且效果好。对正在优化大模型后训练流程的团队来说，这篇值得细读；尤其适合那些资源有限但想快速提升模型推理能力的场景。

## 速览

本文提出 Direct-OPD 方法，解决强语言模型后训练成本高的问题。通过在小模型上运行强化学习，将其策略变化作为隐式奖励信号传输给大模型，无需显式奖励模型或在大模型上运行稀疏奖励强化学习。实验表明，该方法在 AIME 2024 上仅用 8 张 A100 GPU 训练 4 小时，将 Qwen3-1.7B 从 48.3% 提升至 62.4%，并优于直接蒸馏和直接强化学习。

## 关键点

- 提出 Direct-OPD 方法，利用弱教师模型的强化学习策略变化作为隐式奖励信号。
- 无需在强目标模型上运行代价高昂的强化学习，仅需弱模型的两次前向传播。
- 在 AIME 2024 上显著提升 Qwen3-1.7B 性能，并支持多个策略变化的顺序组合。

## 全文

Reinforcement learning with verifiable rewards (RLVR) is a powerful recipe for improving language-model reasoning, but it is expensive to repeat on every new strong model because the target model must generate many rollouts during training. As models scale, post-training itself becomes a bottleneck. We study a weak-to-strong alternative: run RL on a smaller model where rollouts are cheaper, then reuse what that RL run learned to improve a stronger target model. Directly distilling the post-RL weak teacher is not enough, because the teacher's final policy mixes useful RL gains with the limitations of the smaller model. We propose Direct On-Policy Distillation (Direct-OPD), which transfers the teacher's RL-induced policy shift instead. Direct-OPD compares the post-RL teacher with its own pre-RL reference and treats their log-ratio as a dense implicit reward for the student. In plain terms, the checkpoint pair tells us which actions RL made the weak model more or less likely to take, and Direct-OPD applies that signal on the stronger student's own on-policy states. This directly reuses the weak model's RL supervision signal without training an explicit reward model or running sparse-reward RL on the target model. Empirically, Direct-OPD consistently leverages weaker teachers to improve stronger target models; notably, it boosts Qwen3-1.7B from 48.3% to 62.4% on AIME 2024 in just 4 hours on 8 A100 GPUs. It outperforms step-matched direct RL and enables the sequential composition of multiple policy shifts. Our results show that RL outcomes can be reused across model scales as implicit reward signals, not merely as final models to imitate.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_d7fc2c3c037841f3c6dd9500