---
title: "SCOPE-RL：在成功前后优化推理路径的强化学习框架"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.11506v1"
permalink: "https://feed.astralor.com/items/itm_18835a97f5357ad9f880ccc4"
published: "2026-07-13T12:56:28.000Z"
score: 77
channel: "tech"
tags: ["GRPO","Qwen3","SCOPE-RL","奖励密化","强化学习","推理路径优化"]
---

# SCOPE-RL：在成功前后优化推理路径的强化学习框架

## AI 点评

> 这是一篇非常实用的论文，尤其对于正在用RLVR训练LLM的团队。它巧妙地通过过程奖励密化解决了最终奖励稀疏的问题，并且实验证明准确率和token效率双提升。如果你关注推理模型或数学推理任务，值得一读。

## 速览

强化学习中的可验证奖励（RLVR）仅依赖最终答案的稀疏奖励，缺乏对推理过程的直接反馈。为此，本文提出SCOPE-RL两阶段框架：成功前通过自适应脚手架奖励密化，成功后通过质量感知过程奖励优化。在Qwen3-8B-Instruct上，该框架使准确率最高提升11.2个百分点，推理token减少27.1%，同时保持了泛化能力。

## 关键点

- 提出SCOPE-RL两阶段框架，在成功前通过前缀分解奖励密化，成功后通过过程形状奖励优化。
- 在Qwen3-8B-Instruct和Qwen3-0.6B-Instruct上验证，准确率提升最高11.2个百分点，token减少27.1%。
- 引入专家验证的步骤质量评估协议，衡量有用步骤密度、错误定位和token效率。
- 代码和数据已开源。

## 全文

Reinforcement learning with verifiable rewards (RLVR) optimizes LLMs using sparse verifiable final-answer rewards. This sparse anchor reliably verifies whether a trajectory succeeds but provides no direct feedback on the reasoning path that produced it. Before success, prerequisite progress on hard problems receives no reward signal; after success, outcome rewards cannot distinguish well-organized correct trajectories from redundant or locally flawed ones. We introduce SCOPE-RL (Scaffolded Chain Optimization with Process Efficiency), a two-stage framework that densifies this anchor while retaining the GRPO update: Adaptive Scaffolded RL adds prefix-decomposed verifiable rewards on answer-hidden sub-question chains before success, and Quality-Aware Process RL applies correctness-gated process-shape rewards to refine correct trajectories after success. An expert-validated Step-Quality Evaluation Protocol evaluates useful-step density, error localization, and token efficiency beyond final-answer accuracy. On Qwen3-8B-Instruct trained on DAPO-Math and Big-Math, SCOPE-RL improves average accuracy by up to 11.2 pp and reduces reasoning tokens by up to 27.1% over outcome-only GRPO; the gains hold under GSPO and on Qwen3-0.6B-Instruct, indicating that reward-signal densification is complementary to policy-update-level RLVR advances. Code and data are available at https://github.com/tokencraft-lab/SCOPE-RL.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_18835a97f5357ad9f880ccc4