---
title: "SCOPE: Selective Critical-Step On-Policy Self-Distillation for Agentic Reinforcement Learning"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2610.38391v1"
permalink: "https://feed.astralor.com/items/itm_5999d01c15e7fd927c03ea8d"
published: "2026-10-28T07:29:23.000Z"
score: 55
channel: "tech"
tags: ["LLM","AI Agent","推理效率","RAG","多模态","SCOPE","Selective Critical-Step On-Policy","Self-Distillation"]
---

# SCOPE: Selective Critical-Step On-Policy Self-Distillation for Agentic Reinforcement Learning

## 速览

Outcome-based reinforcement learning provides coarse supervision for long-horizon LLM agents: failed trajectories may contain correct steps, while successful trajectories can contain errors. Trajectory-level skills can …

## 关键点

- Outcome-based reinforcement learning provides coarse supervision for long-horizon LLM agents: failed trajectories may contain correct steps, while successful trajectories can contain errors.
- Trajectory-level skills can also shift teacher preferences at unrelated steps, introducing off-target distillation signals.
- We introduce SCOPE, a framework for Selective Critical-Step On-Policy Self-Distillation.

## 全文

Outcome-based reinforcement learning provides coarse supervision for long-horizon LLM agents: failed trajectories may contain correct steps, while successful trajectories can contain errors. Trajectory-level skills can also shift teacher preferences at unrelated steps, introducing off-target distillation signals. We introduce SCOPE, a framework for Selective Critical-Step On-Policy Self-Distillation. After a supervised warm-up, the policy analyzes its own trajectories, generates step-specific hindsight skills, and distills their guidance only at selected error steps. A bounded signed teacher-student coefficient stabilizes auxiliary updates while preserving positive reinforcement and negative correction. A success-relative pairwise length reward further encourages efficient task completion. Experiments on ALFWorld and WebShop demonstrate consistent improvements across multiple model backbones. With Qwen2.5-3B, SCOPE achieves 97.7% success on ALFWorld and 85.2% on WebShop, outperforming SEED by 5.9 and 6.3 percentage points, respectively. On unseen ALFWorld tasks, SCOPE reaches 89.6% success, improving over GRPO by 23.2 percentage points with 46.7% fewer interactions. Scaling to Qwen3-32B, SCOPE achieves 99.2% success on seen ALFWorld tasks and 94.0% on unseen tasks.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_5999d01c15e7fd927c03ea8d