---
title: "SCOPE: Selective Critical-Step On-Policy Self-Distillation for Agentic Reinforcement Learning"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2610.38391v1"
permalink: "https://feed.astralor.com/items/itm_5999d01c15e7fd927c03ea8d"
published: "2026-10-28T07:29:23.000Z"
score: 55
channel: "tech"
tags: ["LLM","AI Agent","推理效率","RAG","多模态","SCOPE","Selective Critical-Step On-Policy","Self-Distillation"]
---

# SCOPE: Selective Critical-Step On-Policy Self-Distillation for Agentic Reinforcement Learning

## At a glance

Outcome-based reinforcement learning provides coarse supervision for long-horizon LLM agents: failed trajectories may contain correct steps, while successful trajectories can contain errors. Trajectory-level skills can …

## Key points

- Outcome-based reinforcement learning provides coarse supervision for long-horizon LLM agents: failed trajectories may contain correct steps, while successful trajectories can contain errors.
- Trajectory-level skills can also shift teacher preferences at unrelated steps, introducing off-target distillation signals.
- We introduce SCOPE, a framework for Selective Critical-Step On-Policy Self-Distillation.

## Full text

Outcome-based reinforcement learning provides coarse supervision for long-horizon LLM agents: failed trajectories may contain correct steps, while successful trajectories can contain errors. Trajectory-level skills can also shift teacher preferences at unrelated steps, introducing off-target distillation signals. We introduce SCOPE, a framework for Selective Critical-Step On-Policy Self-Distillation. After a supervised warm-up, the policy analyzes its own trajectories, generates step-specific hindsight skills, and distills their guidance only at selected error steps. A bounded signed teacher-student coefficient stabilizes auxiliary updates while preserving positive reinforcement and negative correction. A success-relative pairwise length reward further encourages efficient task completion. Experiments on ALFWorld and WebShop demonstrate consistent improvements across multiple model backbones. With Qwen2.5-3B, SCOPE achieves 97.7% success on ALFWorld and 85.2% on WebShop, outperforming SEED by 5.9 and 6.3 percentage points, respectively. On unseen ALFWorld tasks, SCOPE reaches 89.6% success, improving over GRPO by 23.2 percentage points with 46.7% fewer interactions. Scaling to Qwen3-32B, SCOPE achieves 99.2% success on seen ALFWorld tasks and 94.0% on unseen tasks.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_5999d01c15e7fd927c03ea8d