---
title: "TRACE：通过信用估计实现长程Agent的轮次级别奖励分配"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.13988v1"
permalink: "https://feed.astralor.com/items/itm_b6369462d5f9d19bf5733f8e"
published: "2026-07-15T16:16:42.000Z"
score: 81
channel: "ai"
tags: ["BrowseComp-Plus","Qwen3","TRACE","信用分配","强化学习","长程Agent"]
---

# TRACE：通过信用估计实现长程Agent的轮次级别奖励分配

## AI 点评

> 这篇论文提出的TRACE方法在信用分配上非常巧妙，用对数概率比直接构建状态值，避免了额外评论家，实验提升惊人。对于正在做Agent RL训练的团队，这是必须关注的工作，尤其适合长轨迹场景。

## 速览

长程多轮Agent在执行复杂任务时需要大量工具调用，导致结果奖励稀疏、方差大且误导性。本文提出TRACE，一种密集信用分配方法，利用冻结参考模型的金标准对数概率转化为状态值，并通过时序差分推导每步奖励，无需额外的评论家或过程标签。在BrowseComp-Plus基准上，TRACE将Qwen3-4B从7.2提升至35.6，Qwen3-30B-A3B从8.4提升至42.6，且纯强化学习无需冷启动监督微调或网络数据训练，学习曲线显示更早改进和更快收敛。

## 关键点

- TRACE通过冻结参考模型的金标准对数概率构建状态价值，并利用时序差分推导每步奖励，无需额外训练评论家或过程标签。
- 在长程复杂搜索的闭环基准BrowseComp-Plus上，TRACE使Qwen3-4B准确率从7.2提升至35.6，Qwen3-30B-A3B从8.4提升至42.6。
- TRACE仅使用纯强化学习，无需冷启动监督微调或在线网络数据训练，学习曲线显示更早提升和更快收敛。
- 学到的搜索行为可迁移至开放网络基准，表明方法的泛化能力。

## 全文

Multi-turn agents solve complex tasks through extended sequences of tool interactions before producing a final answer, making credit assignment a fundamental challenge during post-training. Outcome rewards provide reliable supervision for short-horizon reasoning, but become sparse and high-variance as trajectories grow to tens or hundreds of tool calls. They can also be misleading: a failed rollout may contain many useful actions that move the agent closer to the goal, yet outcome-only training assigns them the same negative advantage as the eventual mistake. We propose TRACE (Turn-level Reward Assignment via Credit Estimation), a dense credit-assignment method for agentic reinforcement learning. TRACE represents rollouts as state transitions at tool-call boundaries, obtains gold-answer log-probabilities from a frozen reference model, transforms them into log-ratio state values, and derives per-action rewards as Temporal-Difference changes in those values. This requires no additional critic or process-label training, and its one-step log-ratio TD component telescopes across redundant tool calls. On long-horizon complex search, TRACE substantially improves base-model tool-use ability using pure RL, without a cold-start supervised fine-tuning stage, an agentic mid-training stage, or training on live-web data. On the closed-web BrowseComp-Plus benchmark, it raises Qwen3-4B from $7.2$ to $35.6$ and Qwen3-30B-A3B from $8.4$ to $42.6$. The learned search behavior also transfers to open-web benchmarks, and the learning curves show earlier improvement and faster convergence during RL training.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_b6369462d5f9d19bf5733f8e