---
title: "TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.13988v1"
permalink: "https://feed.astralor.com/items/itm_b6369462d5f9d19bf5733f8e"
published: "2026-07-15T16:16:42.000Z"
score: 81
channel: "ai"
tags: ["BrowseComp-Plus","Qwen3","TRACE","Credit Assignment","Reinforcement Learning","Long-Horizon Agent"]
---

# TRACE: Turn-level Reward Assignment via Credit Estimation for Long-Horizon Agents

## AI commentary

> This paper proposes TRACE, an elegant credit assignment method that constructs state values from log-probability ratios, avoiding extra critics. The experimental gains are striking. For teams working on agent RL training, especially with long trajectories, this is a must-read.

## At a glance

Long-horizon multi-turn agents require extensive tool calls, making outcome rewards sparse, high-variance, and misleading. This paper proposes TRACE, a dense credit assignment method that converts gold-answer log-probabilities from a frozen reference model into state values and derives per-action rewards via temporal-difference changes, requiring no critic or process labels. On BrowseComp-Plus, TRACE boosts Qwen3-4B from 7.2 to 35.6 and Qwen3-30B-A3B from 8.4 to 42.6, using pure RL without cold-start SFT or web data, with earlier improvement and faster convergence.

## Key points

- TRACE builds state values from gold-answer log-probabilities of a frozen reference model and derives per-action rewards via TD learning, without extra critic or process labels.
- On the closed-web BrowseComp-Plus benchmark, TRACE raises Qwen3-4B from 7.2 to 35.6 and Qwen3-30B-A3B from 8.4 to 42.6.
- TRACE uses pure RL with no cold-start SFT or live-web data; learning curves show earlier improvement and faster convergence.
- Learned search behavior transfers to open-web benchmarks, demonstrating generalization.

## Full text

Multi-turn agents solve complex tasks through extended sequences of tool interactions before producing a final answer, making credit assignment a fundamental challenge during post-training. Outcome rewards provide reliable supervision for short-horizon reasoning, but become sparse and high-variance as trajectories grow to tens or hundreds of tool calls. They can also be misleading: a failed rollout may contain many useful actions that move the agent closer to the goal, yet outcome-only training assigns them the same negative advantage as the eventual mistake. We propose TRACE (Turn-level Reward Assignment via Credit Estimation), a dense credit-assignment method for agentic reinforcement learning. TRACE represents rollouts as state transitions at tool-call boundaries, obtains gold-answer log-probabilities from a frozen reference model, transforms them into log-ratio state values, and derives per-action rewards as Temporal-Difference changes in those values. This requires no additional critic or process-label training, and its one-step log-ratio TD component telescopes across redundant tool calls. On long-horizon complex search, TRACE substantially improves base-model tool-use ability using pure RL, without a cold-start supervised fine-tuning stage, an agentic mid-training stage, or training on live-web data. On the closed-web BrowseComp-Plus benchmark, it raises Qwen3-4B from $7.2$ to $35.6$ and Qwen3-30B-A3B from $8.4$ to $42.6$. The learned search behavior also transfers to open-web benchmarks, and the learning curves show earlier improvement and faster convergence during RL training.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_b6369462d5f9d19bf5733f8e