---
title: "Task-Efficient Agent Evaluation Using Cached Traces"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2611.06803v1"
permalink: "https://feed.astralor.com/items/itm_962f0a8900e863da8d9ee4f6"
published: "2026-11-04T17:20:09.000Z"
score: 59
channel: "ai"
tags: ["LLM","AI Agent","RAG","Task-Efficient Agent Evaluation","Using Cached Traces","Efficient","Standard","Leveraging"]
---

# Task-Efficient Agent Evaluation Using Cached Traces

## 速览

Efficient benchmarking estimates a system's full benchmark performance from a small number of task executions. Standard approaches rely solely on the binary correctness of executions from a collection of already-evaluat…

## 关键点

- Efficient benchmarking estimates a system's full benchmark performance from a small number of task executions.
- Standard approaches rely solely on the binary correctness of executions from a collection of already-evaluated reference systems.
- An agentic system, however, produces a rich trace for each execution that can reveal substantially more about its capabilities.

## 全文

Efficient benchmarking estimates a system's full benchmark performance from a small number of task executions. Standard approaches rely solely on the binary correctness of executions from a collection of already-evaluated reference systems. An agentic system, however, produces a rich trace for each execution that can reveal substantially more about its capabilities. Leveraging this information requires useful trace representations. We propose representing a trace by extracting a structured summary of it with an LLM and embedding the summary with a standard text encoder. These representations preserve correctness- and task-related signal while minimizing the presence of system identity-, model family- and harness-related signals that dominate raw trace embeddings. Given this representation for each task execution from all reference systems, we construct low-dimensional representations of the systems and predict a new system's score from its position among the reference systems. On SWE-bench Verified and Terminal-Bench 2.0, augmenting an item-response model with this geometry-based prediction reduces mean absolute error from 0.13 to 0.08 (Wilcoxon $p < 10^{-4}$) and from 0.14 to 0.12 (Wilcoxon $p < 10^{-3}$), respectively, when provided a single random executed task. Further, given access to the full set of reference systems, our approach dominates baselines for any given number of executed tasks on both benchmarks.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_962f0a8900e863da8d9ee4f6