---
title: "Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.13417v1"
permalink: "https://feed.astralor.com/items/itm_dc9304b44a7776d06b131c3d"
published: "2026-08-13T16:11:22.000Z"
score: 76
channel: "ai"
tags: ["AI Agent","arXiv","Experience Reuse","Process Metrics","Long-Horizon Task Evaluation"]
---

# Beyond Final Scores: A Systematic Evaluation of Agents for Long-Horizon AI Research and Development

## AI commentary

> This paper deserves close reading by anyone concerned with AI's autonomous research capabilities: it moves beyond coarse 'can agents score high' judgments and dissects the process. You'll find that many seemingly clever results come from experience reuse and harness support, while agents that genuinely design new methods from scratch are rare. For those working on agent training or evaluation frameworks, the analysis here can help you avoid the 'score trap'.

## At a glance

This paper presents a new framework using rule-based metrics to characterize within-run behavior through Solution Framing, Execution, and Feedback Control, and performs controlled comparisons of seven frontier models on 36 long-horizon tasks to assess experience reuse. Results show current agents operate more like engineering optimizers than fully autonomous researchers: they can formulate and implement practical solutions, but performance varies substantially across runs, their strongest solutions mainly adapt or combine established techniques, and genuine methodological novelty remains rare. Detailed analysis reveals distinct process bottlenecks behind similar final outcomes, experience reuse that can help or mislead, and harness designs that affect stability, suggesting concrete directions for improving model training, inference-time strategies, experience management, and harness design.

## Key points

- Proposes an evaluation framework with Solution Framing, Execution, and Feedback Control stages, using rule-based metrics to analyze agent behavior beyond final scores
- Controlled comparison of seven frontier models on 36 long-horizon R&D tasks finds current agents resemble engineering optimizers more than autonomous researchers
- Experience reuse can help or mislead subsequent decisions, and harness design significantly affects performance stability
- Identifies distinct process bottlenecks behind similar final outcomes, offering concrete directions for improving model training and inference strategies

## Full text

Autonomous agents are increasingly capable of improving models, systems, and other technical artifacts through long-horizon experimentation. To understand the current state of this capability, however, evaluation must go beyond final scores, which neither reveal where progress is gained or lost nor indicate whether accumulated experience improves later decisions. We therefore present a systematic evaluation of seven frontier models on 36 long-horizon tasks based on a new framework that uses rule-based metrics to characterize within-run behavior through Solution Framing, Execution, and Feedback Control and controlled comparisons to assess experience reuse within and across tasks. The results show that current agents operate more like engineering optimizers than fully autonomous researchers: they can formulate and implement practical solutions, but their performance varies substantially across runs, their strongest solutions mainly adapt or combine established techniques, and genuine methodological novelty remains rare. Detailed analysis reveals that observed performance is shaped by multiple factors, including distinct process bottlenecks behind similar final outcomes, experience reuse that can help or mislead subsequent decisions, and harness designs that affect performance stability. These findings suggest concrete directions for improving model training, inference-time strategies, experience management, and harness design.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_dc9304b44a7776d06b131c3d