---
title: "超越最终分数：长时程AI研发Agent的系统性评估"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.13417v1"
permalink: "https://feed.astralor.com/items/itm_dc9304b44a7776d06b131c3d"
published: "2026-08-13T16:11:22.000Z"
score: 76
channel: "ai"
tags: ["AI Agent","arXiv","经验复用","过程指标","长期任务评估"]
---

# 超越最终分数：长时程AI研发Agent的系统性评估

## AI 点评

> 这篇论文值得所有关心AI自主科研能力的人细读：它没有停留在“智能体能不能拿高分”的粗粒度判断，而是把过程拆开了看。你会发现，很多看似聪明的结果其实来自经验复用和框架兜底，真正从头设计新方法的智体少之又少。对做Agent训练或评估框架的人来說，这里的分析能帮你避开“分数陷阱”。

## 速览

本文提出一套新框架，用基于规则的指标刻画智能体在长时程任务中的解题框架、执行与反馈控制过程，并对7个前沿模型在36个任务上的表现进行受控对比，考察经验复用效果。结果显示当前智能体更像工程优化器而非完全自主的研究者：能提出并实现实用方案，但跨运行表现波动大，最强方案多依赖组合现有技术，真正方法论创新罕见。分析还发现相似最终结果背后存在不同过程瓶颈，经验复用既可能帮助也可能误导后续决策，提示模型训练、推理策略、经验管理与框架设计均有改进空间。

## 关键点

- 提出包含Solution Framing、Execution、Feedback Control三阶段的评估框架，以规则化指标深入分析智能体运行过程而非只看最终分数
- 对7个前沿模型在36个长时程研发任务上进行受控对比，发现当前智能体表现更接近工程优化器而非自主研究者
- 经验复用既可提升也可误导后续决策，框架设计显著影响性能稳定性
- 识别出相似最终结果背后的不同过程瓶颈，为模型训练与推理策略改进提供具体方向

## 全文

Autonomous agents are increasingly capable of improving models, systems, and other technical artifacts through long-horizon experimentation. To understand the current state of this capability, however, evaluation must go beyond final scores, which neither reveal where progress is gained or lost nor indicate whether accumulated experience improves later decisions. We therefore present a systematic evaluation of seven frontier models on 36 long-horizon tasks based on a new framework that uses rule-based metrics to characterize within-run behavior through Solution Framing, Execution, and Feedback Control and controlled comparisons to assess experience reuse within and across tasks. The results show that current agents operate more like engineering optimizers than fully autonomous researchers: they can formulate and implement practical solutions, but their performance varies substantially across runs, their strongest solutions mainly adapt or combine established techniques, and genuine methodological novelty remains rare. Detailed analysis reveals that observed performance is shaped by multiple factors, including distinct process bottlenecks behind similar final outcomes, experience reuse that can help or mislead subsequent decisions, and harness designs that affect performance stability. These findings suggest concrete directions for improving model training, inference-time strategies, experience management, and harness design.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_dc9304b44a7776d06b131c3d