---
title: "Messier：面向跨基准智能体评估的高分辨率语料库"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.25891v1"
permalink: "https://feed.astralor.com/items/itm_6fd85b9f9c79e49d0a703990"
published: "2026-07-28T15:50:19.000Z"
score: 81
channel: "ai"
tags: ["Epoch AI","Messier","SWE-bench","基准统一","智能体评估","能力标定"]
---

# Messier：面向跨基准智能体评估的高分辨率语料库

## AI 点评

> 这篇论文解决了我一直头疼的问题：不同论文里的Agent分数根本没法比。现在有了Messier，终于可以像看高考分数线一样横向对比了。对做Agent评测框架、想了解各领域能力天花板的团队尤其值得细读。

## 速览

Messier 提供了一个包含957,253条记录的统一语料库，覆盖30个基准、714个智能体、11,891个任务和74,205个验证器。通过标准化模型、框架、环境等维度，揭示了前沿进展在各类基准上的不均衡性：函数调用已饱和，编程进步最快，企业工作流最具挑战。严格的全通过聚合规则可能掩盖进展并改变排名。该语料库可用于智能体能力标定、基准审计和失败分析，与Epoch能力指数排名相关系数达0.81。

## 关键点

- 构建了包含近百万记录的高分辨率语料库，统一了30个基准的评估数据
- 发现前沿进展不均衡：函数调用饱和、编程进步最快、企业工作流最具挑战
- 指出严格全通过聚合规则可能掩盖进展并人为改变智能体排名
- 提供了可复用的基础设施，用于能力标定、基准审计和失败分析

## 全文

Evaluating AI agents in interactive environments is hindered by fragmented tasks, scaffolds, verifiers, and scoring rules. Existing efforts focus on narrow settings, remain limited in scale, or require costly reruns, leaving much of the empirical record incomparable. We introduce Messier, a unified corpus of 957,253 records that span 30 benchmarks, 714 agents, 11,891 tasks, and 74,205 verifiers. Messier consolidates public benchmark scores and supplements them with five-agent runs across six underrepresented professional and scientific domains, including a recent legal benchmark. Each record is standardized by model, scaffold, environment, task, verifier, and aggregation rule, with SOC/NAICS classifications for occupational and industry analysis. Using this corpus, we show frontier progress is uneven across benchmark types, with "function calling" saturated, "programming" improving the fastest, and "enterprise workflows" remaining the most challenging. Furthermore, counterfactual rescoring shows that strict all-pass aggregation in multi-verifier tasks can obscure progress and artificially alter agent rankings. From these standardized records, we derive capability scales that align with Epoch's Evaluation Capability Index rankings at Spearman \r{ho} = 0.81 and can be specialized by domain, occupation, action space, or verifier type. Messier provides a foundational, reusable infrastructure for agent capability scaling, benchmark auditing, and fine-grained analysis of evaluation failures.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_6fd85b9f9c79e49d0a703990