---
title: "Messier: A High-Resolution Corpus for Cross-Benchmark Agent Evaluation"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.25891v1"
permalink: "https://feed.astralor.com/items/itm_6fd85b9f9c79e49d0a703990"
published: "2026-07-28T15:50:19.000Z"
score: 81
channel: "ai"
tags: ["Epoch AI","Messier","SWE-bench","Benchmark Unification","Agent Evaluation","Capability Scaling"]
---

# Messier: A High-Resolution Corpus for Cross-Benchmark Agent Evaluation

## AI commentary

> This paper solves a headache I've had: agent scores across papers are incomparable. With Messier, we can now do cross-benchmark comparisons like standardized test scores. Especially valuable for teams building evaluation frameworks or tracking capability ceilings.

## At a glance

Messier presents a unified corpus of 957,253 records spanning 30 benchmarks, 714 agents, 11,891 tasks, and 74,205 verifiers. By standardizing across model, scaffold, environment, and other dimensions, it reveals uneven frontier progress: function calling is saturated, programming improves fastest, and enterprise workflows remain hardest. Strict all-pass aggregation can obscure progress and alter rankings. The corpus enables agent capability scaling, benchmark auditing, and failure analysis, achieving Spearman ρ=0.81 with Epoch's capability index.

## Key points

- Builds a high-resolution corpus with nearly 1M records, unifying evaluation data from 30 benchmarks.
- Reveals uneven frontier progress: function calling saturated, programming fastest, enterprise workflows hardest.
- Shows strict all-pass aggregation can obscure progress and artificially alter agent rankings.
- Provides reusable infrastructure for capability scaling, benchmark auditing, and failure analysis.

## Full text

Evaluating AI agents in interactive environments is hindered by fragmented tasks, scaffolds, verifiers, and scoring rules. Existing efforts focus on narrow settings, remain limited in scale, or require costly reruns, leaving much of the empirical record incomparable. We introduce Messier, a unified corpus of 957,253 records that span 30 benchmarks, 714 agents, 11,891 tasks, and 74,205 verifiers. Messier consolidates public benchmark scores and supplements them with five-agent runs across six underrepresented professional and scientific domains, including a recent legal benchmark. Each record is standardized by model, scaffold, environment, task, verifier, and aggregation rule, with SOC/NAICS classifications for occupational and industry analysis. Using this corpus, we show frontier progress is uneven across benchmark types, with "function calling" saturated, "programming" improving the fastest, and "enterprise workflows" remaining the most challenging. Furthermore, counterfactual rescoring shows that strict all-pass aggregation in multi-verifier tasks can obscure progress and artificially alter agent rankings. From these standardized records, we derive capability scales that align with Epoch's Evaluation Capability Index rankings at Spearman \r{ho} = 0.81 and can be specialized by domain, occupation, action space, or verifier type. Messier provides a foundational, reusable infrastructure for agent capability scaling, benchmark auditing, and fine-grained analysis of evaluation failures.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_6fd85b9f9c79e49d0a703990