Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?

Computer Science > Software Engineering

arXiv:2607.01211v1 (cs) [Submitted on 1 Jul 2026]

Title:Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?

Authors:Zhi Chen, Zhensu Sun, Yuling Shi, David Lo, Lingxiao Jiang View a PDF of the paper titled Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?, by Zhi Chen and 4 other authors View PDF HTML (experimental)

Abstract:Repository-level performance-optimization benchmarks such as GSO, SWE-Perf and SWE-fficiency evaluate coding agents by applying patches to real repositories and comparing runtime against unoptimized baselines and official reference patches. Their leaderboard scores are increasingly used as evidence of coding-agent progress, but those scores can conflate runtime instability, benchmark-specific scoring rules, and how many tasks are already solved by at least one public submission. We audit these issues across the three benchmarks. First, we replay the official reference patches for 740 code optimization tasks across four common types of Google Cloud machines. Most benchmark tasks can be replayed, but their reference patches satisfy the original benchmark validity rules in every cross-machine replay for only 39/102 GSO tasks, 11/140 SWE-Perf tasks, and 411/498 SWE-fficiency tasks; SWE-Perf is especially fragile because many reference patches produce close-to-zero runtime changes. Second, we show that public submission rankings depend strongly on the benchmark scoring rule. Among eight public submissions shared by GSO and SWE-fficiency, the official rankings disagree on 9 of 28 pairwise submission comparisons, and SWE-fficiency's leaderboard scoring rule assigns the worst ten tasks overly high score weights of 58.5%-82.8%. Third, looking across 10 public submissions for each task, we find that at least one submission matches or beats the reference patch on 85.3% (384/450) of replay-valid GSO and SWE-fficiency tasks, and beats the unoptimized base code on 99.8% (449/450). Our study complements leaderboard scores by identifying tasks with more reliable performance signals, quantifying per-task score contributions, and exposing the remaining performance gaps that are hidden by aggregate rankings.

Subjects:	Software Engineering (cs.SE); Artificial Intelligence (cs.AI)
Cite as:	arXiv:2607.01211 [cs.SE]
	(or arXiv:2607.01211v1 [cs.SE] for this version)
	https://doi.org/10.48550/arXiv.2607.01211 Focus to learn more arXiv-issued DOI via DataCite

Submission history

From: Zhi Chen [view email]
[v1] Wed, 1 Jul 2026 17:50:48 UTC (246 KB)
Full-text links:

Access Paper:

View a PDF of the paper titled Are Performance-Optimization Benchmarks Reliably Measuring Coding Agents?, by Zhi Chen and 4 other authors
• View PDF
• HTML (experimental)
• TeX Source

view license

Current browse context:

cs.SE < prev   |   next >
new | recent | 2026-07 Change to browse by: cs
cs.AI

References & Citations

• NASA ADS
• Google Scholar
• Semantic Scholar

export BibTeX citation Loading...

BibTeX formatted citation

× Data provided by:

Bookmark

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle Bibliographic Explorer (What is the Explorer?) Connected Papers Toggle Connected Papers (What is Connected Papers?) Litmaps Toggle Litmaps (What is Litmaps?) scite.ai Toggle scite Smart Citations (What are Smart Citations?) Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle alphaXiv (What is alphaXiv?) Links to Code Toggle CatalyzeX Code Finder for Papers (What is CatalyzeX?) DagsHub Toggle DagsHub (What is DagsHub?) GotitPub Toggle Gotit.pub (What is GotitPub?) Huggingface Toggle Hugging Face (What is Huggingface?) ScienceCast Toggle ScienceCast (What is ScienceCast?) Demos

Demos

Replicate Toggle Replicate (What is Replicate?) Spaces Toggle Hugging Face Spaces (What is Spaces?) Spaces Toggle TXYZ.AI (What is TXYZ.AI?) Related Papers

Recommenders and Search Tools

Link to Influence Flower Influence Flower (What are Influence Flowers?) Core recommender toggle CORE Recommender (What is CORE?)

• Author
• Venue
• Institution
• Topic

About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)