TestEvo-Bench：代码与测试协同演化的可执行实时基准

Computer Science > Software Engineering

arXiv:2607.02469v1 (cs) [Submitted on 2 Jul 2026]

Title:TestEvo-Bench: An Executable and Live Benchmark for Test and Code Co-Evolution

Authors:Jiale Amber Wang, Kaiyuan Wang, Pengyu Nie View a PDF of the paper titled TestEvo-Bench: An Executable and Live Benchmark for Test and Code Co-Evolution, by Jiale Amber Wang and 2 other authors View PDF HTML (experimental)

Abstract:Software tests and code evolve together: a code change should be followed by new or updated tests that record the new software behavior. Yet existing test generation and update benchmarks often isolate the test from the code change, and rely on static metadata that does not verify whether a test is executable or semantically tied to the code change. This makes it difficult to evaluate whether a test automation agent understands how a code change should propagate into the test suite.
We introduce TestEvo-Bench, a benchmark of test and code co-evolution tasks mined from software repositories, with two tracks: in test generation, the agent shall write new tests to capture the new software behavior; in test update, the agent shall adapt failing existing tests to the changed software behavior. Each task is anchored to a real commit history and packaged with environment configuration to support execution-grounded metrics such as pass rate, coverage, and mutation score. TestEvo-Bench is also a live benchmark: each task records the timestamp of the test and code changes, and new tasks are periodically mined by our automated pipeline, so evaluation can be restricted to tasks postdating a model's training cutoff to reduce data leakage risk. The current snapshot contains 746 test generation and 509 test update tasks, curated from 59,950 candidate co-evolution records across 152 open-source Java projects. We experiment with four state-of-the-art agents that combine strong harnesses (Claude Code, Gemini CLI, and SWE-Agent) with strong foundation models (Claude Opus 4.7 and Gemini 3.1 Pro). Results show that they achieve up to 77.5% success rate on test generation and 74.6% on test update. However, success rate is materially lower on the most recent benchmark tasks and drops significantly under limited per-task cost.

Comments:	TestEvo-Bench leaderboard and data explorer are hosted at this https URL
Subjects:	Software Engineering (cs.SE); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
Cite as:	arXiv:2607.02469 [cs.SE]
	(or arXiv:2607.02469v1 [cs.SE] for this version)
	https://doi.org/10.48550/arXiv.2607.02469 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Pengyu Nie [view email]
[v1] Thu, 2 Jul 2026 17:35:20 UTC (1,039 KB)
Full-text links:

Access Paper:

View a PDF of the paper titled TestEvo-Bench: An Executable and Live Benchmark for Test and Code Co-Evolution, by Jiale Amber Wang and 2 other authors
• View PDF
• HTML (experimental)
• TeX Source

view license

Current browse context:

cs.SE < prev   |   next >
new | recent | 2026-07 Change to browse by: cs
cs.AI
cs.CL

References & Citations

• NASA ADS
• Google Scholar
• Semantic Scholar

export BibTeX citation Loading...

BibTeX formatted citation

× Data provided by:

Bookmark

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle Bibliographic Explorer (What is the Explorer?) Connected Papers Toggle Connected Papers (What is Connected Papers?) Litmaps Toggle Litmaps (What is Litmaps?) scite.ai Toggle scite Smart Citations (What are Smart Citations?) Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle alphaXiv (What is alphaXiv?) Links to Code Toggle CatalyzeX Code Finder for Papers (What is CatalyzeX?) DagsHub Toggle DagsHub (What is DagsHub?) GotitPub Toggle Gotit.pub (What is GotitPub?) Huggingface Toggle Hugging Face (What is Huggingface?) ScienceCast Toggle ScienceCast (What is ScienceCast?) Demos

Demos

Replicate Toggle Replicate (What is Replicate?) Spaces Toggle Hugging Face Spaces (What is Spaces?) Spaces Toggle TXYZ.AI (What is TXYZ.AI?) Related Papers

Recommenders and Search Tools

Link to Influence Flower Influence Flower (What are Influence Flowers?) Core recommender toggle CORE Recommender (What is CORE?)

• Author
• Venue
• Institution
• Topic

About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)