---
title: "SWE-Bench ProMax：面向大规模多语言代码重构的智能体评测基准"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.09802v1"
permalink: "https://feed.astralor.com/items/itm_bed8a5a9110e1e2429bccb18"
published: "2026-08-10T16:23:19.000Z"
score: 77
channel: "ai"
tags: ["AI编程智能体","Hugging Face","SWE-Bench ProMax","代码重构"]
---

# SWE-Bench ProMax：面向大规模多语言代码重构的智能体评测基准

## AI 点评

> 现有基准的饱和与缺陷早已不是秘密，这次把矛头指向很少有人认真做的跨文件重构，确实是更接近真实工程的方向。对研究者和Agent开发者来说，这是一个值得跟进的新标尺；对普通使用者，它也提醒我们：看到的解决率可能远高估了模型在真实任务中的能力。

## 速览

现有AI编程智能体基准快速饱和且评测质量受质疑：审计发现SWE-bench Verified近60%未解实例含缺陷测试，前沿模型还能从训练数据逐字复现金牌补丁。为此，研究者推出SWE-Bench ProMax，包含来自7种语言的170个专家精选代码重构任务，平均每个实例修改11.4个文件、261.6行代码。实验显示，最佳模型在两种Agent框架下仅达到41.2%的解决率，证明该基准具有显著区分度和未饱和挑战。

## 关键点

- 审计发现现有基准存在过窄/过宽测试缺陷及训练数据泄露，评测质量受到挑战
- SWE-Bench ProMax包含170个跨7种语言的真实代码重构任务，平均每实例改动11.4个文件、261.6行代码
- 所有实例经过专家多阶段筛选，重写问题描述并人工审查测试套件，确保任务难度与质量
- 前沿模型在两种Agent框架下最高解决率仅41.2%，说明该基准尚未饱和

## 全文

As AI coding agents take on increasingly complex, long-horizon software engineering tasks, existing benchmarks are rapidly saturating and their evaluation quality has come under serious scrutiny: a recent audit found that nearly 60% of unsolved SWE-bench Verified instances contain flawed tests -- either overly narrow tests that reject correct solutions or overly broad tests that check unstated requirements -- and that frontier models can verbatim reproduce gold patches from training data. Code refactoring, which requires coordinated, behavior-preserving changes across many files, offers a substantially harder and more realistic test of agent capability, yet remains underserved by current benchmarks. We introduce SWE-Bench ProMax, an expert-curated, multilingual code refactoring benchmark of 170 instances drawn from real commits across seven programming languages (Python, Java, TypeScript, Go, C, C++, and Rust). Every instance undergoes rigorous, multi-stage curation that directly addresses the quality problems identified in prior benchmarks: issue descriptions are rewritten from scratch to provide precise, unambiguous specifications, and test suites are manually reviewed to remove overly narrow and overly broad tests. Tasks with insufficient complexity or limited cross-file scope are filtered out, yielding a benchmark of challenging, large-scale refactoring tasks that average 11.4 modified files and 261.6 lines of code per instance, substantially exceeding the scale of existing benchmarks. Experiments with frontier models under two agent scaffolds show that the best model achieves only 41.2% resolve rate, confirming that SWE-Bench ProMax presents a meaningful and unsaturated challenge for current AI coding agents. Our benchmark is available at https://huggingface.co/datasets/swe-bench-promax/SWE-Bench-ProMax.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_bed8a5a9110e1e2429bccb18