---
title: "SWE Refactor Bench: Can Coding Agents Complete a Long-Horizon, Whole-Repository Stack Migration?"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.23564v1"
permalink: "https://feed.astralor.com/items/itm_6ef5a435ce5b864c0255a3b1"
published: "2026-08-24T17:59:04.000Z"
score: 75
channel: "ai"
tags: ["arXiv","SWE Refactor Bench","Coding Agents","Technical Debt"]
---

# SWE Refactor Bench: Can Coding Agents Complete a Long-Horizon, Whole-Repository Stack Migration?

## AI commentary

> The most valuable part is how this benchmark exposes the 'pass tests without migrating' shortcut and reveals what agents can actually do. For anyone building agent evaluation, refactoring tools, or maintaining open source, the three-stage protocol and per-category difficulty breakdown are especially instructive.

## At a glance

SWE Refactor Bench introduces 20 whole-repository technical-debt migration tasks and a three-stage evaluation protocol: Migration Audit verifies the migration happened, Behavioural Tests check correctness on a fixed suite, and Agentic Verification uses six independent coding agents to generate targeted tests for hidden differences. Across 520 runs from 8 frontier models and 26 model-effort configurations, only 28 runs (5.4%) pass all stages, 13 of 20 tasks receive no accepted solution, and the best model (claude-opus-5) scores 47.0/100. The benchmark shows migration completeness and behavioural correctness are distinct abilities: many runs skip the migration to preserve behaviour and fail the audit, while most that attempt migration break behaviour. Among 340 runs passing audit, 58% reach 99% of checks but only 26% reach 100%. Agents score 31.4 on build toolchain rewrites versus 5.6 on language rewrites.

## Key points

- Three-stage protocol: Migration Audit, Behavioural Tests, and Agentic Verification closes the loophole of copying the original implementation to pass tests.
- Only 5.4% of 520 runs pass all stages; the best model, claude-opus-5, scores 47.0/100.
- Migration completeness and behavioural correctness are distinct abilities, and most agents trade one for the other.
- Difficulty varies by migration type: 31.4 on build toolchain rewrites versus 5.6 on language rewrites.

## Full text

Modern software systems accumulate technical debt over decades of development, which makes migration expensive and largely manual. As coding agents become increasingly capable at bug fixing, can they autonomously perform such migrations? Existing benchmarks cannot answer this question because they evaluate only behavioural correctness, not whether the migration actually occurred. This leads an easy hack: agents copy the original implementation to make tests pass. We call this Blindness. To address this problem, we introduce SWE Refactor Bench, a benchmark comprising 20 whole-repository migrations, covering 4 kinds of technical debt. A three-stage evaluation protocol measures both migration completeness and behavioural correctness. (1) Migration Audit verifies that the migration occurred. (2) Behavioural Tests measure correctness with a fixed test suite. (3) Agentic Verification uses 6 independent coding agents to generate targeted tests for hidden behavioural differences. Across 520 runs from 8 frontier models and 26 model-effort configurations, only 28 of 520 runs ($5.4\%$) pass all three stages, 13 of the 20 tasks receive no accepted solution, and the best model (claude-opus-5) scores $47.0/100$. Migration completeness and behavioural correctness are distinct abilities: a few runs preserve behaviour by skipping the migration and are stopped at Migration Audit; most attempt it and break behaviour, and are stopped at Behavioural Tests. Agents cannot deliver a perfect migration: among the 340 runs that pass Migration Audit, $58\%$ reach $99\%$ of the fixed checks, yet only $26\%$ reach $100\%$. Agent capability differs across migration categories: agents score $31.4$ on build toolchain rewrites but only $5.6$ on language rewrites. Together, these findings position SWE Refactor Bench as a rigorous testbed for developing coding agents for reliable whole-repository migrations.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_6ef5a435ce5b864c0255a3b1