---
title: "SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2606.30573v1"
permalink: "https://feed.astralor.com/items/itm_ebc400b5561017d1f699cbf0"
published: "2026-06-29T17:17:45.000Z"
score: 58
channel: "ai"
tags: ["GPT-5.5","Opus 4.8","SWE-INTERACT","Multi-turn Interaction Benchmark","User Simulator","coding-agents"]
---

# SWE-INTERACT: Reimagining SWE Benchmarks as User-Driven Long-Horizon Coding Sessions

## At a glance

SWE-Interact is a new testbed for evaluating coding agents on multi-turn, interactive, user-driven software engineering tasks. Unlike existing benchmarks, it starts with vague or incomplete instructions via a carefully designed user simulator, progressively reveals requirements, inspects the workspace, and provides targeted feedback until the full task goal is handed off. Experiments show that strong performance on single-turn tasks does not reliably transfer to multi-turn user-driven workflows: the best models solve ~50% of single-turn baselines but only 25% of corresponding SWE-Interact tasks. Top models like Opus 4.8 and GPT 5.5 perform well under vague instructions but still suffer from over-agentic coding, forgetting requirements, and technical mistakes. The benchmark measures an orthogonal real-world capability: interactive goal discovery and iterative refinement with a user in the loop.

## Key points

- Proposes SWE-Interact benchmark simulating user-driven multi-turn software engineering tasks starting from vague requirements.
- Top models show significant performance drop from single-turn (50%) to multi-turn interactive tasks (25%).
- Strongest models like Opus 4.8 and GPT 5.5 handle vague instructions but suffer from over-agentic coding, forgetting requirements, and code errors.

## Full text

We introduce SWE-Interact, a new testbed for evaluating coding agents on multi-turn, interactive, user-driven software engineering tasks. Existing frontier SWE benchmarks typically provide complete requirements upfront and evaluate agents on autonomous implementation. In contrast, SWE-Interact places agents in a realistic developer workflow: a carefully designed user simulator starts with vague or incomplete instructions, progressively reveals requirements, inspects the agent's workspace, and provides targeted feedback, revisions, and new constraints until the full task goal has been handed off. Grounded in large-scale studies of real coding-agent interactions, this setup tests whether agents can discover user intent, adapt to evolving requirements, and build on their own prior work. Across a suite of frontier and open-weight models, we find that strong performance on single-turn SWE tasks does not reliably transfer to multi-turn, user-driven workflows: the best-performing models solve roughly 50% of single-turn baseline tasks but only 25% of the corresponding SWE-Interact tasks. The strongest models in our evaluation, including Opus 4.8 and GPT 5.5, start strong even in the face of vague initial instructions, persevere until all the requirements are surfaced by the user, integrate them better and write clean code. However, they still suffer from over-agentic coding, forgetting requirements and technical mistakes. Weaker models start poorly under ambiguity, give up early, forget or ignore instructions and rework their code more. Overall, SWE-Interact measures an orthogonal, real-world capability axis for frontier model development: interactive goal discovery and iterative refinement with a user in the loop.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_ebc400b5561017d1f699cbf0