---
title: "QuoteBench: How Matched Scores Can Hide Command-Path Failures"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.13547v1"
permalink: "https://feed.astralor.com/items/itm_31ceb9467c346ce4aa1c2782"
published: "2026-08-13T17:57:20.000Z"
score: 77
channel: "ai"
tags: ["Bash command execution","GPT-5.6 Sol","LLM coding agents","QuoteBench","Model Evaluation","generation contract"]
---

# QuoteBench: How Matched Scores Can Hide Command-Path Failures

## AI commentary

> I think the most valuable move in this paper is reframing matched scores as a system property rather than an intrinsic model property; for any team deploying coding agents in real terminals, the parsing boundary is what really decides success. I would recommend that model evaluations always report both the raw path and the disclosed-boundary difference.

## At a glance

QuoteBench shows that matched execution scores cannot tell command-generation errors from failures introduced after generation. Using exact final-state validation on 56 one-shot tasks from 14 incident-derived families, the paper crosses the generation contract with an execution transport built around a deliberately unescaped added parser. Replaying the same reply through the added parser lowers success by 55.4-73.2 percentage points across eight same-window configurations; disclosing the boundary recovers 30.4-60.7 points for six configurations. GPT-5.6-sol's matched gap of -3.6 points hides -64.3 points of damage and +60.7 points of compensation. Deployment configuration can reorder models, so evaluations should report the full pipeline rather than treat a matched score as an intrinsic model property.

## Key points

- Matched execution scores cannot separate generation errors from post-generation failures; QuoteBench isolates them via exact final-state validation.
- Across eight same-window configurations, replaying the same reply through an added parser lowers success by 55.4-73.2 pp; disclosure recovers 30.4-60.7 pp in six configurations.
- Raw generation is nearly saturated; boundary adaptation separates models. GPT-5.6-sol's -3.6 matched gap hides -64.3 damage and +60.7 compensation.
- Deployment configuration reorders models; evaluations should report config, generation contract, execution path, operating point, and validator.

## Full text

LLM coding agents issue Bash commands through interfaces that may serialize, wrap, and reparse model output. Matched execution scores alone cannot distinguish command-generation errors from failures introduced after generation. QuoteBench measures this boundary with exact final-state validation on 56 one-shot tasks from 14 incident-derived families, crossing the generation contract with the execution transport around one deliberately unescaped added parser. Escaping at the interpolation point reproduces each replayed reply's raw-path outcome, so any recovery under a disclosed boundary must come from the model changing its generation. Across eight same-window configurations, replaying the same reply through the added parser lowers success by 55.4 to 73.2 percentage points; disclosure recovers 30.4 to 60.7 points for six configurations, and zero or slightly negative for the other two. Raw generation is nearly saturated at the frontier; boundary adaptation is what still separates models. GPT-5.6-sol's matched gap of -3.6 points hides -64.3 points of damage and +60.7 points of compensation. The deployment configuration reorders models: one reversal among 26 comparable pairs is unambiguous and four more sit on single-task margins. Evaluations of command-issuing agents should report the model configuration, generation contract, execution path, operating point, and final-state validator rather than treat a matched score as an intrinsic model property.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_31ceb9467c346ce4aa1c2782