---
title: "QuoteBench：匹配分数如何掩盖命令路径失败"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.13547v1"
permalink: "https://feed.astralor.com/items/itm_31ceb9467c346ce4aa1c2782"
published: "2026-08-13T17:57:20.000Z"
score: 77
channel: "ai"
tags: ["Bash命令执行","GPT-5.6 Sol","LLM编码智能体","QuoteBench","模型评估","生成契约"]
---

# QuoteBench：匹配分数如何掩盖命令路径失败

## AI 点评

> 我认为这篇论文最值得关注的是把匹配分数从模型属性还原成系统属性；对任何在真实终端里部署编码智能体的团队来说，解析边界才是真正决定成败的地方。建议在发布模型评测时，至少同时报告原始路径和披露边界的差异。

## 速览

QuoteBench：匹配执行分数无法区分命令生成错误与执行管线引入的失败。在14个事故衍生族、56个一次任务上校验发现，同一回复经附加解析器后成功率下降55.4-73.2个百分点；披露边界恢复30.4-60.7点。GPT-5.6-sol的-3.6匹配差距掩盖了-64.3损伤和+60.7补偿。部署配置会重排排名，评估须报告完整配置。

## 关键点

- 匹配执行分数无法区分命令生成错误与执行后失败，QuoteBench用精确最终状态校验隔离二者。
- 八个同窗口配置中，同一回复经附加解析器后成功率下降55.4-73.2个百分点；披露边界可在六个配置中恢复30.4-60.7点。
- 原始生成几乎饱和，边界适应力才是模型分水岭；GPT-5.6-sol的-3.6匹配差距掩盖了-64.3损伤与+60.7补偿。
- 部署配置会重排模型排名，评估需报告模型配置、生成契约、执行路径、工作点和最终状态校验器。

## 全文

LLM coding agents issue Bash commands through interfaces that may serialize, wrap, and reparse model output. Matched execution scores alone cannot distinguish command-generation errors from failures introduced after generation. QuoteBench measures this boundary with exact final-state validation on 56 one-shot tasks from 14 incident-derived families, crossing the generation contract with the execution transport around one deliberately unescaped added parser. Escaping at the interpolation point reproduces each replayed reply's raw-path outcome, so any recovery under a disclosed boundary must come from the model changing its generation. Across eight same-window configurations, replaying the same reply through the added parser lowers success by 55.4 to 73.2 percentage points; disclosure recovers 30.4 to 60.7 points for six configurations, and zero or slightly negative for the other two. Raw generation is nearly saturated at the frontier; boundary adaptation is what still separates models. GPT-5.6-sol's matched gap of -3.6 points hides -64.3 points of damage and +60.7 points of compensation. The deployment configuration reorders models: one reversal among 26 comparable pairs is unambiguous and four more sit on single-task margins. Evaluations of command-issuing agents should report the model configuration, generation contract, execution path, operating point, and final-state validator rather than treat a matched score as an intrinsic model property.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_31ceb9467c346ce4aa1c2782