---
title: "LLM数据代理的轨迹完整性：现实系统可审计结构化推理的愿景"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.26036v1"
permalink: "https://feed.astralor.com/items/itm_3ef42f959c5c5e712cde3f77"
published: "2026-08-26T17:15:24.000Z"
score: 76
channel: "ai"
tags: ["BIRD Mini-Dev","LLM","可审计性","执行契约","数据代理"]
---

# LLM数据代理的轨迹完整性：现实系统可审计结构化推理的愿景

## AI 点评

> 这篇论文戳中了当前LLM数据代理评测的软肋：只看答案不看过程。对做AI Infra或可靠性平台的人来说，执行契约和CAIT率完全可以拿来设计回归测试和监控告警。建议所有把Agent放进生产环境的团队认真看看。

## 速览

该论文指出，仅靠答案准确率不足以评估LLM数据代理的可靠性——在结构化数据任务中，基准测试中的正确答案可能由无效计算轨迹产生。为此，作者提出轨迹完整性（Trace Integrity）作为部署可靠性标准，要求答案背后的计算过程具备显式、可执行、模式合规、操作符忠实、可重放、答案一致且可审计等特性。论文识别了“结构鸿沟”这一部署失败模式，并通过执行契约将用户意图与模式、算子计划、假设、查询和验证状态绑定，同时引入CAIT率（正确答案/无效轨迹率）量化静默失败风险。在BIRD Mini-Dev上的实验显示，不同方法在答案准确率与轨迹完整性通过率上差异显著，说明二者是独立的评估信号。

## 关键点

- 提出轨迹完整性（Trace Integrity）作为LLM数据代理的部署可靠性标准，包含七项可审计性要求。
- 识别“结构鸿沟”作为部署失败模式：自然语言推理与自由文本理由无法可靠地生成现实系统所需的操作符级程序。
- 引入执行契约（Execution Contracts）机制，将用户意图与模式元素、算子计划、假设、查询、验证状态及最终答案绑定。
- 在BIRD Mini-Dev上的实验表明，答案准确率与轨迹完整性通过率呈弱相关，CAIT率高达45.8%-59.1%，凸显静默失败风险。

## 全文

Answer accuracy is an insufficient reliability signal for LLM data agents. In structured-data tasks, a benchmark-correct answer can be produced by an invalid trace. This paper introduces Trace Integrity, a deployment reliability criterion for evaluating whether the computation recorded behind an answer is explicit, executable, schema-valid, operator-faithful, replayable, answer-consistent, and auditable. We identify the Structure Gap as the deployment failure mode that makes Trace Integrity necessary: natural-language reasoning and free-form rationales do not reliably specify the operator-level programs required by real-world systems. We operationalize Trace Integrity with execution contracts, structured artifacts that bind user intent to schema elements, operator plans, assumptions, executable queries, verification status, and final-answer linkage. We also introduce CAIT (Correct Answer / Invalid Trace) Rate, which measures how often answer-only evaluation counts computationally unsupported outputs as successes. In an empirical demonstration on BIRD Mini-Dev, Direct SQL, Operation Summary + SQL, and Contract-First SQL achieve answer accuracies of 20%, 22%, and 24%, while their Trace Integrity Pass Rates are 39%, 43%, and 40% and their CAIT Rates remain high at 55%, 59.1%, and 45.8%, showing that answer accuracy, trace validity, and silent-failure risk are distinct evaluation signals. Real-world LLM data agents should, therefore, be evaluated not only by whether their outputs match a reference answer, but by whether those outputs are backed by auditable computation.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_3ef42f959c5c5e712cde3f77