---
title: "Trace Integrity for LLM Data Agents: A Vision for Auditable Structured Reasoning in Real-World Systems"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.26036v1"
permalink: "https://feed.astralor.com/items/itm_3ef42f959c5c5e712cde3f77"
published: "2026-08-26T17:15:24.000Z"
score: 76
channel: "ai"
tags: ["BIRD Mini-Dev","LLM","Auditability","Execution Contract","Data Agent"]
---

# Trace Integrity for LLM Data Agents: A Vision for Auditable Structured Reasoning in Real-World Systems

## AI commentary

> This paper hits the weak point of LLM data agent evaluation: focusing only on answers, not processes. For AI infrastructure or reliability platform builders, execution contracts and CAIT rate are practical for designing regression tests and monitoring alerts. Teams putting agents into production should read this.

## At a glance

This paper argues that answer accuracy alone is an insufficient reliability signal for LLM data agents: in structured-data tasks, a benchmark-correct answer can be produced by an invalid trace. The authors introduce Trace Integrity as a deployment reliability criterion, requiring that the computation behind an answer be explicit, executable, schema-valid, operator-faithful, replayable, answer-consistent, and auditable. They identify the Structure Gap as the key deployment failure mode, and operationalize Trace Integrity via execution contracts that bind user intent to schema elements, operator plans, assumptions, executable queries, verification status, and final-answer linkage. They also introduce the CAIT (Correct Answer / Invalid Trace) Rate to measure how often answer-only evaluation counts computationally unsupported outputs as successes. Experiments on BIRD Mini-Dev show that answer accuracy, trace validity, and silent-failure risk are distinct evaluation signals, with CAIT rates remaining high across methods.

## Key points

- Proposes Trace Integrity as a deployment reliability criterion for LLM data agents, with seven auditability requirements.
- Identifies the Structure Gap as a deployment failure mode: natural-language reasoning and free-form rationales do not reliably specify operator-level programs.
- Introduces execution contracts that bind user intent to schema elements, operator plans, assumptions, queries, verification status, and final-answer linkage.
- Experiments on BIRD Mini-Dev show answer accuracy and trace validity are weakly correlated, with CAIT rates as high as 45.8%-59.1%, highlighting silent-failure risk.

## Full text

Answer accuracy is an insufficient reliability signal for LLM data agents. In structured-data tasks, a benchmark-correct answer can be produced by an invalid trace. This paper introduces Trace Integrity, a deployment reliability criterion for evaluating whether the computation recorded behind an answer is explicit, executable, schema-valid, operator-faithful, replayable, answer-consistent, and auditable. We identify the Structure Gap as the deployment failure mode that makes Trace Integrity necessary: natural-language reasoning and free-form rationales do not reliably specify the operator-level programs required by real-world systems. We operationalize Trace Integrity with execution contracts, structured artifacts that bind user intent to schema elements, operator plans, assumptions, executable queries, verification status, and final-answer linkage. We also introduce CAIT (Correct Answer / Invalid Trace) Rate, which measures how often answer-only evaluation counts computationally unsupported outputs as successes. In an empirical demonstration on BIRD Mini-Dev, Direct SQL, Operation Summary + SQL, and Contract-First SQL achieve answer accuracies of 20%, 22%, and 24%, while their Trace Integrity Pass Rates are 39%, 43%, and 40% and their CAIT Rates remain high at 55%, 59.1%, and 45.8%, showing that answer accuracy, trace validity, and silent-failure risk are distinct evaluation signals. Real-world LLM data agents should, therefore, be evaluated not only by whether their outputs match a reference answer, but by whether those outputs are backed by auditable computation.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_3ef42f959c5c5e712cde3f77