---
title: "LLM Agent失败的实时检测与修复"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.02464v1"
permalink: "https://feed.astralor.com/items/itm_640543ca607ba766d381921a"
published: "2026-08-03T16:34:46.000Z"
score: 77
channel: "ai"
tags: ["CUSUM","Gemini 2.5 Flash","LLM Agent","Qwen2.5","回声状态网络"]
---

# LLM Agent失败的实时检测与修复

## AI 点评

> 这篇论文打动我的不是更高的准确率，而是把检测成本打到微秒级并公开复现。对正在上线Agent的团队，尤其那些被“用大模型裁判大模型”烧掉预算的人，这套“遥测+确定性验证”思路值得立刻借鉴。

## 速览

本文提出一种基于步骤遥测的LLM Agent失败实时检测与修复方案。传统方法用第二个LLM评判每一步，成本高昂；作者采用仅在健康轨迹上训练的一类回声状态网络集成加CUSUM报警，每步耗时约200微秒，比LLM评判低三个数量级。在2823个episode上，5%误报预算下检测出71%失败，AUROC 0.872。额外引入确定性验证层，零误报。修复阶段对失败轨迹回滚重跑，任务成功率从52%提升至73%，平均每次多一次模型调用。

## 关键点

- 基于单类回声状态网络集成与CUSUM报警，仅用步骤遥测即可实时检测Agent失败，AUROC 0.872。
- 每步检测仅约200微秒，比使用第二个LLM评判便宜三个数量级。
- 确定性验证层不依赖健康分布，零误报捕获60%失败。
- 修复阶段通过回滚重跑将任务成功率从52%提升到73%。

## 全文

LLM agents fail mid-episode -- they loop, cascade tool errors, drift off goal, fabricate results, or silently absorb corrupted content -- and the standard remedy, judging every step with a second LLM, costs more than the agent itself. We ask how much detection is achievable from observable step telemetry alone, using monitors costing microseconds per step and trained only on healthy runs. On 2,823 committed agent episodes across three frameworks, three local models (qwen2.5 7b/3b, llama3.1 8b) and a commercial API (gemini-2.5-flash), a one-class echo-state-network ensemble with CUSUM alarms detects 0.71 of failures at a 5% false-alarm budget (AUROC 0.872). Its advantage over a memoryless baseline is a monotone function of post-onset horizon (+0.09 at <=3 steps, +0.40 at >=9), predicting its own failure region out of sample on AFTraj-2K. Ranking transfers with no retraining to two corpora from other groups (AFTraj-2K 0.745, ATBench 0.779). Monitors carry two burdens: a per-deployment healthy null (they do not transfer -- AUROC 0.527 cold against 0.885 recalibrated) and a residual false-alarm rate. We add a layer carrying neither: deterministic verification, which recomputes a run's stated total from the tool results it actually received and confirms every required call was made. Head-to-head it catches 60% of failures (96% with the coverage check) at 0 of 63 false positives against the monitor's 54% at 17%, transfers unchanged to llama3.1:8b (110 of 110 at 0 of 10), and trips on 0 of 1825 healthy episodes. Detection is then closed into repair: each flagged run is rolled back and re-run live, recovering 45% of failures against a 16% resampling control (p=0.0005) and lifting task success from 52% to 73% for about one extra model call per run. The system runs at ~200 microseconds per step, three orders of magnitude below a judge call. Code, traces and results are released.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_640543ca607ba766d381921a