---
title: "Real-Time Detection and Repair of LLM Agent Failures"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.02464v1"
permalink: "https://feed.astralor.com/items/itm_640543ca607ba766d381921a"
published: "2026-08-03T16:34:46.000Z"
score: 77
channel: "ai"
tags: ["CUSUM","Gemini 2.5 Flash","LLM Agent","Qwen2.5","Echo-State Network"]
---

# Real-Time Detection and Repair of LLM Agent Failures

## AI commentary

> What impresses me here is not a higher accuracy number, but pushing detection cost down to microseconds with open code and data. For teams deploying agents—especially those bleeding tokens on LLM-as-a-judge—the telemetry-plus-deterministic-verification idea is worth borrowing right away.

## At a glance

This paper presents a real-time detection and repair framework for LLM agent failures using per-step telemetry. Instead of employing a second LLM to judge every action, the authors train a one-class echo-state network ensemble with CUSUM alarms on healthy trajectories only. On 2,823 episodes across three frameworks and four models, it detects 71% of failures at a 5% false-alarm budget (AUROC 0.872), with a cost of ~200 microseconds per step—three orders of magnitude cheaper than a judge call. A deterministic verification layer recomputes totals and checks required calls, catching 60% of failures with zero false positives. Repairs roll back and re-run flagged trajectories, improving task success from 52% to 73% at roughly one extra model call per run.

## Key points

- Detects agent failures in real time from step telemetry using a one-class echo-state network ensemble with CUSUM alarms, achieving AUROC 0.872.
- Monitor costs ~200 microseconds per step, three orders of magnitude lower than a judge LLM call.
- A deterministic verification layer, free from healthy-run assumptions, catches 60% of failures with zero false positives.
- Repair by rollback and re-run lifts task success from 52% to 73% at roughly one extra model call per run.

## Full text

LLM agents fail mid-episode -- they loop, cascade tool errors, drift off goal, fabricate results, or silently absorb corrupted content -- and the standard remedy, judging every step with a second LLM, costs more than the agent itself. We ask how much detection is achievable from observable step telemetry alone, using monitors costing microseconds per step and trained only on healthy runs. On 2,823 committed agent episodes across three frameworks, three local models (qwen2.5 7b/3b, llama3.1 8b) and a commercial API (gemini-2.5-flash), a one-class echo-state-network ensemble with CUSUM alarms detects 0.71 of failures at a 5% false-alarm budget (AUROC 0.872). Its advantage over a memoryless baseline is a monotone function of post-onset horizon (+0.09 at <=3 steps, +0.40 at >=9), predicting its own failure region out of sample on AFTraj-2K. Ranking transfers with no retraining to two corpora from other groups (AFTraj-2K 0.745, ATBench 0.779). Monitors carry two burdens: a per-deployment healthy null (they do not transfer -- AUROC 0.527 cold against 0.885 recalibrated) and a residual false-alarm rate. We add a layer carrying neither: deterministic verification, which recomputes a run's stated total from the tool results it actually received and confirms every required call was made. Head-to-head it catches 60% of failures (96% with the coverage check) at 0 of 63 false positives against the monitor's 54% at 17%, transfers unchanged to llama3.1:8b (110 of 110 at 0 of 10), and trips on 0 of 1825 healthy episodes. Detection is then closed into repair: each flagged run is rolled back and re-run live, recovering 45% of failures against a 16% resampling control (p=0.0005) and lifting task success from 52% to 73% for about one extra model call per run. The system runs at ~200 microseconds per step, three orders of magnitude below a judge call. Code, traces and results are released.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_640543ca607ba766d381921a