---
title: "Beyond Attack-Success Rate: Action-Graded Severity Scale for Tool-Using AI Agents"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.07474v1"
permalink: "https://feed.astralor.com/items/itm_69be98291ef21efaa1963bcc"
published: "2026-07-08T14:38:01.000Z"
score: 81
channel: "ai"
tags: ["AgentDojo","AI Agent Security","Harm Assessment","Red Teaming"]
---

# Beyond Attack-Success Rate: Action-Graded Severity Scale for Tool-Using AI Agents

## AI commentary

> This paper hits the pain point of current AI agent security evaluation—binary metrics mask real harm. For AI safety researchers and defenders, this seven-level scale offers a practical assessment framework. Particularly noteworthy is its exposure of a 'zero attack-success rate' defense, highlighting how simple metrics can create false security. Readers should focus on the experimental section to learn how to apply the scale to their own red-team logs.

## At a glance

Current AI agent red-teaming benchmarks report attack success as a single binary bit, discarding information about the actual harm. This paper introduces a seven-level ordinal harm scale (L0-L6) based on whether the executed action is reversible, crosses scope to affect another party, or expands privilege. Evaluated on the AgentDojo workspace suite, the scale reveals cases hidden by binary metrics, including a defense reporting zero attack-success rate while allowing cross-scope leakage. A panel of three frontier LLM judges achieves high ordinal agreement with a deterministic oracle (Krippendorff's alpha=0.91) but shares systematic blind spots, such as failing to recognize escalation chains. Code, prompts, and logs are open-sourced.

## Key points

- Proposes a seven-level ordinal harm scale (L0-L6) assessing tool-call trajectories by reversibility, cross-scope impact, and privilege escalation.
- On the AgentDojo dataset, the scale reveals vulnerabilities hidden by binary attack-success rate, e.g., cross-scope leakage despite a 0% attack-success defense.
- A panel of three frontier LLM judges achieves high ordinal agreement (Krippendorff's alpha=0.91) with a deterministic oracle but shares systematic blind spots like failing to recognize escalation chains.

## Full text

Agentic red-teaming benchmarks report whether an injected agent was compromised as a single bit: the attack succeeded, or it did not. We argue that this binary attack-success rate discards the information a defender most needs, namely how harmful the resulting action was. We introduce an action-graded harm rubric that scores an agent's tool-call trajectory on a seven-level ordinal scale (L0 to L6) according to whether the executed action was reversible, whether it crossed scope to reach another party, and whether it expanded privilege. We compute the scale two ways: a deterministic oracle that reads the trajectory and the attacker's stated goal, and a panel of three frontier language-model judges that read a tag-free account of the same trajectory. Across four victim models and two defenses on the AgentDojo workspace suite, severity grading exposes three cases the binary metric hides, including a defense that reports a zero attack-success rate while still permitting an externally visible cross-scope leak through an unfiltered tool. The judge panel reproduces the oracle with high ordinal agreement (Krippendorff's alpha = 0.91) but shares systematic blind spots that we characterize, most notably a failure to recognize escalation chains. Unlike prior work that provides harm taxonomies, harmful-task completion tests, execution-level safety benchmarks, or severity-aware simulation, our contribution is a reusable, trace-grounded severity instrument applied to the actual actions recorded in existing red-team logs. All code, prompts, and per-episode logs are released.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_69be98291ef21efaa1963bcc