---
title: "Agent Hacks Agent: Autoresearch for Production-Agent Red-Teaming"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.11698v1"
permalink: "https://feed.astralor.com/items/itm_1a2d973f6a2d311b4e04b0b1"
published: "2026-07-13T15:31:04.000Z"
score: 81
channel: "ai"
tags: ["AHA","Claude Code","Codex","LLM Agent Safety","Red-Teaming","Vulnerability Concept Graph"]
---

# Agent Hacks Agent: Autoresearch for Production-Agent Red-Teaming

## AI commentary

> This paper upgrades red-teaming from one-off vulnerability hunting to systematic knowledge accumulation. If you are deploying agents like Claude Code or Codex, or responsible for AI safety evaluation, the AHA framework and VCG concept can save you significant repetitive testing effort. More importantly, it makes vulnerability discovery traceable and transferable — no longer a one-time bug report, but a continuously growing safety knowledge base.

## At a glance

Production LLM agents (e.g., Claude Code, Codex) operate over untrusted content, files, commands, and workspace state, making safety failures directly actionable. Existing red-teaming methods mainly optimize attack success and preserve artifacts such as benchmarks, payloads, or attack programs, but fail to record the enabling conditions behind unsafe behavior. This paper proposes AHA (Autoresearch for Production-Agent Red-Teaming), a falsifiable discovery loop that uses one agentic research environment to discover reusable vulnerability knowledge about another. AHA proposes a vulnerability hypothesis, constructs a falsifier, instantiates a valid attack, executes it in a sandboxed harness, reflects on the trajectory, and promotes confirmed findings into a Vulnerability Concept Graph (VCG). Each concept links an attacker-facing surface to an unsafe trajectory through a claim, enabling condition, falsifier, transfer prediction, and supporting evidence. Across Claude Code and Codex on three scenarios covering direct and indirect attacks, the discovered concepts reveal a reusable vulnerability core across models and agents. A frozen VCG outperforms the strongest frozen discovery baseline by 14.2 percentage points under the same single-shot protocol, while transferring across scenarios and attack channels. The resulting VCG provides an auditable artifact for production safety teams to inspect vulnerabilities, validate patches, and accumulate reusable safety knowledge.

## Key points

- Proposes AHA framework for automated red-teaming of production LLM agents, generating a reusable Vulnerability Concept Graph (VCG).
- VCG records attack surfaces, enabling conditions, falsifiers, and transfer predictions, surpassing traditional artifacts that only capture successful attacks.
- Validated on Claude Code and Codex; frozen VCG outperforms the strongest frozen baseline by 14.2 percentage points and transfers across scenarios.
- Provides an auditable artifact for safety teams to inspect vulnerabilities, validate patches, and accumulate reusable safety knowledge.

## Full text

Production LLM agents such as Claude Code and Codex operate over untrusted content, files, commands, and workspace state, making safety failures directly actionable. Red-teaming must therefore keep pace with evolving models and tools. Existing approaches mainly optimize attack success and preserve artifacts such as benchmarks, payloads, or attack programs, which record where attacks succeed but not the enabling conditions behind unsafe agent behavior. We study automated red-teaming for production LLM agents using one agentic research environment to discover reusable vulnerability knowledge about another. We present AHA, a falsifiable discovery loop that proposes a vulnerability hypothesis, constructs a falsifier, instantiates a valid attack, executes it in a sandboxed harness, reflects on the trajectory, and promotes confirmed findings into a Vulnerability Concept Graph (VCG). Each concept links an attacker-facing surface to an unsafe trajectory through a claim, enabling condition, falsifier, transfer prediction, and supporting evidence. Across Claude Code and Codex on three scenarios covering direct and indirect attacks, the discovered concepts reveal a reusable vulnerability core across models and agents. A frozen VCG requires no further search and outperforms the strongest frozen discovery baseline by 14.2 percentage points under the same single-shot protocol, while transferring across scenarios and attack channels. The resulting VCG provides an auditable artifact for production safety teams to inspect vulnerabilities, validate patches, and accumulate reusable safety knowledge. Our code is available at https://github.com/henrymao2004/Auto-research-red-teaming-in-sleep.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_1a2d973f6a2d311b4e04b0b1