---
title: "Measuring the Wrong Thing: Internal Harmfulness Scores Anti-Rank Successful Jailbreaks"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.09624v1"
permalink: "https://feed.astralor.com/items/itm_120f4500b947ac17f34d2a47"
published: "2026-08-10T14:05:32.000Z"
score: 78
channel: "ai"
tags: ["arXiv","AUROC","Llama","Active Attention Probing","Safety Evaluation","Jailbreak"]
---

# Measuring the Wrong Thing: Internal Harmfulness Scores Anti-Rank Successful Jailbreaks

## AI commentary

> This paper makes me rethink the foundation of safety evaluation—are we validating metrics that actually measure the attacks we want to block? For safety teams and red teams, it warns that a score distinguishing harmful from benign on average can completely fail on the most dangerous cases. Instead of relying on one proxy metric, direct evaluation of generation outcomes or cross-validation with active probing is necessary.

## At a glance

This paper argues that internal safety scores judge a prompt before text generation and are validated by how well they separate harmful from benign prompts, yet such separation does not guarantee catching attacks that actually succeed. The authors introduce Active Attention Probing, which supplies a fixed content-independent measurement coordinate. On Llama, wrapping harmful prompts raises harmful generation from 0.05 to 0.27 while harmful intent AUROC falls from 0.936 to 0.803; among wrapped harmful prompts, the outcome AUROC is 0.220, placing successful attacks below failed ones. The reversal persists across three target models, seven attack families, and two judges. Distribution shift degrades calibration and threshold transfer before degrading ranking, meaning filters tuned on the wrong quantity waste their false positive budget on attacks that would have failed anyway.

## Key points

- Internal safety scores measure harmful intent of a prompt, not jailbreak success, creating a systematic mismatch.
- Active Attention Probing using fixed coordinates shows wrapped harmful prompts become more dangerous in generation while looking safer to the score.
- Successful jailbreaks anti-rank below failed ones (AUROC 0.220), proving that separating harmful intent is not the same as separating actual attack outcomes.
- The reversal persists across models, attack families, and judges; distribution shift degrades calibration and threshold transfer before ranking. 

## Full text

Internal safety scores judge a prompt before any text is generated, and they are validated by how well they separate harmful prompts from benign ones. That separation is then read as evidence that the score will also catch the attacks that succeed. Harmful intent is a property of the prompt. Jailbreak success is an outcome produced later by a particular target model, decoding policy, and judge. A filter tuned on a score that measures the wrong quantity spends its false positive budget on attacks that would have failed anyway. In this paper we audit that inference. Attention based measurements are usually read from prompt dependent locations, so a wrapper changes both the content being judged and the place the signal is taken from. We therefore introduce Active Attention Probing, which supplies a fixed content independent measurement coordinate. We pair every base goal with a plain and a wrapped version and generate real completions from the target models. On Llama, wrapping raises harmful generation from 0.05 to 0.27 while harmful intent AUROC falls from 0.936 to 0.803, so the attacks grow more dangerous while the prompts look safer to the score. Among wrapped harmful prompts the outcome AUROC is 0.220, which places the attacks that succeeded below the attacks that failed. Rare token, passive, and detector derived channels reproduce the reversal on the same matched design, and the reversal itself persists across three target models, seven attack families, and two independent judges. Distribution shift then degrades calibration and threshold transfer before it degrades ranking.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_120f4500b947ac17f34d2a47