---
title: "Train the Model, Not the Reader: Decodability Supervision for Verifiable Activation Explanations"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.20379v1"
permalink: "https://feed.astralor.com/items/itm_f136ac3711f18fe255c10a62"
published: "2026-07-22T17:10:23.000Z"
score: 79
channel: "ai"
tags: ["AI Security","Pythia-160M","Qwen-2.5-7B","ReCap","Interpretability"]
---

# Train the Model, Not the Reader: Decodability Supervision for Verifiable Activation Explanations

## AI commentary

> This paper is a must-read for anyone working on AI interpretability and safety. It strikes at the heart of a popular evaluation method—reconstruction scores don't certify individual claims—and offers a practical fix with RECAP. For those studying model introspection and red-teaming, the independent verifiability RECAP provides is a major step forward.

## At a glance

This paper reveals that natural-language autoencoders score explanations of hidden activations by reconstruction, but the score is insensitive to individual false claims, leading to unfaithful passing of the test. The authors propose two audit protocols and RECAP: training linear heads alongside the target model to keep designated content decodable. Experiments show RECAP substantially improves verifiability and maintains lie detection under adversarial attacks.

## Key points

- Reconstruction test is insensitive to individual false claims, explanations may be unfaithful
- Propose RECAP: train auxiliary linear heads to keep designated content decodable
- RECAP validated on small models and Pythia-160M
- Under adversarial attacks, RECAP probe still detects lies (AUC 0.95)

## Full text

Natural-language autoencoders score explanations of hidden activations by reconstruction: an explanation is deemed faithful if the activation can be regenerated from it. The test is structurally insensitive to individual false claims: if flipping a claim does not change the reconstruction, the claim is never penalized. We show the test is passed in two ways, neither faithful. On a released Qwen-2.5-7B verbalizer, explanations reconstruct well above chance while ~2% of specific claims are reconstruction-dependent, so the score tracks gist, not specific facts. Under exact synthetic ground truth, the standard recipe develops co-adapted private codes (false wording the reconstruction depends on) in 5/5 runs, and fixes that leave the target model unchanged do not help. We contribute two audit protocols, the grounded-vs-true cross and the evaluator swap, and RECAP (Readable Encodings via Co-trained Auxiliary Predictors): linear heads trained alongside the target model to keep designated content decodable. On RECAP-trained sandbox models, fresh verbalizers state the designated content truly and the codes vanish, at a +0.001-nat cost. This replicates on a pretrained Pythia-160M: the content becomes reliably probe-decodable, though a fresh verbalizer conveys it only in part (truth 0.44-0.46 vs a near-zero control). For interpretability, high reconstruction does not certify individual claims. For AI safety, RECAP makes designated internal content independently checkable against probes rather than asserted by prose a model can game: an independent probe scores the verbalizer's true claims above its false ones (AUC 0.96, vs 0.82 without RECAP). Against an adversary that edits an explanation to maximize the reconstruction score while lying (suppressing ~87% of its lie penalty), the RECAP probe still flags the lies (AUC 0.95) while the control probe collapses to chance (0.51).

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_f136ac3711f18fe255c10a62