---
title: "Item Response Theory for AI Safety: Reading, Reducing, and Auditing Benchmarks"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.05086v1"
permalink: "https://feed.astralor.com/items/itm_6732e2879262dd6b9d7444ff"
published: "2026-08-05T17:25:27.000Z"
score: 78
channel: "ai"
tags: ["AI Security","Large Language Models","Security Benchmark","Item Response Theory"]
---

# Item Response Theory for AI Safety: Reading, Reducing, and Auditing Benchmarks

## AI commentary

> What stands out is how the paper repurposes a classic psychometric tool for AI safety, directly tackling the pain points of untrustworthy, expensive, and gameable benchmarks. It is a high-density read for evaluation teams, safety compliance leads, and investors tracking LLM risk. The claim that ten items can replace a whole benchmark may sound bold, but it's worth thinking through carefully.

## At a glance

Language models differ in safety behavior, and these differences are measured by safety benchmarks. However, aggregated scores are hard to trust due to duplicated items, high correlations, and possible sandbagging when models detect evaluation. This study applies Item Response Theory (IRT) to 192 models across eight safety benchmarks—the largest psychometric analysis of LLM safety evaluations to date. Three interpretable factors—refusal strictness, truthfulness, and contextual harm—explain most model variance. Psychometrically selected items recover full benchmark scores with lower error than random subsets, and roughly ten adaptive items suffice for several benchmarks, cutting evaluation cost by 97–99%. IRT also enables audits of individual models, detecting naive sandbagging and model swaps behind APIs. The authors recommend frontier labs and evaluators adopt IRT as a ready-made toolkit.

## Key points

- IRT analysis of 192 models across 8 safety benchmarks reveals three interpretable factors—refusal strictness, truthfulness, and contextual harm—dominating model variance.
- Psychometrically selected items recover full benchmark scores with lower error than random subsets; about 10 adaptive items suffice for several benchmarks, cutting cost by 97–99%.
- IRT supports audits of individual models, detecting naive sandbagging and model swaps behind APIs.

## Full text

Language models differ in how safely they behave and these differences are measured by safety benchmarks. But aggregated benchmark scores are hard to trust and interpret, because benchmarks duplicate one another, correlate heavily, and models may sandbag when they detect evaluation. To address these issues, we draw on Item Response Theory (IRT), a statistical toolkit for measuring these latents from performance on items with inferred psychometric properties. We fit IRT models to eight safety benchmarks across 192 language models, the largest psychometric analysis of LLM safety evaluations to date, and contribute three results. First, we find that three interpretable factors of refusal strictness, truthfulness, and contextual harm explain most of the variance between models across benchmarks. Second, psychometrically selected items recover full benchmark scores with lower error than random subsets of the same size, and roughly ten adaptively chosen items suffice for several individual benchmarks, cutting evaluation cost by 97-99%. Third, IRT supports audits of individual models, showing that it can be used to detect naive sandbagging and changes of model behind APIs. Overall, we show IRT is a ready-made toolkit for reading, reducing, and auditing safety benchmarks, which we recommend frontier labs and evaluators adopt.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_6732e2879262dd6b9d7444ff