---
title: "人工智能安全领域的项目反应理论：基准测试的读取、缩减与审计"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.05086v1"
permalink: "https://feed.astralor.com/items/itm_6732e2879262dd6b9d7444ff"
published: "2026-08-05T17:25:27.000Z"
score: 78
channel: "ai"
tags: ["AI安全","大语言模型","安全基准","项目反应理论"]
---

# 人工智能安全领域的项目反应理论：基准测试的读取、缩减与审计

## AI 点评

> 这篇论文最值得看的地方，是把教育测量学的老工具重新用到AI安全上，直接回应了安全基准不可信、太贵、容易被骗三个痛点。对做模型评估的团队、安全合规负责人和关注大模型风险的投资者来说，都是高信息密度的参考。它没有把结论说死，但“十个题就能替代整份基准”这点，仔细想想还挺刺激的。

## 速览

语言模型在安全行为上存在差异，通常通过安全基准衡量，但聚合分数因题项重复、高度相关以及模型检测到评估时可能“沙袋化”而难以信任和解释。本研究将项目反应理论应用于8个安全基准、192个语言模型的评估，这是迄今最大规模的LLM安全评估心理测量学分析。研究发现拒绝严格性、真实性和情境危害三个可解释因子解释了模型间大部分差异；心理测量学选取的题项能以更低误差恢复完整基准分数，约10个自适应题项即可替代若干单个基准，评估成本降低97%-99%；该方法还可用于检测简单沙袋化和API背后模型的变更。研究建议前沿实验室和评估机构采用IRT作为现成工具。

## 关键点

- 对192个模型、8个安全基准进行IRT分析，发现拒绝严格性、真实性和情境危害三个可解释因子主导模型间差异。
- 心理测量学选取的题项比同规模随机子集误差更低，约10个自适应题项可复现多个基准得分，评估成本降低97%至99%。
- IRT可用于个体模型审计，能初步检测沙袋化行为和API背后的模型替换。

## 全文

Language models differ in how safely they behave and these differences are measured by safety benchmarks. But aggregated benchmark scores are hard to trust and interpret, because benchmarks duplicate one another, correlate heavily, and models may sandbag when they detect evaluation. To address these issues, we draw on Item Response Theory (IRT), a statistical toolkit for measuring these latents from performance on items with inferred psychometric properties. We fit IRT models to eight safety benchmarks across 192 language models, the largest psychometric analysis of LLM safety evaluations to date, and contribute three results. First, we find that three interpretable factors of refusal strictness, truthfulness, and contextual harm explain most of the variance between models across benchmarks. Second, psychometrically selected items recover full benchmark scores with lower error than random subsets of the same size, and roughly ten adaptively chosen items suffice for several individual benchmarks, cutting evaluation cost by 97-99%. Third, IRT supports audits of individual models, showing that it can be used to detect naive sandbagging and changes of model behind APIs. Overall, we show IRT is a ready-made toolkit for reading, reducing, and auditing safety benchmarks, which we recommend frontier labs and evaluators adopt.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_6732e2879262dd6b9d7444ff