---
title: "小型语言模型口头化不确定性的可证明极限与认证推迟"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.05064v1"
permalink: "https://feed.astralor.com/items/itm_990698fca5916fdb794ddeb8"
published: "2026-08-05T17:11:04.000Z"
score: 78
channel: "ai"
tags: ["小型语言模型","温度缩放","置信度校准","认证推迟","风险控制"]
---

# 小型语言模型口头化不确定性的可证明极限与认证推迟

## AI 点评

> 这篇论文的价值在于把‘模型说我不知道’从口号变成可验证的工程承诺。对于在边缘设备或私有化环境中部署小模型的团队来说，文中给出的风险证书方法可以直接用来自定义安全阈值，而不是凭感觉决定是否放行。即使你不做模型训练，只要你需要向合规方证明模型行为的安全性，这篇文章的框架也是很好的参考。

## 速览

这项研究系统评估了三个家族11个参数为0.5B到14B的指令微调小模型，在ARC-Challenge和TruthfulQA上共2.5万余次预测，考察口头化置信度能否支持风险受控的推迟。理论证明严格单调校准保持风险覆盖率前沿，但温度缩放存在不可行下限；Clopper-Pearson方法可将200题校准集变成有限样本风险证书。实验显示Platt缩放可将ECE降至0.02，但仅有少数模型在20%风险预算下获得认证自主权，10%时则没有。研究还修复了TruthfulQA的答案排序伪影。

## 关键点

- 理论上证明严格单调校准不会改变风险覆盖率前沿，但温度缩放对某些模型存在不可校准的失效边界。
- 使用Clopper-Pearson方法，仅需200个校准问题即可生成有限样本风险证书。
- 实验发现，尽管Platt缩放能把ECE降到0.02，但多数小模型在20%风险预算下仍无法获得认证自主权，10%则全部失败。

## 全文

Small open-weight language models increasingly run in private, offline, and cost-sensitive settings, where the key deployment question is not only what a model answers but when it should defer to a human. We study whether verbalized confidence can support risk-controlled deferral, evaluating eleven instruction-tuned models from three families, 0.5B to 14B parameters, on ARC-Challenge and TruthfulQA with 25,168 local predictions. Three theoretical results delimit what calibration can provide: strictly monotone calibration preserves the risk-coverage frontier and error-detection AUROC; temperature scaling cannot calibrate models whose confidence stays above one half while accuracy falls below it; and a Clopper-Pearson procedure converts a 200-question calibration set into a finite-sample risk certificate under an i.i.d. deployment assumption. Empirically, eight of 22 model-task pairs hit the temperature-scaling infeasibility floor within one percentage point of the predicted bound. Platt scaling reduces ECE to as low as 0.02, yet certified autonomy at a 20% risk budget is granted to only three model-task pairs and to none at 10%. We also identify and repair an answer-ordering artifact in the multiple-choice form of TruthfulQA. Calibration gives confidence semantics; certified deferral determines when small models are safe to use.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_990698fca5916fdb794ddeb8