---
title: "Provable Limits and Certified Deferral for Verbalized Uncertainty in Small Language Models"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.05064v1"
permalink: "https://feed.astralor.com/items/itm_990698fca5916fdb794ddeb8"
published: "2026-08-05T17:11:04.000Z"
score: 78
channel: "ai"
tags: ["Small Language Models","Temperature Scaling","Confidence Calibration","Certified Deferral","Risk Control"]
---

# Provable Limits and Certified Deferral for Verbalized Uncertainty in Small Language Models

## AI commentary

> This paper’s value lies in turning 'the model says I don’t know' from a slogan into a verifiable engineering commitment. For teams deploying small models on edge devices or in private environments, the risk-certification procedure here offers a direct way to set safety thresholds instead of relying on intuition. Even if you don’t train models, this framework is a great reference when you need to demonstrate safety guarantees to compliance stakeholders.

## At a glance

This study systematically evaluates eleven instruction-tuned models from three families, ranging from 0.5B to 14B parameters, on ARC-Challenge and TruthfulQA with 25,168 local predictions, asking whether verbalized confidence can support risk-controlled deferral. The authors prove three theoretical results: strictly monotone calibration preserves the risk-coverage frontier and error-detection AUROC; temperature scaling cannot calibrate models whose confidence stays above one half while accuracy falls below it; and a Clopper-Pearson procedure converts a 200-question calibration set into a finite-sample risk certificate under an i.i.d. deployment assumption. Empirically, eight of 22 model-task pairs hit the temperature-scaling infeasibility floor within one percentage point of the predicted bound. Platt scaling reduces ECE to as low as 0.02, yet certified autonomy at a 20% risk budget is granted to only three model-task pairs and none at 10%. The paper also identifies and repairs an answer-ordering artifact in TruthfulQA's multiple-choice form. Calibration gives confidence semantics; certified deferral determines when small models are safe to use.

## Key points

- Theoretically, strictly monotone calibration preserves the risk-coverage frontier, while temperature scaling has an infeasibility boundary for models whose confidence is systematically overconfident.
- A Clopper-Pearson procedure can turn a 200-question calibration set into a finite-sample risk certificate under an i.i.d. deployment assumption.
- Empirically, despite Platt scaling reducing ECE to 0.02, only three model-task pairs achieve certified autonomy at a 20% risk budget, and none at 10%.

## Full text

Small open-weight language models increasingly run in private, offline, and cost-sensitive settings, where the key deployment question is not only what a model answers but when it should defer to a human. We study whether verbalized confidence can support risk-controlled deferral, evaluating eleven instruction-tuned models from three families, 0.5B to 14B parameters, on ARC-Challenge and TruthfulQA with 25,168 local predictions. Three theoretical results delimit what calibration can provide: strictly monotone calibration preserves the risk-coverage frontier and error-detection AUROC; temperature scaling cannot calibrate models whose confidence stays above one half while accuracy falls below it; and a Clopper-Pearson procedure converts a 200-question calibration set into a finite-sample risk certificate under an i.i.d. deployment assumption. Empirically, eight of 22 model-task pairs hit the temperature-scaling infeasibility floor within one percentage point of the predicted bound. Platt scaling reduces ECE to as low as 0.02, yet certified autonomy at a 20% risk budget is granted to only three model-task pairs and to none at 10%. We also identify and repair an answer-ordering artifact in the multiple-choice form of TruthfulQA. Calibration gives confidence semantics; certified deferral determines when small models are safe to use.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_990698fca5916fdb794ddeb8