---
title: "Judge, Retrieve, or Abstain: Uncertainty-Guarded LLM Judging with Provable Risk Guarantees"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.17994v1"
permalink: "https://feed.astralor.com/items/itm_3aaf3c66fccdd44f17f66e5b"
published: "2026-08-18T16:42:02.000Z"
score: 78
channel: "ai"
tags: ["arXiv","Clopper-Pearson","LLM","Open-domain QA"]
---

# Judge, Retrieve, or Abstain: Uncertainty-Guarded LLM Judging with Provable Risk Guarantees

## AI commentary

> This paper hits the core issue of not trusting LLM judges on objective tasks. For teams building evaluation platforms or agent quality control, the framework can serve as a risk-control layer, eliminating blind trust in single outputs.

## At a glance

This paper proposes a risk-controlled framework for LLM judging, calibrating uncertainty thresholds on a held-out set to keep the false discovery rate among accepted verdicts below a user-specified level α with high probability, using finite-sample Clopper-Pearson intervals. When the parametric mode is not sufficiently confident, the instance is routed to a retrieval-augmented mode where the judge gathers web evidence and re-evaluates under a second calibrated threshold. The finite-sample guarantee carries over to this two-threshold routing without additional assumptions. Across open-domain QA benchmarks and judges of varying scales, the framework maintains target error rate while achieving substantially higher coverage than single-mode baselines.

## Key points

- Proposes a risk-controlled LLM judging framework that calibrates uncertainty thresholds to control the false discovery rate among accepted verdicts.
- Employs two-mode routing: when parametric mode is not confident, it switches to retrieval-augmented mode with a second calibrated threshold.
- Provides finite-sample guarantees using Clopper-Pearson intervals without additional assumptions.
- On open-domain QA benchmarks, it substantially improves coverage over single-mode baselines while maintaining target error rate.

## Full text

Using LLMs as judges has become standard practice for evaluating model outputs at scale. This is particularly common for subjective, open-ended tasks such as assessing helpfulness or alignment, where no single reference answer exists. However, objective tasks introduce a distinct reliability challenge for reference-free LLM judging. In the absence of a reference answer, the judge evaluates factual correctness either through its parametric knowledge or through tool augmentation. Although the former enables efficient evaluation, the judge may hallucinate or lack sufficient evidence for its verdict. Conversely, tool augmentation can provide additional evidence but introduces extra computational cost and requires an appropriate mechanism to determine when and how that evidence should be used reliably. More importantly, neither approach alone provides formal control over the risk of accepted verdicts or guarantees their reliability at a specified level. We propose a risk-controlled framework that calibrates uncertainty thresholds on a held-out set so that the false discovery rate among accepted verdicts remains below a user-specified level~$α$ with high probability, using finite-sample Clopper--Pearson intervals. When the parametric mode is not sufficiently confident, the instance is routed to a retrieval-augmented mode, where the judge gathers web evidence and re-evaluates the instance under a second calibrated threshold. The finite-sample guarantee carries over to this two-threshold routing without additional assumptions. Across open-domain QA benchmarks and judges of varying scales, the framework maintains the target error rate while achieving substantially higher coverage than single-mode baselines.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_3aaf3c66fccdd44f17f66e5b