---
title: "InsufficiencyBench: Evaluating LLM Legal Advice on Underspecified User Queries"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.20220v1"
permalink: "https://feed.astralor.com/items/itm_67969febfbbe2af8090f1875"
published: "2026-08-20T16:14:47.000Z"
score: 76
channel: "ai"
tags: ["InsufficiencyBench","Benchmark","Large Language Models","Legal AI"]
---

# InsufficiencyBench: Evaluating LLM Legal Advice on Underspecified User Queries

## AI commentary

> This benchmark hits the core pain point of legal AI deployment: real user queries are often ambiguous and incomplete. For legal tech companies, the paper's value lies in its clear error taxonomy and reusable evaluation data, enabling targeted model improvements. For AI safety observers, it further demonstrates frontier models' weakness in expressing uncertainty.

## At a glance

InsufficiencyBench is the first legal benchmark specifically designed to address query-side insufficiency in legal AI systems. It evaluates whether LLMs can recognize when a user query lacks legally material information, identify what exactly is missing, and refrain from drawing premature legal conclusions. The authors formalize a taxonomy of eight canonical missing-element categories across three structural failure modes: switch, gating, and fatal prerequisite. They construct 202 benchmark items derived from 58 base queries spanning six legal domains and 24 US jurisdictions, all annotated by practicing attorneys. In their evaluation of ten frontier models, none exceeded an F2 score of 0.46 on missing-element identification, and the median recall was only 0.44. The models either hedged indiscriminately or silently answered under fabricated presumptions. Importantly, no model was able to both identify and qualify responses to deficient queries while directly addressing complete ones. This work highlights a critical gap in current legal AI systems.

## Key points

- First benchmark specifically evaluating underspecified queries in legal AI, formalizing 8 missing-element categories and 3 structural failure modes.
- Constructs 202 items spanning six legal domains and 24 US jurisdictions, annotated by practicing attorneys.
- All ten frontier models underperform, with the best F2 at 0.46 and median recall 0.44.
- Models either indiscriminately hedge or answer under fabricated assumptions, failing to handle complete and deficient queries simultaneously.

## Full text

Legal AI systems are increasingly used to answer legal questions, yet existing benchmarks assume queries arrive fully specified. In practice, users omit facts that materially determine the legal outcome. We introduce InsufficiencyBench, the first legal benchmark targeting query-side insufficiency: whether a model recognizes when a query lacks legally material information, identifies what is missing, and refrains from premature conclusions. We formalize a taxonomy of eight canonical missing-element categories across three structural failure modes---switch, gating, and fatal prerequisite--- and construct 202 benchmark items (58 base queries, 144 deficient variants) spanning six legal domains and 24 US jurisdictions and annotated by practising attorneys. Evaluating ten frontier models, we find that no model exceeds F2 = 0.46 on missing-element identification and that the median recall is 0.44. Models either hedge indiscriminately or answer silently under fabricated presumptions. No model both identifies and qualifies responses to deficient queries while directly addressing complete ones.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_67969febfbbe2af8090f1875