InsufficiencyBench: Evaluating LLM Legal Advice on Underspecified User Queries
Score
76首次提出查询侧信息不足的法律基准，填补评估空白，信息增量显著。
研究设计严谨，经执业律师标注，覆盖多领域多辖区，质量较高。
对法律AI产品开发者和安全研究者具有直接指导意义，实用性强。
紧跟法律AI应用趋势，针对当前模型薄弱环节，时效性强。
来源为arXiv预印本，尚未正式同行评审，但作者团队专业可信度较高。
Base 84.5 × T2 Source weight ×0.9 = 76
Curation line 70Selected
Why selected
The paper introduces an original benchmark with rigorous evaluation and high practical value, though limited to legal consultation, and is highly timely.
AI commentary
「This benchmark hits the core pain point of legal AI deployment: real user queries are often ambiguous and incomplete. For legal tech companies, the paper's value lies in its clear error taxonomy and reusable evaluation data, enabling targeted model improvements. For AI safety observers, it further demonstrates frontier models' weakness in expressing uncertainty.」
Quick overview
InsufficiencyBench is the first legal benchmark specifically designed to address query-side insufficiency in legal AI systems. It evaluates whether LLMs can recognize when a user query lacks legally material information, identify what exactly is missing, and refrain from drawing premature legal conclusions. The authors formalize a taxonomy of eight canonical missing-element categories across three structural failure modes: switch, gating, and fatal prerequisite. They construct 202 benchmark items derived from 58 base queries spanning six legal domains and 24 US jurisdictions, all annotated by practicing attorneys. In their evaluation of ten frontier models, none exceeded an F2 score of 0.46 on missing-element identification, and the median recall was only 0.44. The models either hedged indiscriminately or silently answered under fabricated presumptions. Importantly, no model was able to both identify and qualify responses to deficient queries while directly addressing complete ones. This work highlights a critical gap in current legal AI systems.
- First benchmark specifically evaluating underspecified queries in legal AI, formalizing 8 missing-element categories and 3 structural failure modes.
- Constructs 202 items spanning six legal domains and 24 US jurisdictions, annotated by practicing attorneys.
- All ten frontier models underperform, with the best F2 at 0.46 and median recall 0.44.
- Models either indiscriminately hedge or answer under fabricated assumptions, failing to handle complete and deficient queries simultaneously.
Full text
This is an AI-generated Chinese digest; use "View original" for the source text.
Related items
85
FeaturedSetoka: A Benchmark for Hierarchical User Understanding in Personalized Agents over Heterogeneous Data
Setoka：面向异构数据个性化Agent的层级用户理解基准
This paper is well worth attention, especially for researchers and engineers working on personalized agents, dialogue systems, and memory-augmented AI. It not only provides a benchmark but defines levels of user understanding from psychology, revealing systematic failures of current systems at higher levels. If you are designing memory-based agents, this paper reminds you that memory systems need to support cross-temporal, cross-source abstract reasoning, not just retrieval.
85
FeaturedBAND: Bayesian Network Distribution Regression for High-Dimensional Distribution Estimation – Breaking the Curse
BAND：高维分布估计的贝叶斯网络回归方法——打破维度诅咒
This paper is a must-read for anyone working on high-dimensional statistics, probabilistic modeling, and generative AI. It provides a simple yet powerful theoretical answer: sparsity can break the curse of dimensionality, and Bayesian networks offer a natural framework. Compared to black-box methods like VAEs and GANs, BAND is interpretable and provably convergent, especially valuable for industry needs like reliable confidence intervals. If you research time series forecasting or risk management, this paper may change how you approach high-dimensional density estimation.
84
FeaturedCycle-Consistent and Uncertainty-Aware Neural Surrogates for Tokamak Edge Plasmas
循环一致与不确定性感知的托卡马克边缘等离子体神经代理模型
This paper is worth attention because it demonstrates how advanced AI techniques (cycle consistency, uncertainty quantification) directly address a core computational bottleneck in nuclear fusion. For fusion researchers and engineers, it offers a practical tool to speed up simulations by orders of magnitude without sacrificing accuracy; for AI practitioners, it provides an excellent case of self-supervised learning and inverse problem solving in scientific applications.
83
FeaturedEquivariant Learning of a Transferable Three-Dimensional Classical Density Functional
等变学习可迁移的三维经典密度泛函
The most striking aspect of this paper is its direct coupling of machine learning to a core problem in statistical mechanics, using equivariant networks to learn a transferable density functional without any labels. For researchers in liquid physics or soft matter, this could be a new paradigm: do we need to run separate simulations for every new system anymore? For AI researchers, it also shows how to embed physical symmetries and variational principles into models rather than just fitting data. I recommend focusing on the methods, especially how equivariance and variational consistency are implemented—that's the real essence.
83
FeaturedComputational and Statistical Guarantees of the c-Rectified Flow
c-Rectified Flow 的计算与统计保证
This paper moves rectified flow from 'works in practice' to 'theoretically controllable,' which is especially important for those working on the underlying algorithms of generative models. If you are tuning pipelines like FLUX or SD3, understanding that ordinary rectified flow can fail when covariances do not commute may help you avoid hidden training pitfalls.
83
FeaturedLearning Deployable Manipulation Policies from High-Fidelity UMI Data Alone
从高保真UMI数据学习可部署操作策略
The most striking point of this paper is proving that pure robot-free data can match real teleoperation performance, breaking the convention of needing a small real-robot anchor for post-training. For research groups looking to reduce hardware costs and industrial robot application developers, HiFi-UMI's data production system and open-source dataset are directly usable resources. Pay attention to the engineering details of head-mounted SLAM and microsecond synchronization—they are key to achieving the reported accuracy.
83
FeaturedCapital Markets LLM Reliability Score: From Plausible to Bankable
资本市场LLM可靠性评分：从“看似合理”到“可银行化”
This paper is valuable for both finance and AI communities: it provides an actionable, workflow-oriented evaluation framework that's closer to real needs than generic benchmarks. If you're deploying LLMs in fintech or responsible for explainability/auditability, this is a must-read. The finding on decision usefulness is particularly striking—judges agree it's critical, yet models vary the most on it.
83
FeaturedWhere Should Optimizer State Live? Tiered State Allocation for Memory-Efficient Mixture-of-Experts Training
优化器状态应放在哪里？面向内存高效MoE训练的分层状态分配
This paper is highly valuable for teams training MoE models: it demonstrates that with careful optimizer state allocation, one can dramatically reduce memory without sacrificing accuracy—and even improve performance. It is especially relevant for labs or enterprises constrained by 40 GB or 80 GB GPU memory. The tiered strategy and ablation on momentum retention are worth studying for optimizing training frameworks like DeepSpeed or Megatron.