PredAct-Bench: Benchmarking Tool-Augmented Dialogue under Controlled Tool Noise
Score
75现有基准多假设工具输出完美，该论文率先系统评估工具噪声下的对话信任校准，填补空白。
包含新基准、新指标、13个模型评估与人类实验，方法扎实，结果有说服力。
为教育、医疗等高风险场景的AI决策支持系统提供可直接使用的评估工具与改进方向。
针对当前LLM工具增强落地中的关键短板，符合近期AI可靠性研究趋势。
来自arXiv预印本，尚未经过同行评审，但作者机构与实验设计可信。
Base 83.5 × T2 Source weight ×0.9 = 75
Curation line 70Selected
Why selected
The paper delivers substantial contributions in dialogue evaluation under tool noise, spanning benchmark design, novel metrics, multi-model evaluation, and human studies.
AI commentary
「Tool noise is the norm in real deployments, yet most benchmarks evaluate LLMs in a sterile environment of perfect tool outputs. This paper advances trust calibration from single decisions to multi-turn dialogue, and even demonstrates with human studies that teachers over-rely on AI suggestions — a must-read for edtech and health AI product teams. Rather than chasing higher accuracy, RAIR/RSR-style metrics may be the better way to decide when the system should hand control back to humans.」
Quick overview
Large language models are increasingly deployed in task-oriented dialogue systems for high-stakes domains such as education, healthcare, and finance, yet existing benchmarks typically assume perfectly accurate tool outputs. This paper introduces PredAct-Bench, a benchmark for evaluating tool-augmented dialogue under controlled tool noise, using education as a measurable testbed. The authors propose episode-level Relative AI-Reliance (RAIR) and Relative Self-Reliance (RSR) metrics, extending trust calibration to multi-turn dialogue. They evaluate 13 state-of-the-art closed and open-source LLMs on two educational datasets, OULAD and PREDACT-CS, alongside a human study with instructors and teaching assistants. The results show that when tools are noisy, current SOTA models fail to provide sufficient visibility to teachers, leading to over-reliance on wrong suggestions or hallucinations. The benchmark aims to support building better LLMs as AI decision-support systems for educators.
- Introduces PredAct-Bench, a benchmark for evaluating tool-augmented dialogue under controlled tool noise, addressing the gap left by benchmarks assuming perfect tool outputs.
- Proposes RAIR and RSR metrics that extend trust calibration from single decisions to multi-turn dialogue, measuring users' relative reliance on AI versus themselves.
- Evaluates 13 state-of-the-art closed and open-source LLMs on OULAD and PREDACT-CS, complemented by a human study with instructors and teaching assistants.
- Finds that current models fail to provide sufficient visibility under tool noise, leading educators to over-rely on wrong suggestions or hallucinations.
Full text
This is an AI-generated Chinese digest; use "View original" for the source text.
Related items
FeaturedDeepSeek V4 Doubles Token Share on OpenRouter, Driven by Agentic Workloads
DeepSeek V4 在 OpenRouter 上的代币份额翻倍，智能体工作负载成主要驱动力
The rapid adoption of DeepSeek V4 signals a strong market preference for high-performance, low-cost inference models, especially as agentic use cases reshape AI deployment.
91
FeaturedEnabling Two Settings Triples ARC-AGI-3 Benchmark Scores
启用两项设置使ARC-AGI-3基准测试分数提升三倍
This finding is crucial! It shows that model performance depends not only on training but also on inference-time configuration. For AI engineers and researchers, this is a must-know optimization direction. Teams deploying models on resource-constrained devices in particular could find a low-cost way to boost capabilities.
91
FeaturedGPT-5.6: Fusing Frontier Intelligence with Frontier Efficiency
GPT-5.6：前沿智能与前沿效率的融合
This news is worth watching as it marks a shift from pure parameter scaling to efficiency competition. For AI developers and business decision-makers, GPT-5.6 means higher performance at lower cost, especially for long-context or agentic tasks previously hindered by inference expenses. OpenAI's move may redefine what 'frontier' means.
91
FeaturedLFM2.5-Encoders: New Models for Fast Long-Context Inference on CPU
LFM2.5-Encoder：面向CPU快速长上下文推理的新模型
This blog post catches my eye because it directly solves the pain point of running long-context LLMs on CPU. For teams or individual developers without high-end GPUs, the LFM2.5-Encoder series offers a ready-to-use open-source solution. Highly recommended for those working on RAG, document search, or code analysis—now a standard server can handle million-token documents.
91
FeaturedGPT-Red: OpenAI Automates Red Teaming via Self-Play to Boost Model Robustness
GPT-Red：OpenAI通过自我对弈实现AI红队自动化，提升模型鲁棒性
GPT-Red marks a milestone from human to automated red-teaming—essential for safety engineers and alignment researchers. While reducing cost, it also raises ethical questions about automated attacks, worth pondering for everyone in AI governance.
91
FeaturedOpenAI Launches GPT-5.5 Bio Bug Bounty Program
OpenAI推出GPT-5.5生物漏洞赏金计划
This program deserves attention as it pioneers the extension of AI bug bounty to biosecurity, offering a new research direction for the security community and a practical training ground for AI risk governance. It is particularly important for model safety researchers, biosecurity experts, and practitioners concerned about dual-use AI risks.
91
FeaturedvLLM Introduces Native-Speed Transformers Modeling Backend for Enhanced Inference Performance
vLLM推出原生速度Transformers建模后端，大幅提升推理性能
This update is critical for AI engineering teams: eliminating format conversion speeds up model iteration, and 2-5x speedup directly reduces GPU costs. If you're building LLM online services or batch inference, this is practically a must-read technical update.
90
FeaturedGoogle DeepMind Introduces SL2T, a Sign Language-to-Text Model for Deaf Users
Google DeepMind 发布手语转文本模型 SL2T，助力聋人用户
This news matters not only as a technical breakthrough but also as a sign that AI is beginning to address the needs of marginalized groups. For developers, the architecture and technical approach of SL2T merit deep study; for accessibility advocates, it's a key case for advancing industry standards.