---
title: "PredAct-Bench：在受控工具噪声下评估工具增强对话的新基准"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.02372v1"
permalink: "https://feed.astralor.com/items/itm_5cc395bebc6f0c3eaf5c30ef"
published: "2026-08-03T15:15:55.000Z"
score: 75
channel: "tech"
tags: ["PredAct-Bench","信任校准","大语言模型","工具噪声","英国开放大学"]
---

# PredAct-Bench：在受控工具噪声下评估工具增强对话的新基准

## AI 点评

> 工具噪声是AI系统落地的常态，但大多数评测都在“无菌环境”里比拼准确率。这篇论文把信任校准从单轮决策推进到多轮对话，甚至用人类实验验证了教师对AI建议的过度依赖——对教育科技和医疗AI产品团队来说，这是非常值得细读的路线图。与其继续堆模型分数，不如学学RAIR/RSR这类指标，让系统知道何时该把控制权交还给人类。

## 速览

大语言模型正被用于教育、医疗、金融等高风险的对话系统，但现有评测多假设工具输出完美。本文提出PredAct-Bench，在可控工具噪声下评估工具增强对话，以教育为测试床，引入RAIR与RSR等相对依赖指标，覆盖13个开源及闭源大模型、两个教育数据集，并开展教师与助教的人类研究。结果发现，当工具输出带噪声时，顶尖模型无法为教师提供足够可见性，可能导致过度依赖错误建议或幻觉。该基准旨在推动更可靠的AI决策支持系统。

## 关键点

- 提出PredAct-Bench基准，在受控工具噪声下评估工具增强对话，填补现有基准假设工具输出完美的空白。
- 引入RAIR和RSR指标，将信任校准从单轮决策扩展到多轮对话，衡量用户对AI与自身判断的相对依赖。
- 评估13个SOTA开源/闭源LLM在OULAD和PREDACT-CS数据集上的表现，并结合教师与助教的人类研究。
- 发现当前模型在工具噪声下无法提供足够可见性，易导致教育者过度依赖错误建议或幻觉，亟需改进。

## 全文

Large Language Models (LLMs) are increasingly deployed in task-oriented dialogue systems that support multi-step decision-making in high-stakes domains such as education, healthcare, and finance. However, existing benchmarks typically assume perfectly accurate tool outputs, overlooking the reality that deployed systems must operate with noisy tools and human decision-makers whose trust in the agent is itself uncertain. Such conditions are common in practice, for example, a clinician using a diagnostic prediction tool or an advisor relying on a model that forecasts student outcomes from historical records. We introduce PREDACTBENCH, a benchmark for evaluating dialogue agents paired with statistically imperfect tools, using education as a measurable testbed where ground truth outcomes and clear intervention decisions are available. First, we build a benchmark for AI-assisted human decision-making, where the AI uses noisy predictors to help guide a user. Second, we introduce episode-level Relative AI-Reliance (RAIR) and Relative self-reliance (RSR) metrics, extending prior trust calibration framework to multi-turn dialogue. Third, we evaluate 13 state-of-the-art closed and open source LLMs on two educational datasets, OULAD (real assessment trajectories from the UK Open University) and PREDACT-CS (60 courses with real final grade outcomes and synthetically generated weekly score trajectories), alongside a human study with instructors and teaching assistants. We find that when tools are noisy, SOTA models are supposed to provide visibility to teachers so that they do not over-rely on wrong suggestions or hallucinations, but current models fail to do that. We offer PREDACTBENCH to help build better LLMs as AI decision support systems to help teachers.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_5cc395bebc6f0c3eaf5c30ef