---
title: "MM-ToolSandBox：统一评估视觉工具调用代理的框架"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.11818v1"
permalink: "https://feed.astralor.com/items/itm_4eb173ac3316e95ab2a40c48"
published: "2026-07-13T17:13:09.000Z"
score: 79
channel: "tech"
tags: ["MM-ToolSandBox","多模态代理评估","苹果","视觉工具调用"]
---

# MM-ToolSandBox：统一评估视觉工具调用代理的框架

## AI 点评

> 这篇论文最值得看的是它用大量实验数据彻底打破了‘工具调用代理的难点在规划’的刻板印象。对模型开发者而言，51%的失败来自视觉感知而非决策，意味着与其死磕推理链，不如先去优化视觉编码器的细粒度精度。对产品经理和AI应用设计师来说，这个框架提供的场景生成管线是可以直接复用的，能大幅降低自身评估成本。

## 速览

MM-ToolSandBox 是苹果发布的新基准与评估框架，专为视觉驱动的工具调用代理设计。框架提供包含500+工具、覆盖16个应用领域的有状态执行环境，支持多图像、多轮任务，并模拟真实对话中的目标修订、错误纠正等场景。通过自动生成与人工验证，得到258个名义场景和50个UI变体。评估12个模型（从4B开源到前沿商业系统）后发现，最佳模型成功率不足50%，53%的失败源于图像信息提取错误，而非规划错误。小模型决策失误多，大模型感知不足，提示不同规模模型需要差异化改进方向。

## 关键点

- MM-ToolSandBox 覆盖500+工具和16个应用领域，支持多图像、多轮、有状态交互。
- 自动场景生成与人工验证结合，产生258个名义场景和50个UI交互变体。
- 当前最佳模型成功率不足50%，视觉精度是主要瓶颈，53%失败源于图像信息提取错误。
- 小模型决策能力差，大模型感知不足，需要针对不同规模模型采取不同改进策略。

## 全文

We introduce MM-ToolSandBox, a benchmark and evaluation framework for visually grounded tool-calling agents. The framework provides a stateful execution environment spanning 500+ tools across 16 application domains, supporting multi-image, multi-turn tasks where agents must ground progressively arriving visual inputs into executable tool calls while handling realistic conversational phenomena (goal revisions, error corrections, state mutations). An automated scenario generation pipeline produces diverse, visually grounded scenarios through information-flow-guided planning and multi-stage quality filtering, yielding 258 human-verified nominal scenarios and 50 variants targeting interactive UI applications. Evaluating 12 state-of-the-art models, from 4B open-weight to frontier proprietary systems, shows that current models still lack robust visual tool-calling capability: even the best model achieves below 50% success rate. Our failure analysis further reveals that visual precision, not only planning, is a primary bottleneck for capable models: 53% of failures stem from incorrect information extraction from images despite otherwise correct task workflows. A planning-to-precision crossover emerges with scale: smaller models fail at deciding what to do, while larger models fail at perceiving what they see, suggesting fundamentally different research directions for improving models at different capability levels. The framework and the benchmark are publicly available at https://github.com/apple/ml-mmtoolsandbox

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_4eb173ac3316e95ab2a40c48