---
title: "MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agents"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.11818v1"
permalink: "https://feed.astralor.com/items/itm_4eb173ac3316e95ab2a40c48"
published: "2026-07-13T17:13:09.000Z"
score: 79
channel: "tech"
tags: ["MM-ToolSandBox","Multimodal Agent Evaluation","Apple","Visual Tool-Calling"]
---

# MM-ToolSandBox: A Unified Framework for Evaluating Visual Tool-Calling Agents

## AI commentary

> This paper is worth reading because it shatters the stereotype that 'tool-calling agents' main challenge is planning' with substantial experimental data. For model developers, 51% of failures stem from visual perception rather than decision-making, meaning it's better to first optimize the fine-grained accuracy of vision encoders than to obsess over reasoning chains. For product managers and AI application designers, the scenario generation pipeline in this framework is directly reusable, greatly reducing their own evaluation costs.

## At a glance

MM-ToolSandBox is a new benchmark and evaluation framework from Apple for visually grounded tool-calling agents. It provides a stateful execution environment with 500+ tools across 16 domains, supporting multi-image, multi-turn tasks with realistic conversational phenomena such as goal revisions and error corrections. An automated pipeline generated 258 human-verified nominal scenarios and 50 UI variants. Evaluating 12 models (from 4B open-weight to frontier proprietary systems), the best model achieves under 50% success rate, with 53% of failures stemming from visual information extraction errors rather than planning. Small models falter in decision-making while large models struggle with perception, indicating different research directions for models at different scales.

## Key points

- MM-ToolSandBox covers 500+ tools across 16 domains with multi-image, multi-turn, stateful interaction. 
- Automated scenario generation with human verification yields 258 nominal scenarios and 50 UI variants. 
- The best model achieves under 50% success rate; visual precision is the main bottleneck, with 53% of failures coming from image information extraction errors.
- Small models struggle with decision-making, large models with perception, requiring different improvement strategies for different scales.

## Full text

We introduce MM-ToolSandBox, a benchmark and evaluation framework for visually grounded tool-calling agents. The framework provides a stateful execution environment spanning 500+ tools across 16 application domains, supporting multi-image, multi-turn tasks where agents must ground progressively arriving visual inputs into executable tool calls while handling realistic conversational phenomena (goal revisions, error corrections, state mutations). An automated scenario generation pipeline produces diverse, visually grounded scenarios through information-flow-guided planning and multi-stage quality filtering, yielding 258 human-verified nominal scenarios and 50 variants targeting interactive UI applications. Evaluating 12 state-of-the-art models, from 4B open-weight to frontier proprietary systems, shows that current models still lack robust visual tool-calling capability: even the best model achieves below 50% success rate. Our failure analysis further reveals that visual precision, not only planning, is a primary bottleneck for capable models: 53% of failures stem from incorrect information extraction from images despite otherwise correct task workflows. A planning-to-precision crossover emerges with scale: smaller models fail at deciding what to do, while larger models fail at perceiving what they see, suggesting fundamentally different research directions for improving models at different capability levels. The framework and the benchmark are publicly available at https://github.com/apple/ml-mmtoolsandbox

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_4eb173ac3316e95ab2a40c48