---
title: "UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.08768v1"
permalink: "https://feed.astralor.com/items/itm_06a3358d54809db66745e760"
published: "2026-07-09T17:59:32.000Z"
score: 79
channel: "ai"
tags: ["HKU-MMLab","UniClawBench","Proactive Agents","Benchmark","MLLM","Large Language Models"]
---

# UniClawBench: A Universal Benchmark for Proactive Agents on Real-World Tasks

## AI commentary

> This paper's UniClawBench is exactly what we urgently need—it stops assuming perfect environments and throws agents into real Docker containers, testing actual abilities with fine-grained checkpoints. For those working on agent frameworks and model evaluation, this is a must-read reference benchmark.

## At a glance

UniClawBench is the first capability-driven benchmark designed to evaluate proactive agents operating in dynamic real-world settings. It builds 400 bilingual real-world tasks around five foundational model capabilities: Skill Usage, Exploration, Long-Context Reasoning, Multimodal Understanding, and Cross-Platform Coordination. Unlike static benchmarks, it evaluates agents in live Docker containers with fine-grained step-by-step checkpoints and a closed-loop evaluation strategy that simulates multi-turn human feedback. The study compares multiple agent frameworks to reveal how base model capabilities and framework designs jointly shape real-world performance.

## Key points

- Proposes five core capability dimensions to systematically evaluate proactive agents in real-world scenarios.
- Uses live Docker containers and step-by-step completion checkpoints instead of static pre-recorded answers.
- Designs a closed-loop evaluation strategy simulating realistic multi-turn user feedback without leaking grading criteria.
- Disentangles base model capabilities from framework-level design choices through cross-model and cross-framework comparisons.

## Full text

The rapid development of large language models and multimodal large language models has accelerated the emergence of proactive agents capable of operating everyday tools and assisting users in real-world environments. However, existing benchmarks struggle to evaluate such agents effectively, as they often rely on sandboxed environments and single-turn evaluation paradigms. Moreover, their scenario-based task taxonomies mix multiple model capabilities within the same task category, making it difficult to identify the root causes of agent failures. To address these limitations, we introduce UniClawBench, the first capability-driven benchmark designed to evaluate proactive agents in dynamic, real-world settings. UniClawBench is built around five foundational model capabilities: Skill Usage, Exploration, Long-Context Reasoning, Multimodal Understanding, and Cross-Platform Coordination. Based on these capabilities, we design 400 bilingual real-world tasks. Unlike previous benchmarks that rely on static, pre-recorded answers, our benchmark evaluates agents in live Docker containers using fine-grained, step-by-step completion checkpoints. Furthermore, we design a closed-loop evaluation strategy comprising an executor agent, a hidden supervisor agent, and a user agent to simulate realistic multi-turn human feedback without leaking grading criteria. To disentangle base model capabilities from framework-level design choices, we evaluate state-of-the-art models under multiple agent frameworks. Through comprehensive comparisons across both models and frameworks, we show how base model capabilities and agent framework designs jointly shape performance in real-world environments. To facilitate future research, we make our benchmark and code publicly available at https://github.com/HKU-MMLab/UniClawBench.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_06a3358d54809db66745e760