---
title: "OmniaBench: Benchmarking General AI Agents Across Diverse Scenarios"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.14989v1"
permalink: "https://feed.astralor.com/items/itm_eb6117d3f664b16623e2bd00"
published: "2026-07-16T13:38:07.000Z"
score: 81
channel: "ai"
tags: ["AI Evaluation","Claude Sonnet 5","GPT-5.6 Sol","OmniaBench","Benchmark","Generalist Agent"]
---

# OmniaBench: Benchmarking General AI Agents Across Diverse Scenarios

## AI commentary

> This paper is a must-read for any researcher or engineer working on LLM agent development or evaluation. It not only provides a rich task set covering ToC, ToB, and ToE, but also uses a ten-dimensional capability taxonomy and difficulty factors to precisely pinpoint current model weaknesses—planning, constraint maintenance, and adaptive correction. If you want to know where your agent's upper bound lies in real-world scenarios, OmniaBench is an excellent litmus test.

## At a glance

Large language models are evolving from text generators into general agents, yet existing benchmarks focus on limited scenarios. OmniaBench covers ToC, ToB, and ToE with 90 level-1 and 354 level-2 domains, synthesizing 1,431 tasks via four routes. It introduces a ten-dimensional capability taxonomy and eight compositional atomic difficulty factors. State-of-the-art models like Claude-Sonnet-5 and GPT-5.6-Sol achieve Overall Pass@1 scores of only 58.54 and 57.14, revealing persistent limitations in planning, constraint maintenance, and adaptive correction.

## Key points

- OmniaBench constructs a hierarchical scenario knowledge taxonomy covering ToC/ToB/ToE from app stores, product documents, etc.
- Synthesizes 1,431 single-turn and multi-turn tasks via four complementary routes: DAG, DAG-S, Solver, and Program.
- Introduces a ten-dimensional capability taxonomy and eight compositional atomic difficulty factors for fine-grained evaluation.
- State-of-the-art models show persistent limitations in planning, constraint maintenance, and adaptive correction, with highest pass rates around 58%.

## Full text

Large language models are increasingly evolving from text generators into general agents capable of understanding user requests, invoking external tools, and completing complex tasks through interaction. However, existing agent benchmarks often focus on limited scenarios, tool ecosystems, or interaction formats, making it difficult to systematically characterize model capabilities across heterogeneous application settings. We introduce OmniaBench, a benchmark for evaluating general agents across diverse scenarios with explicit state spaces. We derive application-oriented scenario knowledge from app stores, product documents, industry resources, Web retrieval, and human refinement, forming a hierarchical taxonomy that spans ToC, ToB and ToE with 90 level-1 and 354 level-2 domains. Based on this taxonomy, we construct executable environments and synthesize single-turn and multi-turn tasks through four complementary routes: DAG, DAG-S, Solver, and Program. OmniaBench further introduces a ten-dimensional capability taxonomy and eight compositional atomic difficulty factors to support fine-grained evaluation and analysis. The resulting dataset contains 1,431 tasks, together with a challenging subset of 644 tasks designed to reduce evaluation cost and mitigate potential contamination of the full set after public release. The bench presents substantial challenges to current frontier models, with even Claude-Sonnet-5 and GPT-5.6-Sol achieving Overall Pass@1 scores of only 58.54 and 57.14, respectively. Further analyses reveal clear differences across domains and capabilities, as well as persistent limitations in planning, constraint maintenance, and adaptive correction. OmniaBench provides a broad and diagnostic benchmark for characterizing the capability boundaries of general agents.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_eb6117d3f664b16623e2bd00