---
title: "StartupBench: Benchmarking General-Purpose Agents on Market-Validated End-to-End Workflows"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.17800v1"
permalink: "https://feed.astralor.com/items/itm_c9653bff40fde1583698cf0e"
published: "2026-08-18T14:01:32.000Z"
score: 78
channel: "ai"
tags: ["arXiv","StartupBench","Benchmark","Large Language Models","Agent"]
---

# StartupBench: Benchmarking General-Purpose Agents on Market-Validated End-to-End Workflows

## AI commentary

> What's most valuable here is swapping benchmark task sources from 'researcher guessing' to 'market selection'—a pragmatic shift. For agent product teams, the 30% completion rate is a high-value signal: stop chasing academic benchmarks; the moat lies in truly end-to-end deliverable workflows. Pay special attention to its task taxonomy and rubric design; if data is open-sourced, it can directly reveal your own model's weaknesses.

## At a glance

Existing agent benchmarks mostly rely on researcher-selected tasks, so it remains unclear whether progress extends to work that real-world users actually demand. StartupBench systematically studies AI products with demonstrated adoption and extracts their workflows into deliverable-oriented end-to-end tasks with fine-grained rubrics. Under a unified agent harness, even the strongest model completes only about 30% of StartupBench, with complex instruction following and domain-specific expertise as major failure sources. This establishes an empirical measure of progress toward end-to-end completion of real-world user tasks.

## Key points

- StartupBench derives real-user workflows from market-validated AI startup products instead of researcher-selected tasks.
- Tasks are deliverable-oriented and evaluated with fine-grained rubrics capturing complex requirements.
- Even the strongest model completes only about 30% under a unified agent harness, showing market-validated workflows remain beyond current general-purpose agents.
- Complex instruction following and domain-specific expertise are major failure sources.

## Full text

Recent advances in Large Language Models(LLMs) and agents have substantially improved the ability of AI systems to execute complex tasks. Yet existing benchmarks largely rely on researcher-selected tasks, leaving uncertain whether such progress extends to the work that real-world users actually demand from AI systems. We introduce \textbf{StartupBench}, an E2E agent benchmark grounded in market-validated AI startup products. Rather than defining tasks from pre-defined assumptions about useful agent capabilities, we systematically study AI products with demonstrated adoption, together with their product workflows and users, to identify real-world tasks for which AI has established practical demand across diverse professional domains. We translate these workflows into complete deliverable-oriented tasks and evaluate them with fine-grained rubrics capturing their complex requirements. Across representative models evaluated under a unified agent harness, even the strongest model successfully completes only approximately 30\% of StartupBench, despite making substantial partial progress on many tasks. Further analysis identifies aspects like complex instruction following and domain-specific expertise as major sources of failure. Our results reveal that many market-validated workflows remain beyond the reliable capabilities of current general-purpose agents, establishing StartupBench as an empirical measure of progress toward E2E completions of real-world user tasks.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_c9653bff40fde1583698cf0e