---
title: "StartupBench：基于市场验证端到端工作流的新型智能体基准"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.17800v1"
permalink: "https://feed.astralor.com/items/itm_c9653bff40fde1583698cf0e"
published: "2026-08-18T14:01:32.000Z"
score: 78
channel: "ai"
tags: ["arXiv","StartupBench","基准测试","大语言模型","智能体"]
---

# StartupBench：基于市场验证端到端工作流的新型智能体基准

## AI 点评

> 这个基准最值得看的地方在于它把评测任务来源从“研究者猜”换成“市场挑”，思路很务实。对做Agent产品的人来说，30%的完成率其实是个高价值信号：别只刷学术benchmark，真正能端到端交付的工作流才是护城河。建议重点关注其任务分类和rubric设计，后续若开源数据，可直接用来检验自家模型短板。

## 速览

现有智能体基准多由研究者主观选任务，难以反映真实用户需求。StartupBench从有真实市场采纳的AI初创产品中提取工作流，构建端到端、以交付物为导向的任务与细粒度评分标准。统一测试下，最强模型完成率仅约30%，复杂指令遵循和领域专业知识是主要失败点。该基准为衡量通用智能体完成真实世界任务提供实证标尺。

## 关键点

- StartupBench从有市场验证的AI初创产品中提取真实用户工作流，而非研究者主观设定任务。
- 任务以完整交付物为导向，使用细粒度评分标准评估复杂要求。
- 最强模型在统一智能体框架下完成率仅约30%，显示市场验证工作流仍超出当前通用智能体可靠能力。
- 复杂指令遵循与领域专业知识是主要失败原因。

## 全文

Recent advances in Large Language Models(LLMs) and agents have substantially improved the ability of AI systems to execute complex tasks. Yet existing benchmarks largely rely on researcher-selected tasks, leaving uncertain whether such progress extends to the work that real-world users actually demand from AI systems. We introduce \textbf{StartupBench}, an E2E agent benchmark grounded in market-validated AI startup products. Rather than defining tasks from pre-defined assumptions about useful agent capabilities, we systematically study AI products with demonstrated adoption, together with their product workflows and users, to identify real-world tasks for which AI has established practical demand across diverse professional domains. We translate these workflows into complete deliverable-oriented tasks and evaluate them with fine-grained rubrics capturing their complex requirements. Across representative models evaluated under a unified agent harness, even the strongest model successfully completes only approximately 30\% of StartupBench, despite making substantial partial progress on many tasks. Further analysis identifies aspects like complex instruction following and domain-specific expertise as major sources of failure. Our results reveal that many market-validated workflows remain beyond the reliable capabilities of current general-purpose agents, establishing StartupBench as an empirical measure of progress toward E2E completions of real-world user tasks.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_c9653bff40fde1583698cf0e