---
title: "TraceML: An Empirical Analysis of Human-Agent Planning in Machine Learning Development"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.26086v1"
permalink: "https://feed.astralor.com/items/itm_a363e546de7c6fe0491347a6"
published: "2026-08-26T17:50:13.000Z"
score: 76
channel: "ai"
tags: ["Kaggle","MLEvolve","OpenAI Codex","TraceML","Agent Planning","ML Development"]
---

# TraceML: An Empirical Analysis of Human-Agent Planning in Machine Learning Development

## AI commentary

> What makes this paper worth reading is that it turns the human-agent gap from scores into behavioral trajectories, letting us see exactly how agents get stuck in narrow loops over long tasks. For engineers working on agent reasoning, planning, or AutoML, this open-source dataset may be more valuable than a pile of benchmarks. Pair it with human trajectories and prompt design becomes far more grounded.

## At a glance

Large language models write correct code for isolated problems but remain far weaker at autonomous machine-learning development. Outcome-based benchmarks record the gap but not its cause. TraceML pairs human and agent work under a version-level schema: 4,465 human Kaggle trajectories across 134 competitions, plus 430 paired human and 207 agent trajectories from seven competitions worked by two agent scaffolds. Every code version carries score, timestamp, and labeled action, intent, edit size, and score effect. Experts alternate strategies and revisit abandoned approaches, while agents collapse into narrow loops. A short planning prompt distilled from human practice moves behaviors closer to humans but leaves the effort profile agent-shaped.

## Key points

- TraceML includes 4,465 human Kaggle trajectories and 207 agent trajectories across 134 competitions, with every version labeled for score, timestamp, action, and intent.
- Experts flexibly switch among data work, validation, model changes, and ensembling, and revisit abandoned approaches, while both agent scaffolds collapse into narrow loops.
- A short planning prompt distilled from human practice improves scores but leaves the effort profile agent-shaped, showing instruction closes only part of the gap.
- The corpus, schema, labelers, and extraction pipeline are publicly released for further research.

## Full text

Large language models write correct code for isolated problems but remain far weaker at autonomous machine-learning development, where an agent must revise data pipelines, models, and validation over hours of feedback, and on most competitions still finishes below strong human competitors. Outcome-based benchmarks record this gap but not its cause, because they grade the final submission and discard the development process behind it. We introduce TraceML, which pairs human and agent work on the same competitions under one version-level schema: 4,465 human Kaggle trajectories across 134 competitions, seven of which are also worked by two agent scaffolds, giving 430 paired human and 207 agent trajectories. Every code version carries its score, its timestamp, and labels for the action taken, its intent, the edit size, and the score effect. Read this way, the gap becomes concrete. Experts alternate data work, validation, model changes, and ensembling, and return to approaches they had set aside. Each agent scaffold instead collapses into a narrow loop: Codex spends its steps re-weighting ensembles and tuning submissions, MLEvolve mutates its model in place, and neither pivots at the human rate nor reopens abandoned work. A short planning prompt distilled from human practice moves the behaviors it names toward the human profile and lifts scores, but the effort profile stays agent-shaped: instruction closes only the part of the gap that reduces to instructions. We release the corpus, the schema, the labelers, and the extraction pipeline at https://huggingface.co/datasets/jerryyan/TraceML.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_a363e546de7c6fe0491347a6