---
title: "DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.24717v1"
permalink: "https://feed.astralor.com/items/itm_d5a14ed6d9f12e5f59fa04be"
published: "2026-07-27T17:54:12.000Z"
score: 76
channel: "tech"
tags: ["DataOrchestra","LLM","Data Orchestration","Pretraining Data Processing"]
---

# DataOrchestra: Learning to Orchestrate Per-Example Curation of Pretraining Data

## AI commentary

> What excites me most about this paper is that it reduces the granularity of data processing from corpus level to per-example level and uses learning to make decisions, aligning well with the current trend of 'quality over quantity'. For research and engineering teams optimizing pretraining pipelines, DataOrchestra offers a plug-and-play candidate, especially for format-sensitive domains like math and code. However, the training cost and generalization of the orchestrator itself still need further validation.

## At a glance

Pretraining data processing is critical for downstream performance of large language models, but existing methods often apply fixed strategies at corpus or domain level without adapting to per-example needs. This paper proposes DataOrchestra, a framework that unifies multiple processing operations and dynamically orchestrates an example-specific pipeline: an orchestrator decides to drop, keep, or clean each chunk; for cleaning, it selects programmatic editing or LLM-based rewriting with concrete instructions. Experiments from 0.5B to 7B models show stable average gains over individual methods across 11 benchmarks, outperforming stronger baselines in math continued pretraining while reducing compute by skipping unnecessary operations.

## Key points

- DataOrchestra dynamically orchestrates per-example decisions (drop, keep, or clean) for pretraining data, avoiding one-size-fits-all strategies.
- The framework supports programmatic editing and various LLM-based rewriting with automatically generated instructions executed by tool models.
- Models from 0.5B to 7B show stable average gains over individual processing methods across 11 benchmarks, with larger gains in math continued pretraining.
- Processing compute is reduced by skipping unnecessary downstream operations without sacrificing performance.

## Full text

Pretraining data processing is critical to the downstream performance of Large Language Models (LLMs). However, many existing approaches define a fixed processing strategy at the corpus or domain level and apply it uniformly to many examples, without adapting to the needs of each example. We propose DataOrchestra, a framework that unifies different processing operations and orchestrates an example-specific pipeline for each example. Given a chunk of pretraining data, an orchestrator decides whether to drop, untouch, or clean it. For a chunk to be cleaned, it selects one or more downstream operations, ranging from programmatic editing to different forms of LLM-based rewriting. For each rewriting step, it further generates a concrete instruction, which is executed by the corresponding downstream tool model. We pretrain models from 0.5B to 7B from scratch on web data processed by DataOrchestra and observe stable average gains over individual data-processing methods across 11 benchmarks. DataOrchestra is also effective for math continued pretraining and outperforms stronger processing baselines, while reducing processing compute by skipping unnecessary downstream operations.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_d5a14ed6d9f12e5f59fa04be