---
title: "DataOrchestra：为预训练数据逐样本编排处理流程"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.24717v1"
permalink: "https://feed.astralor.com/items/itm_d5a14ed6d9f12e5f59fa04be"
published: "2026-07-27T17:54:12.000Z"
score: 76
channel: "tech"
tags: ["DataOrchestra","LLM","数据编排","预训练数据处理"]
---

# DataOrchestra：为预训练数据逐样本编排处理流程

## AI 点评

> 这篇论文最吸引我的是它把数据处理的粒度从语料层级降到样本层级，并通过学习来决策，这很符合当下“数据质量>数量”的趋势。对于正在优化预训练管线的研究团队和工程团队，DataOrchestra提供了一种即插即用的候选方案，尤其是数学、代码等对格式敏感的场景。不过编排器本身的训练开销和泛化性还需要更多验证。

## 速览

大型语言模型的预训练数据处理对下游性能至关重要，但现有方法常对整个语料或领域采用固定处理策略，缺乏对每个样本的适应性。本文提出DataOrchestra框架，统一多种处理操作，为每个样本动态编排个性化流水线：编排器决定丢弃、保留或清洗样本；对需要清洗的样本，进一步选择程序化编辑或基于LLM的重写操作，并生成具体指令。在0.5B至7B模型上的实验表明，DataOrchestra在11个基准上稳定优于单一处理方法，并在数学持续预训练中表现更佳，同时通过跳过不必要操作降低计算成本。

## 关键点

- DataOrchestra为每个预训练样本动态编排去重、清洗或保留决策，避免全局固定策略。
- 框架支持程序化编辑与多种LLM重写，并可自动生成具体指令，由对应工具模型执行。
- 在0.5B至7B参数规模、11项基准上稳定优于单一处理方法，数学持续预训练中提升更显著。
- 通过跳过不必要下游操作，在提升效果的同时降低处理算力消耗。

## 全文

Pretraining data processing is critical to the downstream performance of Large Language Models (LLMs). However, many existing approaches define a fixed processing strategy at the corpus or domain level and apply it uniformly to many examples, without adapting to the needs of each example. We propose DataOrchestra, a framework that unifies different processing operations and orchestrates an example-specific pipeline for each example. Given a chunk of pretraining data, an orchestrator decides whether to drop, untouch, or clean it. For a chunk to be cleaned, it selects one or more downstream operations, ranging from programmatic editing to different forms of LLM-based rewriting. For each rewriting step, it further generates a concrete instruction, which is executed by the corresponding downstream tool model. We pretrain models from 0.5B to 7B from scratch on web data processed by DataOrchestra and observe stable average gains over individual data-processing methods across 11 benchmarks. DataOrchestra is also effective for math continued pretraining and outperforms stronger processing baselines, while reducing processing compute by skipping unnecessary downstream operations.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_d5a14ed6d9f12e5f59fa04be