---
title: "UltraX: Refining Pre-Training Data at Scale with Adaptive Programmatic Editing"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.08646v1"
permalink: "https://feed.astralor.com/items/itm_9f9b8f17cff3e791e1ef8c24"
published: "2026-07-09T16:18:07.000Z"
score: 78
channel: "ai"
tags: ["UltraX","Data Refinement","Programmatic Editing","Scaling Law","Pre-training Data"]
---

# UltraX: Refining Pre-Training Data at Scale with Adaptive Programmatic Editing

## AI commentary

> This paper precisely addresses the core pain point of current LLM training—the data quality bottleneck. UltraX's innovation lies in replacing direct refinement with programmatic editing, retaining the efficiency of rule-based methods while absorbing the flexibility of LLMs. It's a highly practical framework worth attention for researchers and engineers working on pre-training data cleaning and model training.

## At a glance

As available training data approaches its physical limit, gains from Scaling Laws diminish, shifting focus to data quality. Existing refinement methods face trade-offs: rule-based approaches are rigid, LLM-based ones inefficient. This paper proposes UltraX, a framework completing the editing function space by introducing insertion, enabling fine-grained instance-level editing. It builds a reliable program-supervision pipeline with dataset-adaptive prompt optimization, line alignment mapping, dynamic context replacement, low-confidence filtering, and ratio-controlled sampling. During inference, sliding-window prediction, global operation aggregation, and post-processing ensure stability. Experiments show UltraX achieves the highest average performance across all corpora and matches or surpasses baselines with fewer training tokens, demonstrating superior data efficiency and refinement reliability.

## Key points

- UltraX completes the editing function space by introducing insertion for fine-grained instance-level editing.
- It proposes a supervision pipeline with dataset-adaptive prompt optimization, line alignment mapping, and dynamic context replacement.
- Low-confidence filtering and ratio-controlled sampling stabilize the training distribution.
- Sliding-window prediction and global operation aggregation improve reliability at scale.

## Full text

As available training data approaches its physical limit, gains from Scaling Laws have begun to diminish. Consequently, improving Large Language Models (LLMs) now depends less on data expansion and more on higher-quality data utilization. However, in the context of large-scale corpora, existing refinement methodologies face significant limitations in quality, efficiency, and reliability: Rule-based approaches are constrained by fixed heuristics and struggle with instance-level variations; LLM-based approaches improve quality but fail to meet the efficiency and reliability requirements of large-scale data processing. To address these challenges, we propose UltraX, a function-calling refinement framework for large-scale pre-training data that completes the editing function space by introducing insertion in addition to deletion and modification, enabling fine-grained instance-level editing. Specifically, UltraX builds a reliable program-supervision generation pipeline. In this pipeline, dataset-adaptive prompt optimization first guides an expert LLM to produce high-quality end-to-end refined texts, and Line Alignment Mapping and Dynamic Context Replacement then convert original-refined text pairs into structured program supervision. Meanwhile, UltraX improves supervision quality and stabilizes the training distribution with low-confidence example filtering and ratio-controlled sampling by operation combination. During inference and execution, it normalizes and validates model outputs through sliding-window prediction, global operation aggregation, and systematic post-processing, improving the stability and reliability of large-scale execution. Experiments show that UltraX achieves the highest average performance across all corpora and also matches or surpasses baselines with fewer training tokens, demonstrating stronger data efficiency and refinement reliability.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_9f9b8f17cff3e791e1ef8c24