---
title: "UltraX：基于自适应程序化编辑的大规模预训练数据精炼框架"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.08646v1"
permalink: "https://feed.astralor.com/items/itm_9f9b8f17cff3e791e1ef8c24"
published: "2026-07-09T16:18:07.000Z"
score: 78
channel: "ai"
tags: ["UltraX","数据精炼","程序化编辑","缩放定律","预训练数据"]
---

# UltraX：基于自适应程序化编辑的大规模预训练数据精炼框架

## AI 点评

> 这篇论文精准切中了当前大模型训练的核心痛点——数据质量瓶颈。UltraX的创新之处在于用程序化编辑替代了传统的直接精炼，既保留了规则方法的高效性，又吸收了LLM的灵活性。对于从事预训练数据清洗和模型训练的研究者与工程师来说，这是一个非常值得关注的实用框架。

## 速览

随着可用训练数据接近物理极限，缩放定律带来的收益递减，提升大语言模型的关键转向数据质量。现有精炼方法在质量、效率和可靠性方面各有局限：规则方法僵化，LLM方法效率不足。本文提出UltraX框架，通过引入插入操作完善编辑函数空间，实现细粒度实例级编辑。该框架构建了可靠的程序监督生成管线，包括数据集自适应提示优化、行对齐映射、动态上下文替换、低置信度过滤和按操作组合比例采样，并在推理执行阶段通过滑动窗口预测、全局操作聚合和后处理增强稳定性。实验表明UltraX在所有语料上平均性能最高，且以更少训练token达到或超越基线，展现了更强的数据效率和精炼可靠性。

## 关键点

- UltraX通过引入插入操作完善编辑函数空间，实现细粒度实例级编辑。
- 提出数据集自适应提示优化、行对齐映射和动态上下文替换的监督生成管线。
- 采用低置信度过滤和按操作组合比例采样稳定训练分布。
- 滑动窗口预测与全局操作聚合提升大规模执行可靠性。

## 全文

As available training data approaches its physical limit, gains from Scaling Laws have begun to diminish. Consequently, improving Large Language Models (LLMs) now depends less on data expansion and more on higher-quality data utilization. However, in the context of large-scale corpora, existing refinement methodologies face significant limitations in quality, efficiency, and reliability: Rule-based approaches are constrained by fixed heuristics and struggle with instance-level variations; LLM-based approaches improve quality but fail to meet the efficiency and reliability requirements of large-scale data processing. To address these challenges, we propose UltraX, a function-calling refinement framework for large-scale pre-training data that completes the editing function space by introducing insertion in addition to deletion and modification, enabling fine-grained instance-level editing. Specifically, UltraX builds a reliable program-supervision generation pipeline. In this pipeline, dataset-adaptive prompt optimization first guides an expert LLM to produce high-quality end-to-end refined texts, and Line Alignment Mapping and Dynamic Context Replacement then convert original-refined text pairs into structured program supervision. Meanwhile, UltraX improves supervision quality and stabilizes the training distribution with low-confidence example filtering and ratio-controlled sampling by operation combination. During inference and execution, it normalizes and validates model outputs through sliding-window prediction, global operation aggregation, and systematic post-processing, improving the stability and reliability of large-scale execution. Experiments show that UltraX achieves the highest average performance across all corpora and also matches or surpasses baselines with fewer training tokens, demonstrating stronger data efficiency and refinement reliability.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_9f9b8f17cff3e791e1ef8c24