---
title: "CausalMix：将数据混合视为语言模型训练的因果推断"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.01104v1"
permalink: "https://feed.astralor.com/items/itm_8b8206eadc49c0ff53479e22"
published: "2026-07-01T15:56:11.000Z"
score: 77
channel: "ai"
tags: ["CausalMix","Qwen2.5","因果推断","大语言模型","数据混合优化","条件平均处理效应"]
---

# CausalMix：将数据混合视为语言模型训练的因果推断

## AI 点评

> CausalMix 最让人兴奋的是它用因果推断打破了数据混合中的“静态假设魔咒”。对于正在做大规模预训练或持续训练的团队，这篇论文直接解决了“数据变了就要重调比例”的痛点。虽然长链思维数据上的验证还不算深入，但核心思路干净利落——值得所有关注数据配比自动优化的研究员优先读。

## 速览

CausalMix 提出将大语言模型训练中的数据混合优化建模为因果推断问题。通过将数据池统计特征视为协变量、域混合视为处理，在512次Qwen2.5-0.5B运行上拟合因果模型估计条件平均处理效应（CATE），然后外推至800K数据池的最优混合，并成功应用于7B模型训练。该方法无需静态数据分布假设，能动态推断状态依赖的最优混合，避免重新从头训练。实验表明CausalMix在多个下游任务上一致优于RegMix等基线，并提供了基于CATE的可视化分析框架。

## 关键点

- 将数据混合优化建模为因果推断问题，引入CATE估计，突破静态分布假设限制
- 在512次小模型运行上拟合因果模型，成功外推至更大数据池和7B模型训练
- 动态推断状态依赖最优混合，避免重新从头训练，支持从短文本扩展到长思维链数据
- 提供CATE解释器进行可视化分析，提升混合策略的可解释性

## 全文

In Large Language Model (LLM) training, data mixing plays a pivotal role in determining model performance. Recent methods optimize mixture weights via proxy models, but they rely on the assumption of static data distributions. As a result, when the underlying data pool shifts, these methods require costly retraining from scratch. This limitation restricts their ability to scale seamlessly from small settings to larger data pools and model sizes. In this paper, we propose CausalMix to address this limitation by casting data mixture optimization as a causal inference problem. We formulate the statistical features of the data pool as covariates and the domain mixture as the treatment. After fitting a causal model on 512 runs of Qwen2.5-0.5B to estimate the Conditional Average Treatment Effect (CATE), we extrapolate the optimal mixture for an 800K data pool and apply it to train a 7B model. Furthermore, we successfully generalize the framework to long chain-of-thought data on Qwen3-4B-Base. By leveraging causal modeling to isolate confounding biases, CausalMix dynamically infers state-dependent optimal data mixtures. Extensive experiments show that the mixture guided by CausalMix consistently improves performance across multiple downstream tasks, outperforming RegMix and other baselines. In addition, we use the CATE Interpreter to provide visual analysis of the learned mixing strategy. Overall, CausalMix offers a causal and interpretable framework for optimizing LLM data mixtures.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_8b8206eadc49c0ff53479e22