---
title: "PPL-Factory: Task-Aware and Budget-Aware Data Selection from Language Modeling to Reasoning"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.18199v1"
permalink: "https://feed.astralor.com/items/itm_1d6c7c7faf8e13e237c2029e"
published: "2026-07-20T17:38:38.000Z"
score: 74
channel: "tech"
tags: ["LLM","Task-Aware","Perplexity","Fine-tuning Efficiency","Data Selection"]
---

# PPL-Factory: Task-Aware and Budget-Aware Data Selection from Language Modeling to Reasoning

## AI commentary

> This paper is worth attention as it offers a plug-and-play data selection solution for efficient fine-tuning, especially for research teams and practitioners urgently needing to reduce training costs. Compared to complex selection strategies, PPL-Factory only requires perplexity computation, with low barrier and remarkable results.

## At a glance

PPL-Factory proposes a data selection framework combining task-aware perplexity scores with data budget-aware criteria for LLM fine-tuning. Experiments show it outperforms existing methods with only 1% training data, and exceeds full-data fine-tuning by 0.9 on GSM8K and 4.8 on MATH with 10% data. The method is simple, interpretable, and effectively reduces fine-tuning computational cost.

## Key points

- Existing data selection methods rely on indirect heuristics and struggle to generalize across tasks
- PPL-Factory uses task-aware perplexity scores to distinguish language modeling and reasoning objectives
- Incorporates data budget-aware criteria for flexible selection
- Achieves or surpasses full-data fine-tuning performance on GSM8K and MATH with minimal data

## Full text

Not all training samples contribute equally to large language model fine-tuning. Selecting informative training samples can reduce the computational cost while preserving downstream performance. Many existing data selection methods rely on indirect heuristics, such as data quality, diversity or reasoning trace length. However, the effectiveness of these fixed criteria is task-dependent and difficult to generalize across diverse downstream tasks. Perplexity-based data selection provides a simple and model-aware solution to estimate the sample difficulty, but existing approaches typically score the entire training sequence and ignore the difference in learning objectives of language modeling and reasoning tasks. In this paper, we propose PPL-Factory, a simple and interpretable data selection framework that combines task-aware perplexity-based scores and data budget-aware selection criteria. Experiments on GSM8K demonstrate that PPL-Factory outperforms other state-of-the-art data selection methods using only $1\%$ of the training set. With $10\%$ of the data, PPL-Factory exceeds full-data fine-tuning accuracy by 0.9 on GSM8K and 4.8 on MATH. Overall, our results demonstrate that task-aware and budget-aware perplexity-based selection provides an effective and applicable approach for efficient fine-tuning.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_1d6c7c7faf8e13e237c2029e