---
title: "Follow the Features: SAE Feature-Guided Data-Efficient Instruction-Tuning of Large Language Models"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2611.04085v1"
permalink: "https://feed.astralor.com/items/itm_1b2a39f1b0daf96036fdbd91"
published: "2026-11-02T23:50:56.000Z"
score: 57
channel: "ai"
tags: ["LLM","RAG","Follow","Features","SAE Feature-Guided Data-Efficient","Instruction-Tuning","Large Language Models","Selecting"]
---

# Follow the Features: SAE Feature-Guided Data-Efficient Instruction-Tuning of Large Language Models

## 速览

Selecting small, effective instruction-tuning datasets requires identifying which aspects of the original data to retain. We introduce Follow the Features (FtF), a data-selection method that uses sparse autoencoder (SAE…

## 关键点

- Selecting small, effective instruction-tuning datasets requires identifying which aspects of the original data to retain.
- We introduce Follow the Features (FtF), a data-selection method that uses sparse autoencoder (SAE) features to characterize training data and guide subset construction.
- FtF ranks features using dataset-level activation statistics, scores samples by their expression of highly ranked features, and incorporates feature resampling to encourage diversity.

## 全文

Selecting small, effective instruction-tuning datasets requires identifying which aspects of the original data to retain. We introduce Follow the Features (FtF), a data-selection method that uses sparse autoencoder (SAE) features to characterize training data and guide subset construction. FtF ranks features using dataset-level activation statistics, scores samples by their expression of highly ranked features, and incorporates feature resampling to encourage diversity. The LLM-SAE pair used for selection is independent of the model being fine-tuned, enabling reuse across target models. We evaluate FtF on two model-dataset settings at a training budget of 100 samples with a fixed sequence length of 256 tokens. FtF achieves performance competitive with GRAPE and S2L, with relative rankings varying across settings. On Qwen 3 8B trained on Alpaca, FtF exceeds full-dataset training by 1.59 percentage points in the harmonic mean of an AlpacaEval-based win rate and Open LLM Leaderboard performance, although full-dataset training retains a higher Leaderboard score. These results support SAE feature statistics as a useful signal for data-efficient instruction tuning.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_1b2a39f1b0daf96036fdbd91