---
title: "Requential Coding：利用自生成训练数据突破模型压缩极限"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.11883v1"
permalink: "https://feed.astralor.com/items/itm_637cf9e3d1259704983cdb93"
published: "2026-07-13T17:58:50.000Z"
score: 79
channel: "ai"
tags: ["PAC-Bayes","Requential Coding","大语言模型","模型压缩","泛化理论"]
---

# Requential Coding：利用自生成训练数据突破模型压缩极限

## AI 点评

> 这篇论文提出的Requential Coding思路非常巧妙，不仅压缩效果惊人，而且统一了压缩与泛化的分析。对做模型压缩、大模型泛化或数据质量量化的研究者来说，绝对是近期必读之作。

## 速览

论文提出Requential Coding，一种基于教师模型为学生模型选择训练样本的压缩框架。与依赖参数量的量化方法或编码精确数据序列的预序列编码不同，Requential Coding只记录教师与学生不一致的样本选择，使得编码长度与参数和数据熵无关，通常比预序列编码短几个数量级，且优势随规模增长。该方法揭示了量化无法观察到的压缩现象：固定损失下，更大模型和集成模型压缩量更大；在PAC-Bayes界中为十亿参数大语言模型提供最先进的泛化保证，且界在计算最优尺度下随规模收紧。此外，它还能预测多轮训练中的逐渐过拟合，分离数据中的可学习信息与随机噪声，揭示低熵文本比高熵图像包含更多可学习结构。

## 关键点

- Requential Coding通过教师与学生的不一致来选择训练样本，编码长度与参数数量和数据熵无关。
- 该方法在大模型和集成模型上压缩量更大，揭示了参数量化无法捕捉的压缩现象。
- 在PAC-Bayes框架中为十亿参数大语言模型提供最新泛化保证，且保证随规模收紧。
- 可以区分数据集中的可学习信息与随机内容，表明低熵文本比高熵图像拥有更多可学习结构。

## 全文

Compression is fundamental to intelligence. A model that can represent its training data as a short code has discovered regularities that enable generalization. Large neural networks may learn functions far simpler than their parameter counts suggest, but it is challenging to construct codes that realize this simplicity. Parameter-based methods such as quantization produce code lengths that scale with model size, insensitive to how much information the parameters store. Prequential coding bypasses this issue by compressing the training trajectory, but codes the exact data sequence regardless of how much the model learns, yielding large codes when the data has high entropy. We introduce requential coding, where a teacher model selects training samples drawn from the student's own distribution. The student's code records only these selections, which cost bits only where teacher and student disagree. The resulting code length is independent of parameter count and data entropy, and often orders of magnitude shorter than the prequential counterpart, with an advantage that grows with scale. This compression sheds light on phenomena inaccessible to prior compressors. Holding loss fixed, larger models and ensembles compress to much smaller sizes despite more parameters. Plugged into a PAC-Bayes bound, the requential code yields state-of-the-art generalization guarantees for billion-parameter LLMs, outperforming bounds built on aggressive post-training quantization even granted zero error. The bound tightens with scale in the compute-optimal regime, as models become increasingly compressible relative to dataset size. The same code predicts that models gradually overfit when trained for multiple epochs. It also isolates the learnable information in a dataset from its unpredictable, random content, revealing that lower-entropy text holds far more learnable structure than higher-entropy image data.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_637cf9e3d1259704983cdb93