---
title: "Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Data"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.11883v1"
permalink: "https://feed.astralor.com/items/itm_637cf9e3d1259704983cdb93"
published: "2026-07-13T17:58:50.000Z"
score: 79
channel: "ai"
tags: ["PAC-Bayes","Requential Coding","Large Language Models","Model Compression","Generalization Theory"]
---

# Requential Coding: Pushing the Limits of Model Compression with Self-Generated Training Data

## AI commentary

> This paper's Requential Coding idea is remarkably clever, achieving impressive compression results while unifying compression and generalization analysis. For researchers in model compression, large model generalization, or data quality quantification, this is a must-read recent work.

## At a glance

This paper introduces Requential Coding, a compression framework where a teacher model selects training samples for the student model. Unlike parameter-based methods or prequential coding that encode exact data sequences, Requential Coding records only the selections where teacher and student disagree, making code length independent of parameter count and data entropy, often orders of magnitude shorter, with the advantage growing with scale. It reveals compression phenomena unseen by prior methods: under fixed loss, larger models and ensembles compress to much smaller sizes. Plugged into a PAC-Bayes bound, it yields state-of-the-art generalization guarantees for billion-parameter LLMs, tightening with scale in the compute-optimal regime. It also predicts gradual overfitting across epochs and isolates learnable information from random noise, showing lower-entropy text holds far more learnable structure than higher-entropy images.

## Key points

- Requential Coding selects training samples based on teacher-student disagreement, achieving code length independent of parameter count and data entropy.
- The method compresses larger models and ensembles more, revealing compression phenomena invisible to parameter quantization.
- It provides state-of-the-art generalization guarantees for billion-parameter LLMs in a PAC-Bayes bound, tightening with scale.
- It distinguishes learnable information from random content in datasets, showing lower-entropy text has far more learnable structure than higher-entropy images.

## Full text

Compression is fundamental to intelligence. A model that can represent its training data as a short code has discovered regularities that enable generalization. Large neural networks may learn functions far simpler than their parameter counts suggest, but it is challenging to construct codes that realize this simplicity. Parameter-based methods such as quantization produce code lengths that scale with model size, insensitive to how much information the parameters store. Prequential coding bypasses this issue by compressing the training trajectory, but codes the exact data sequence regardless of how much the model learns, yielding large codes when the data has high entropy. We introduce requential coding, where a teacher model selects training samples drawn from the student's own distribution. The student's code records only these selections, which cost bits only where teacher and student disagree. The resulting code length is independent of parameter count and data entropy, and often orders of magnitude shorter than the prequential counterpart, with an advantage that grows with scale. This compression sheds light on phenomena inaccessible to prior compressors. Holding loss fixed, larger models and ensembles compress to much smaller sizes despite more parameters. Plugged into a PAC-Bayes bound, the requential code yields state-of-the-art generalization guarantees for billion-parameter LLMs, outperforming bounds built on aggressive post-training quantization even granted zero error. The bound tightens with scale in the compute-optimal regime, as models become increasingly compressible relative to dataset size. The same code predicts that models gradually overfit when trained for multiple epochs. It also isolates the learnable information in a dataset from its unpredictable, random content, revealing that lower-entropy text holds far more learnable structure than higher-entropy image data.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_637cf9e3d1259704983cdb93