---
title: "通过变换编码视角压缩KV缓存：注意力感知的AATC方法"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.14191v1"
permalink: "https://feed.astralor.com/items/itm_84889d4162a0bc7a6cdd9bd6"
published: "2026-08-14T11:08:01.000Z"
score: 78
channel: "tech"
tags: ["AATC","arXiv","KV缓存","变换编码","量化压缩","长上下文推理"]
---

# 通过变换编码视角压缩KV缓存：注意力感知的AATC方法

## AI 点评

> 这篇论文的价值在于跳出单纯的量化误差视角，真正从注意力机制的信号传播特性出发设计压缩方法。对做长上下文推理和LLM部署的工程师而言，AATC可能在内存严重受限的场景下成为关键工具，值得大家关注它后续的工程化与扩展性。

## 速览

KV缓存是长上下文推理中的主要存储瓶颈，现有量化方法仅聚焦缓存重建误差，忽略其对注意力机制的影响。本文在变换编码与逆注水原理基础上提出注意力感知变换编码（AATC），通过校准集分配比特以最小化注意力感知失真。在Llama-3.1-8B-Instruct和Qwen-2.5-7B-Instruct上的LongBench、RULER、GSM8K、MMLU-Pro、MATH-500基准测试中，AATC在约5.8倍压缩下实现近无损精度，而所有基线至少在某些场景中性能下降。

## 关键点

- 理论证明：在白噪声量化模型下，期望的注意力感知失真可分解为键和值的可加性贡献，跨令牌和通道因子分解。
- AATC方法：结合变换编码和逆注水原理，在离线校准集上分配比特，直接优化注意力感知失真。
- 实验结果：在Llama-3.1-8B-Instruct和Qwen-2.5-7B-Instruct上，AATC在约5.8倍压缩下保持近无损精度，优于现有基线。
- 基准覆盖：LongBench、RULER、GSM8K、MMLU-Pro、MATH-500等多样任务验证泛化性。

## 全文

The key-value (KV) cache stores information from past tokens and is a major memory bottleneck in long-context inference. Existing quantization methods address this bottleneck by representing the KV cache uniformly with lower-precision data types and designing quantization schemes to minimize reconstruction error in the cache itself, without accounting for how that error propagates through attention mechanisms. We prove that, under a white-noise quantization model, the expected attention-aware distortion decomposes into additive key and value contributions that factor across tokens and channels. Building on transform coding and reverse water-filling, which are classical tools from signal processing and rate-distortion theory, we introduce Attention-Aware Transform Coding (AATC), which allocates bits over a calibration set to minimize attention-aware distortion. On Llama-3.1-8B-Instruct and Qwen-2.5-7B-Instruct, evaluated across LongBench, RULER, GSM8K, MMLU-Pro, and MATH-500, our method achieves near-lossless accuracy at approximately $5.8\times$ compression, whereas each baseline degrades in at least some settings.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_84889d4162a0bc7a6cdd9bd6