---
title: "DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compression"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.06523v1"
permalink: "https://feed.astralor.com/items/itm_8b0702e3e27b10790db8bc17"
published: "2026-07-07T17:29:01.000Z"
score: 78
channel: "ai"
tags: ["DepthWeave-KV","KV-cache compression","Token-Adaptive","Long-context Reasoning"]
---

# DepthWeave-KV: Token-Adaptive Cross-Layer Residual Factorization for Long-Context KV Cache Compression

## AI commentary

> This paper tackles the most painful KV cache memory issue in long-context LLM inference. Its highlight is moving away from one-size-fits-all compression and letting the model decide which tokens need more detail—a smart and practical token-adaptive approach. For engineers working on model deployment and inference optimization, the 8.3x memory reduction and calibration-free feature mean it can be directly integrated into production; the open-source implementation is worth watching.

## At a glance

Long-context language model inference is increasingly limited by the memory bandwidth and capacity required to store key-value caches, yet existing compression methods often apply uniform budgets across layers or tokens and degrade retrieval when lexical cues and semantic states require different preservation. We introduce DepthWeave-KV, a token-adaptive cache compression method that factorizes key and value states across neighboring transformer layers using shared low-rank channel bases while retaining lightweight token-specific residuals where attention behavior is sensitive. DepthWeave-KV combines cross-depth residual factorization with a token-conditional depth router that allocates higher reconstruction rank to instruction-bearing and retrieval-critical tokens, and uses calibration-free online error tracking from attention-output probes to adapt compression during generation without retraining the base model. A fused CUDA implementation jointly performs basis lookup, residual dequantization, and attention projection to reduce decode-time memory traffic. Across LongBench, Needle-in-a-Haystack, L-Eval, and long-form QA and summarization benchmarks, DepthWeave-KV achieves near-full-cache task quality with substantially lower memory use, improving average score and retrieval accuracy over prior compressed caches while reaching 8.3x KV memory reduction and 72.8 tokens per second at 64K context.

## Key points

- Proposes DepthWeave-KV, which adaptively compresses KV cache via cross-layer shared low-rank basis decomposition and token-specific residuals, avoiding retrieval degradation from uniform budgets.
- Introduces a token-conditional depth router and online error tracking via attention-output probes for calibration-free dynamic compression without retraining.
- A fused CUDA implementation jointly performs basis lookup, residual dequantization, and attention projection to significantly reduce decode-time memory traffic.
- Achieves near-full-cache task quality on long-context benchmarks, with 8.3x KV memory reduction and 72.8 tokens/s throughput at 64K context.

## Full text

Long-context language model inference is increasingly limited by the memory bandwidth and capacity required to store key-value caches, yet existing compression methods often apply uniform budgets across layers or tokens and degrade retrieval when lexical cues and semantic states require different preservation. We introduce DepthWeave-KV, a token-adaptive cache compression method that factorizes key and value states across neighboring transformer layers using shared low-rank channel bases while retaining lightweight token-specific residuals where attention behavior is sensitive. DepthWeave-KV combines cross-depth residual factorization with a token-conditional depth router that allocates higher reconstruction rank to instruction-bearing and retrieval-critical tokens, and uses calibration-free online error tracking from attention-output probes to adapt compression during generation without retraining the base model. A fused CUDA implementation jointly performs basis lookup, residual dequantization, and attention projection to reduce decode-time memory traffic. Across LongBench, Needle-in-a-Haystack, L-Eval, and long-form QA and summarization benchmarks, DepthWeave-KV achieves near-full-cache task quality with substantially lower memory use, improving average score and retrieval accuracy over prior compressed caches while reaching 8.3x KV memory reduction and 72.8 tokens per second at 64K context.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_8b0702e3e27b10790db8bc17