---
title: "JoLT: Near-Lossless KV Cache Compression via Joint Tucker and JL-Residual Allocation"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.12550v1"
permalink: "https://feed.astralor.com/items/itm_870cc2b28619f2746b0241eb"
published: "2026-07-14T09:23:20.000Z"
score: 79
channel: "tech"
tags: ["JoLT","KV Cache Compression","Meta","Mistral","LLM Inference Optimization","Tensor Factorization"]
---

# JoLT: Near-Lossless KV Cache Compression via Joint Tucker and JL-Residual Allocation

## AI commentary

> This paper is a must-read for anyone focused on LLM inference efficiency. It applies classical tensor decomposition to the pressing KV cache explosion problem with surprisingly good results. JoLT's core insight—that the cache is naturally a third-order tensor, yet previous work only compressed 2D slices—is highly inspiring. For teams deploying long-context models (e.g., 128K, 1M tokens), this 2-3x near-lossless compression plus FlashJoLT speedup means directly saving over half the memory or extending context further. I particularly appreciate the unified Lagrangian dual for allocating ranks and bit-widths, making the compression strategy adaptive across layers and key/value differences.

## At a glance

The KV cache dominates memory in large model inference as context length grows. Existing low-rank and quantization methods fail to exploit the third-order tensor structure of the cache. This paper introduces JoLT, which applies a partial Tucker decomposition on token and feature axes and restores truncated energy via JL-rotated low-bit residuals. A Lagrangian dual jointly allocates Tucker ranks and residual bit-widths, achieving 2-3x near-lossless compression: perplexity, GSM8K accuracy, and RULER retrieval remain within statistical noise on Mistral-7B and LLaMA-2-13B. Frobenius error is an order of magnitude lower than cross-layer SVD and 4-bit quantization. The randomized SVD variant FlashJoLT provides 5-13x compression speedup.

## Key points

- First to treat KV cache as a third-order tensor (heads, tokens, features) and apply partial Tucker decomposition on token and feature axes.
- Uses JL-rotated low-bit residuals to restore truncated energy, jointly allocating Tucker ranks and residual bit-widths.
- Achieves 2-3x near-lossless compression on Mistral-7B and LLaMA-2-13B without performance degradation.
- FlashJoLT variant speeds up compression by 5-13x via randomized SVD.

## Full text

The key-value (KV) cache has become the dominant memory cost of transformer inference. It grows with batch size, context length, and depth, and at long context it, rather than the model weights, sets the ceiling on throughput. Two families of methods reduce it. Low-rank methods factor two-dimensional slices of the cache, either per-head matrices or cross-layer feature blocks, and quantization methods lower the bit-width of every entry. Neither family exploits the fact that the cache at a layer is naturally a third-order tensor whose three axes, the heads, the tokens, and the features, carry very different amounts of redundancy. We take this tensor view directly. Our method, JoLT, applies a partial Tucker decomposition that compresses only the token and feature axes while leaving the head and layer axes intact, and then restores the energy that truncation discards with a Johnson-Lindenstrauss (JL) rotated low-bit residual. A single Lagrangian dual allocates the Tucker ranks and the residual bit-widths together, per layer group and separately for keys and values, under one byte budget. The result is a near-lossless 2-3x compression: perplexity, GSM8K accuracy, and RULER needle-in-a-haystack retrieval all stay at or within statistical noise of the uncompressed baseline on both a grouped-query-attention model (Mistral-7B-v0.3) and a multi-head-attention model (LLaMA-2-13B). At 2x, JoLT reconstructs the cache to relative Frobenius error 0.009 (K) and 0.006 (V) on both architectures, roughly an order of magnitude below cross-layer SVD and 4-bit quantization. A randomized-SVD variant, FlashJoLT, delivers a 5-13x compression-time speedup at matched quality.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_870cc2b28619f2746b0241eb