---
title: "vToken: Token-Level Virtualization for Reclaimable KV Caches"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.13263v1"
permalink: "https://feed.astralor.com/items/itm_97958df04623aa4beb7dc596"
published: "2026-08-13T14:01:21.000Z"
score: 77
channel: "tech"
tags: ["KV cache","LLM Inference Optimization","PagedAttention","vLLM","vToken"]
---

# vToken: Token-Level Virtualization for Reclaimable KV Caches

## AI commentary

> What stands out is using a virtualization approach to resolve the granularity mismatch, rather than stacking yet another cache algorithm. For engineers working on inference system internals, it offers a low-intrusion, reusable abstraction that could influence future KV manager designs.

## At a glance

Large language model serving faces a critical memory bottleneck: the KV cache grows with sequence length and batch size. PagedAttention uses fixed-size memory blocks to reduce allocator-level fragmentation, but recent KV eviction algorithms operate at a token granularity finer than block-level management. This mismatch causes intra-block fragmentation, leaving a large fraction of allocated KV memory unreclaimable. vToken is a lightweight token-level virtualization layer that decouples logical token liveness from physical block placement. It maintains a stable logical token view through token-table indirection and realizes physical reclamation by repacking live tokens asynchronously. The design preserves PagedAttention kernels and CUDA Graph compatibility. Implemented in vLLM and evaluated with H2O, Random, and Scissorhands, vToken reduces retained KV blocks per request by 27.2%--72.3% and improves SLA-constrained throughput by up to 1.37x. Under a constrained active-KV budget, it extends maximum feasible concurrency by up to 2x, while reducing per-policy integration footprint from 500+ lines to under 50.

## Key points

- vToken decouples logical token liveness from physical block placement via token-table indirection, solving unreclaimable memory caused by intra-block fragmentation.
- Asynchronous repacking of live tokens enables physical reclamation while preserving PagedAttention kernels and CUDA Graph compatibility.
- Integrated with H2O, Random, and Scissorhands in vLLM, vToken reduces retained KV blocks per request by 27.2%--72.3% and improves SLA-constrained throughput by up to 1.37x.
- Per-policy integration footprint drops from 500+ lines to under 50, and constrained active-KV budgets extend maximum feasible concurrency by up to 2x.

## Full text

Large language model serving faces a critical memory bottleneck: the KV cache grows with sequence length and batch size. PagedAttention uses fixed-size memory blocks to reduce allocator-level fragmentation, but recent KV eviction algorithms operate at a token granularity finer than block-level management. This mismatch causes intra-block fragmentation, leaving a large fraction of allocated KV memory unreclaimable. We present vToken, a lightweight token-level virtualization layer that decouples logical token liveness from physical block placement. vToken maintains a stable logical token view through token-table indirection and realizes physical reclamation by repacking live tokens asynchronously. The design preserves PagedAttention kernels and CUDA Graph compatibility. We implement vToken in vLLM and evaluate it with H2O, Random, and Scissorhands across models. Compared with a paired Naive-Evict baseline, vToken reduces retained KV blocks per request by 27.2\%--72.3\% and improves SLA-constrained throughput by up to 1.37$\times$. Under a constrained active-KV budget, it extends the maximum feasible concurrency by up to 2$\times$, while reducing the per-policy integration footprint from 500+ lines to under 50.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_97958df04623aa4beb7dc596