---
title: "InferScale: GPU-Native KV Injection for Personalized LLM Serving"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.27090v1"
permalink: "https://feed.astralor.com/items/itm_257682fd1ea757f12ea609a3"
published: "2026-07-29T16:18:22.000Z"
score: 81
channel: "ai"
tags: ["Chunked RoPE","Context-Window Encoding","InferScale","vLLM"]
---

# InferScale: GPU-Native KV Injection for Personalized LLM Serving

## AI commentary

> This paper stands out by moving KV cache reuse from system scheduling to GPU memory level, directly addressing the pain point where longer memories lead to slower first tokens in personalized services. Engineering teams building chatbots, virtual assistants, or knowledge-based Q&A systems should try this immediately—especially vLLM users can integrate with near-zero cost. The slight accuracy drop is well worth the 3-4x throughput gain.

## At a glance

InferScale introduces a GPU-native KV injection system that eliminates repeated prompt prefilling in personalized LLM serving. It precomputes KV representations of memory facts, stores them with semantic embeddings on GPU, and injects them directly into vLLM’s paged cache at serving time. With Chunked RoPE and Context-Window Encoding to handle position encoding and cross-fact context loss, InferScale keeps TTFT nearly constant as retrieval budget grows, reducing TTFT by 72-79% at k=50, boosting throughput 3.7-4.5x, and achieving accuracy close to Mem0 baseline (60.3% vs 63.3%) without serving-time recomputation.

## Key points

- InferScale reduces first-token latency in personalized LLM serving by replacing repeated prefilling with precomputed KV caches.
- Introduces Chunked RoPE and Context-Window Encoding to handle position encoding and missing cross-fact context during dynamic memory assembly.
- Implemented via vLLM's KV-connector interface without engine modifications or model fine-tuning.
- On LoCoMo benchmark, at k=50, TTFT is reduced by 72-79%, concurrent throughput improved by 3.7-4.5x, with only a 3 percentage point accuracy drop vs Mem0.

## Full text

Large language models are increasingly deployed with persistent personalized context, such as accumulated memory profiles or long conversation histories, that is shared across a user's many requests. Production memory systems (e.g., Mem0, MemGPT, and Zep) retrieve a relevant subset of this memory and inject it into the prompt, forcing the serving engine to repeatedly prefill the same content. As the retrieval budget grows, time-to-first-token (TTFT) increases even though the underlying memory is reused across requests. We present InferScale, a GPU-native LLM memory system that replaces repeated prompt prefilling with reusable KV state. InferScale precomputes each memory fact's KV representation, stores it alongside a semantic embedding on the GPU, retrieves relevant facts at serving time, and injects their KV directly into vLLM's paged cache. To support dynamically assembled memories under rotary position embeddings, we introduce Chunked RoPE, which stores keys before rotation and applies their serving-time positions during injection. However, encoding memory facts independently omits the cross-fact context available during joint prefilling. We mitigate this with Context-Window Encoding, which encodes each memory fact together with a small window of preceding conversation context while caching only the target fact's KV. InferScale is implemented through vLLM's KV-connector interface, requiring neither engine modifications nor model fine-tuning. Across three open-weight models on LoCoMo, InferScale keeps TTFT nearly constant as the retrieval budget increases: at k=50 it reduces TTFT by 72-79% (3.6-4.8x), achieves 60.3% accuracy versus 63.3% for Mem0 without serving-time recomputation, and delivers 3.7-4.5x the throughput under concurrent load. Reusable KV state thus decouples memory-conditioned serving latency from retrieved-context size while preserving application quality.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_257682fd1ea757f12ea609a3