---
title: "InferScale：面向个性化LLM服务的GPU原生KV注入系统"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.27090v1"
permalink: "https://feed.astralor.com/items/itm_257682fd1ea757f12ea609a3"
published: "2026-07-29T16:18:22.000Z"
score: 81
channel: "ai"
tags: ["Chunked RoPE","Context-Window Encoding","InferScale","vLLM"]
---

# InferScale：面向个性化LLM服务的GPU原生KV注入系统

## AI 点评

> 这篇论文让我眼前一亮——它将 KV 缓存复用的思路从系统调度层推到了 GPU 内存层，直接解决了个性化服务中“记忆越长，首 Token 越慢”的痛点。对于正在搭建聊天机器人、虚拟助手或知识库问答系统的工程团队来说，这是个值得立即试用的方案，尤其是 vLLM 用户几乎零成本接入。虽然精度略有损失，但 3-4 倍的吞吐提升完全值得。

## 速览

InferScale 提出了 GPU 原生的 KV 缓存注入方案，用于解决个性化 LLM 服务中重复预填充用户记忆导致的 TTFT 延迟问题。该方案将记忆事实的 KV 表示预计算并存储在 GPU 上，通过 Chunked RoPE 和 Context-Window Encoding 保持位置编码和上下文质量，在 vLLM 框架中无需修改引擎或微调即可实现检索预算增长时 TTFT 几乎不变，k=50 时降低 TTFT 72-79%，吞吐量提升 3.7-4.5 倍，精度接近 Mem0 基线。

## 关键点

- InferScale 通过预计算 KV 缓存替代重复预填充，显著降低个性化 LLM 服务的首 Token 延迟。
- 提出 Chunked RoPE 和 Context-Window Encoding 两项创新，解决动态记忆组装下的位置编码和上下文缺失问题。
- 基于 vLLM 的 KV-connector 接口实现，无需修改推理引擎或微调模型。
- 在 LoCoMo 基准上，k=50 时 TTFT 降低 72-79%，并发吞吐提升 3.7-4.5 倍，精度仅下降 3 个百分点。

## 全文

Large language models are increasingly deployed with persistent personalized context, such as accumulated memory profiles or long conversation histories, that is shared across a user's many requests. Production memory systems (e.g., Mem0, MemGPT, and Zep) retrieve a relevant subset of this memory and inject it into the prompt, forcing the serving engine to repeatedly prefill the same content. As the retrieval budget grows, time-to-first-token (TTFT) increases even though the underlying memory is reused across requests. We present InferScale, a GPU-native LLM memory system that replaces repeated prompt prefilling with reusable KV state. InferScale precomputes each memory fact's KV representation, stores it alongside a semantic embedding on the GPU, retrieves relevant facts at serving time, and injects their KV directly into vLLM's paged cache. To support dynamically assembled memories under rotary position embeddings, we introduce Chunked RoPE, which stores keys before rotation and applies their serving-time positions during injection. However, encoding memory facts independently omits the cross-fact context available during joint prefilling. We mitigate this with Context-Window Encoding, which encodes each memory fact together with a small window of preceding conversation context while caching only the target fact's KV. InferScale is implemented through vLLM's KV-connector interface, requiring neither engine modifications nor model fine-tuning. Across three open-weight models on LoCoMo, InferScale keeps TTFT nearly constant as the retrieval budget increases: at k=50 it reduces TTFT by 72-79% (3.6-4.8x), achieves 60.3% accuracy versus 63.3% for Mem0 without serving-time recomputation, and delivers 3.7-4.5x the throughput under concurrent load. Reusable KV state thus decouples memory-conditioned serving latency from retrieved-context size while preserving application quality.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_257682fd1ea757f12ea609a3