---
title: "Windowed-MTP：去除百万token上下文中的草稿KV读取瓶颈"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.21535v1"
permalink: "https://feed.astralor.com/items/itm_62233e453547de43caee2b4c"
published: "2026-07-23T17:21:44.000Z"
score: 77
channel: "ai"
tags: ["Mamba2","Qwen","StreamingLLM","投机解码","长上下文"]
---

# Windowed-MTP：去除百万token上下文中的草稿KV读取瓶颈

## AI 点评

> 这篇论文直接切中了长上下文投机解码的痛点，对于正在部署百万token级输入的应用团队极具参考价值。无需训练即可获得显著加速，且保持输出分布不变，值得重点跟进。

## 速览

本文针对百万token长上下文场景中投机解码的草稿模型成本问题，提出Windowed-MTP。其对草稿模型的注意力层应用StreamingLLM式的滑动窗口+注意力汇，将KV工作集限制为常数，去除了全上下文草稿KV读取瓶颈，保持目标模型全注意力验证不变。实验表明，在Qwen GDN-MoE 35B/122B及Mamba2-hybrid 120B上，单GPU SGLang中每步解码成本降低28%-44%，且无损接受分布。同时可回收未读取的草稿KV缓存约7.7%-11%。

## 关键点

- 长上下文下MTP草稿模型的KV读取成本随上下文线性增长，抵消投机解码收益。
- Windowed-MTP对草稿注意力采用滑动窗口+注意力汇，将KV工作集限制为常数，无需训练且输出无损。
- 在1M上下文时减少约99%的KV条目，单GPU解码成本降低28%-44%。
- 可通过环形缓冲区回收未读取的草稿KV，额外节省7.7%-11%总KV。

## 全文

Speculative decoding accelerates autoregressive generation by having a cheap draft propose tokens that a target verifies in parallel. Frontier models increasingly ship a built-in Multi-Token-Prediction (MTP/NEXTN) draft head under the assumption that the draft is negligibly cheap. At million-token context this breaks: an MTP draft head typically runs full attention over the entire KV cache at every draft step, so its read grows linearly with context and comes to dominate the draft cost -- precisely where speculation is most valuable. The effect compounds with draft length (a deep native draft can turn net-negative, slower than no speculation) and sharpens under hybrid/linear-attention targets, where cheaper verification leaves the draft's full-attention read exposed. We apply a StreamingLLM-style sliding window plus attention sink to the draft's attention only (Windowed-MTP), leaving full-attention verification intact. It is training-free, drop-in, and lossless by construction: the full-attention target still decides every accepted token, so windowing changes only which tokens are proposed, never which are accepted. It bounds the draft's KV working set to a constant, dropping ~99% of KV entries at 1M. Across three architecture families (Qwen GDN-MoE 35B/122B and a Mamba2-hybrid NoPE 120B) at 1M context on a single GPU in SGLang, windowing cuts the per-decode-step cost over the shipping native MTP draft by +28% to +44%, an input-invariant margin that widens with context. Since per-token latency is this cost divided by acceptance length, at matched acceptance end-to-end decode latency improves by the same amount, and more where windowing also lifts acceptance, while preserving the target's verified output distribution. Finally, the unread draft KV -- 7.7-11% of total KV at 1M -- is reclaimed via a compact ring buffer at no acceptance or quality cost.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_62233e453547de43caee2b4c