---
title: "Windowed-MTP: Removing the Full-Context Draft-KV Tax at Million-Token Context"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.21535v1"
permalink: "https://feed.astralor.com/items/itm_62233e453547de43caee2b4c"
published: "2026-07-23T17:21:44.000Z"
score: 77
channel: "ai"
tags: ["Mamba2","Qwen","StreamingLLM","Speculative Decoding","Long Context"]
---

# Windowed-MTP: Removing the Full-Context Draft-KV Tax at Million-Token Context

## AI commentary

> This paper directly hits the pain point of speculative decoding with long contexts, and is highly valuable for teams deploying million-token-level inputs. The training-free acceleration with preserved output distribution makes it a must-watch.

## At a glance

This paper addresses the draft model cost issue in speculative decoding under million-token contexts by proposing Windowed-MTP. It applies a StreamingLLM-style sliding window plus attention sink only to the draft's attention, bounding the KV working set to a constant and eliminating the full-context draft-KV read tax while preserving full-attention verification in the target model. Experiments on Qwen GDN-MoE 35B/122B and Mamba2-hybrid 120B in single-GPU SGLang show a 28%-44% reduction in per-decode-step cost with no loss in output distribution. Additionally, unread draft KV cache (~7.7%-11%) is reclaimed via a compact ring buffer.

## Key points

- In long contexts, the KV read cost of MTP draft models grows linearly with context, negating the benefits of speculative decoding.
- Windowed-MTP applies a sliding window plus attention sink to the draft's attention, bounding the KV working set to a constant, training-free and lossless.
- At 1M context, it drops ~99% of KV entries and reduces per-decode-step cost by 28%-44% on a single GPU.
- Unread draft KV is reclaimed via a ring buffer, saving an additional 7.7%-11% of total KV.

## Full text

Speculative decoding accelerates autoregressive generation by having a cheap draft propose tokens that a target verifies in parallel. Frontier models increasingly ship a built-in Multi-Token-Prediction (MTP/NEXTN) draft head under the assumption that the draft is negligibly cheap. At million-token context this breaks: an MTP draft head typically runs full attention over the entire KV cache at every draft step, so its read grows linearly with context and comes to dominate the draft cost -- precisely where speculation is most valuable. The effect compounds with draft length (a deep native draft can turn net-negative, slower than no speculation) and sharpens under hybrid/linear-attention targets, where cheaper verification leaves the draft's full-attention read exposed. We apply a StreamingLLM-style sliding window plus attention sink to the draft's attention only (Windowed-MTP), leaving full-attention verification intact. It is training-free, drop-in, and lossless by construction: the full-attention target still decides every accepted token, so windowing changes only which tokens are proposed, never which are accepted. It bounds the draft's KV working set to a constant, dropping ~99% of KV entries at 1M. Across three architecture families (Qwen GDN-MoE 35B/122B and a Mamba2-hybrid NoPE 120B) at 1M context on a single GPU in SGLang, windowing cuts the per-decode-step cost over the shipping native MTP draft by +28% to +44%, an input-invariant margin that widens with context. Since per-token latency is this cost divided by acceptance length, at matched acceptance end-to-end decode latency improves by the same amount, and more where windowing also lifts acceptance, while preserving the target's verified output distribution. Finally, the unread draft KV -- 7.7-11% of total KV at 1M -- is reclaimed via a compact ring buffer at no acceptance or quality cost.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_62233e453547de43caee2b4c