---
title: "Split-KV: Efficient Context Parallelism for Training Diffusion Drafters"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2611.05926v1"
permalink: "https://feed.astralor.com/items/itm_4a36aaf2f494213c91d9f42b"
published: "2026-11-04T04:16:46.000Z"
score: 58
channel: "tech"
tags: ["Split-KV","Efficient Context Parallelism","Training Diffusion Drafters","Diffusion","Their"]
---

# Split-KV: Efficient Context Parallelism for Training Diffusion Drafters

## At a glance

Diffusion drafters accelerate speculative decoding by denoising multiple draft tokens in parallel. Their training uses anchor sampling to construct compact draft queries and performs cross-attention with a block-sparse …

## Key points

- Diffusion drafters accelerate speculative decoding by denoising multiple draft tokens in parallel.
- Their training uses anchor sampling to construct compact draft queries and performs cross-attention with a block-sparse mask over a much longer context.
- We empirically show that acceptance length remains stable as context scales with a fixed query budget, while increasing the query budget yields diminishing quality gains, jointly establishing a practical regime with que…

## Full text

Diffusion drafters accelerate speculative decoding by denoising multiple draft tokens in parallel. Their training uses anchor sampling to construct compact draft queries and performs cross-attention with a block-sparse mask over a much longer context. We empirically show that acceptance length remains stable as context scales with a fixed query budget, while increasing the query budget yields diminishing quality gains, jointly establishing a practical regime with queries far shorter than the context. No existing context parallelism (CP) method jointly exploits this query--context asymmetry and meets the nonlinear, sample-dependent load-balancing requirement created by anchor sampling. We present Split-KV, an exact CP method that keeps key/value shards local and limits per-layer attention communication to query-sized tensors, making this communication independent of context length. Its global query view further enables a cost-aware longest-processing-time-first (LPT) strategy that balances context key/value chunks by their realized attention cost. At a CP size of 64, Split-KV retains 85--91\% weak-scaling efficiency and achieves up to $3.67\times$ speedup over the fastest available CP baseline. LPT achieves $<$1\% imbalance and plans three orders of magnitude faster than the best load-balancing baseline. Code is available at https://github.com/zxgx/verl.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_4a36aaf2f494213c91d9f42b