---
title: "Latent Sparse Attention: Towards Subquadratic Ultra-Long Context Attention"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2610.42147v1"
permalink: "https://feed.astralor.com/items/itm_e407015fbb2c96e3b8659831"
published: "2026-10-30T17:59:34.000Z"
score: 55
channel: "tech"
tags: ["AI Agent","Latent Sparse Attention","Towards Subquadratic Ultra-Long","Context Attention Agentic","However","Agentic","Yet"]
---

# Latent Sparse Attention: Towards Subquadratic Ultra-Long Context Attention

## At a glance

Agentic workflows are increasingly shifting towards long-horizon tasks, such as open-ended scientific work, where models benefit from long context windows. However, at ultra-long context (above 1M tokens), full attentio…

## Key points

- Agentic workflows are increasingly shifting towards long-horizon tasks, such as open-ended scientific work, where models benefit from long context windows.
- However, at ultra-long context (above 1M tokens), full attention is prohibitively expensive.
- A promising approach is sparse attention, in which a dedicated indexer selects a subset of keys for the attention.

## Full text

Agentic workflows are increasingly shifting towards long-horizon tasks, such as open-ended scientific work, where models benefit from long context windows. However, at ultra-long context (above 1M tokens), full attention is prohibitively expensive. A promising approach is sparse attention, in which a dedicated indexer selects a subset of keys for the attention. Yet current methods of indexing either still have quadratic cost in context size or lose accuracy at ultra-long context scale. We address this with Latent Sparse Attention (LSA), an efficient attention method that maintains accuracy at ultra-long context. In LSA, keys are hierarchically clustered in a learned latent space, and queries score the hierarchy of clusters instead of scoring every key, making prefill, decode and training faster at long context. We propose two variants, Distilled-LSA (D-LSA), which can be applied to pre-trained models via distillation, and Native-LSA (N-LSA), which is trained natively on the next-token loss alone. On GLM-5.3-FLASH (QWEN-3.8-27B) at 10M tokens, D-LSA speeds up one attention layer by 22.7x (72.4x) in prefill and 4.6x (42.5x) in decode over its original sparse attention (full attention), while keeping similar accuracy across a variety of tasks. N-LSA matches the accuracy of full attention with a training speedup of 1.6x per layer at 128k tokens. LSA offers practical and efficient attention for models to benefit from ultra-long context.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_e407015fbb2c96e3b8659831