---
title: "TriAttention: Unified Sparse Mixed-Precision Attention for Long-Context LLM Inference"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2611.07256v1"
permalink: "https://feed.astralor.com/items/itm_2ae3f3871d72ef7d4124ca57"
published: "2026-11-05T00:14:13.000Z"
score: 60
channel: "tech"
tags: ["LLM","推理效率","RAG","TriAttention","Unified Sparse Mixed-Precision","Attention","Long-Context LLM Inference","Sparsity"]
---

# TriAttention: Unified Sparse Mixed-Precision Attention for Long-Context LLM Inference

## 速览

Sparsity and quantization reduce the cost of attention by skipping interactions and lowering the precision of those retained. However, useful interactions differ in their precision requirements: some tolerate low-precis…

## 关键点

- Sparsity and quantization reduce the cost of attention by skipping interactions and lowering the precision of those retained.
- However, useful interactions differ in their precision requirements: some tolerate low-precision scores while remaining sensitive to removal.
- We therefore introduce TriAttention, which assigns query-key (QK) tiles to skipping, 4-bit, or 8-bit computation, using 4-bit to retain quantization-tolerant interactions at lower cost.

## 全文

Sparsity and quantization reduce the cost of attention by skipping interactions and lowering the precision of those retained. However, useful interactions differ in their precision requirements: some tolerate low-precision scores while remaining sensitive to removal. We therefore introduce TriAttention, which assigns query-key (QK) tiles to skipping, 4-bit, or 8-bit computation, using 4-bit to retain quantization-tolerant interactions at lower cost. To represent compactly, we encode each head's computation regions with two diagonal-distance boundaries, keeping policy storage independent of context length. Offline calibration then evaluates candidate boundary pairs using normalized projected-output errors and complete-tile execution costs. Using these measurements, a model-wide allocator selects each head's boundaries to minimize calibration error under a shared computation budget. During prefill, the selected boundaries generate precision-grouped traversals for a fused kernel that skips omitted tiles and combines contributions from both QK precisions into one normalized attention output. The quantized keys and values prepared during prefill are then reused in a compressed key-value (KV) cache for decoding, reducing repeated reads of prompt data. Experiments show that TriAttention achieves end-to-end generation speedups of up to 7.88x over the Torch attention and 2.65x over the BF16 baseline. Meanwhile, TriAttention maintains quality comparable to the original models on general-purpose and long-context tasks. These results show how a compact retention-and-precision policy translates local sensitivities into attention computation and repeated KV-cache access. The implementation is https://github.com/HanzhiZhang-Ulrica/TriAttention.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_2ae3f3871d72ef7d4124ca57