---
title: "Exact Fusion of Softmax Attention Heads Can Require Exponentially Many Atoms"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2611.02943v1"
permalink: "https://feed.astralor.com/items/itm_623a46116ca4de9b7243c767"
published: "2026-11-02T11:10:08.000Z"
score: 56
channel: "ai"
tags: ["Exact Fusion","Softmax Attention Heads","Can Require Exponentially","Many Atoms Fusing","Fusing"]
---

# Exact Fusion of Softmax Attention Heads Can Require Exponentially Many Atoms

## At a glance

Fusing independently normalized attention heads can increase the number of explicit key-value entries, or atoms, needed to preserve their output. We prove a sharp separation for $W_d(q)=\sum_{j=1}^dσ(q_j)$: its $d$ two-…

## Key points

- Fusing independently normalized attention heads can increase the number of explicit key-value entries, or atoms, needed to preserve their output.
- We prove a sharp separation for $W_d(q)=\sum_{j=1}^dσ(q_j)$: its $d$ two-atom heads use $2d$ head-atom terms, whereas its minimum single-head representation has exactly $2^d$ atoms, even with arbitrary fixed replacement…
- The same lower bound holds for every fixed uniform error below $1/2$ on $\mathbb R^d$, and the threshold is sharp.

## Full text

Fusing independently normalized attention heads can increase the number of explicit key-value entries, or atoms, needed to preserve their output. We prove a sharp separation for $W_d(q)=\sum_{j=1}^dσ(q_j)$: its $d$ two-atom heads use $2d$ head-atom terms, whereas its minimum single-head representation has exactly $2^d$ atoms, even with arbitrary fixed replacement keys, scalar values, and positive masses. The same lower bound holds for every fixed uniform error below $1/2$ on $\mathbb R^d$, and the threshold is sharp. The proof uses exposed-key normal cones; exact equality on any nonempty open query domain inherits the bound. The resulting atom bound for a fixed head count is tight when that count divides $d$. A normalized family retains its exponential exact minimum while approaching a constant on fixed-radius query balls. For pretrained-model caches, a classical affine-rank bound and a certified instance exclude exact vector preservation at a specified atom count. Learned-cache comparisons on two disjoint banks of 64 contexts show budget-dependent reversals and calibration effects.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_623a46116ca4de9b7243c767