---
title: "The Key to Going Linear: Analysis-Driven Transformer Linearization"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.07706v1"
permalink: "https://feed.astralor.com/items/itm_4be2136f844bc0cae3236101"
published: "2026-07-08T17:59:09.000Z"
score: 78
channel: "ai"
tags: ["Llama","Qwen","Transformer","Long-context Reasoning"]
---

# The Key to Going Linear: Analysis-Driven Transformer Linearization

## AI commentary

> This paper's value lies not in proposing yet another linearization trick, but in deeply analyzing softmax's role in state updates and designing targeted structural interventions accordingly. It is worth reading for engineers dealing with ultra-long documents or dialogues, as well as researchers optimizing attention mechanisms. The experiments covering LLaMA and Qwen enhance the generalizability of the findings.

## At a glance

This paper addresses the quadratic cost of causal self-attention in long-context transformer inference by proposing an analysis-driven post-training linearization pipeline. By isolating the effect of state update design, it reveals that softmax relies on key-dependent, rank-1 orthogonal projections, explaining why delta-style networks outperform purely gated accumulation. Identifies approximation error sources and introduces structural interventions—sink tokens, short convolutions, and fixed-budget cache routing—to reduce the gap. Scaled to LLaMA and Qwen models up to 32B parameters, outperforming prior post-hoc baselines on MMLU and matching complex adaptive caching frameworks in long-context retrieval.

## Key points

- Quadratic cost of causal self-attention is a bottleneck for long-context inference; this paper proposes analysis-driven post-training linearization.
- Reveals key-dependent rank-1 orthogonal projections in softmax, explaining delta-style network advantage.
- Introduces structural interventions: sink tokens, short convolutions, and fixed-budget cache routing to reduce approximation error.
- Validated on LLaMA and Qwen models up to 32B parameters; outperforms prior post-hoc baselines on MMLU and matches adaptive caching in long-context retrieval.

## Full text

The quadratic cost of causal self-attention severely bottlenecks long-context transformer inference. While numerous post hoc linearization pipelines exist, it is difficult to identify which components preserve model quality. This work isolates the effect of state update design in a strict frozen-backbone regime. We show that softmax relies on key-dependent, rank-1 orthogonal projections, elucidating why delta-style networks outperform purely gated accumulation. We identify a potential source of approximation errors and introduce structural interventions, specifically sink tokens, short convolutions, and fixed-budget cache routing, which reduces the remaining gap. We scale this linearization approach across LLaMA and Qwen models up to 32B parameters, outperforming prior post hoc baselines on MMLU and matching the long-context retrieval of complex adaptive-caching frameworks.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_4be2136f844bc0cae3236101