---
title: "LoRA需要多大秩？Transformer注意力的秩-误差界"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.26052v1"
permalink: "https://feed.astralor.com/items/itm_7619f8bec4943f223c545bdf"
published: "2026-08-26T17:25:03.000Z"
score: 74
channel: "ai"
tags: ["LoRA","Transformer","低秩近似","注意力机制"]
---

# LoRA需要多大秩？Transformer注意力的秩-误差界

## AI 点评

> 这篇论文把LoRA秩选择从玄学变成了数学。虽然公式有点烧脑，但核心洞见很实用：别只盯着logits的秩，注意力层面的饱和可能让低秩更可行。对做微调和模型压缩的工程师来说，这提供了理论支撑，值得精读。

## 速览

该论文为低秩适配（LoRA）在Transformer注意力中的近似误差提供了任务相关的理论刻画。作者固定预训练注意力头、目标注意力函数和下游任务输入分布，推导了秩为r的查询LoRA更新所能达到的最小期望KL误差的上下界。在目标注意力概率远离零时，下界与注意力分数差异的截断二次/线性函数成正比；同时给出无条件上界。进一步在可实现性、几何和矩条件下，将最优秩r误差与目标更新的下游加权尾部能量关联。研究还发现softmax饱和可使匹配注意力函数所需秩严格小于匹配有限logits所需秩，并将分析扩展到多头融合LoRA和联合查询/键更新。

## 关键点

- 建立了LoRA秩与注意力近似误差之间的理论上下界，将秩选择从经验调参转变为可分析的任务依赖问题。
- 证明了秩r误差与目标更新的下游加权尾部能量T_r相关，并给出了显式的比例常数。
- 发现softmax饱和情况下，匹配注意力函数所需秩可能小于匹配logits所需秩。
- 扩展了理论到多头融合LoRA和联合查询/键更新，揭示了秩共享与因子分解约束的影响。

## 全文

Choosing the rank of a low-rank adaptation (LoRA) update is usually an empirical task. In this paper, we provide a task-dependent theory of the approximation error achievable at each LoRA rank for Transformer attention. We fix a pretrained attention head, a target attention function, and a distribution over inputs from the downstream task, and bound the smallest expected Kullback--Leibler (KL) error achievable by a rank-$r$ query LoRA update. When target attention probabilities are bounded away from zero, we prove a lower bound of the error proportional to $ψ(\|d\|_2)$, where $d$ is the difference between candidate and target attention scores and $ψ(t)=\min\{t^2,t\}$. We also prove an unconditional upper bound $\min\{\|d\|_2^2/4,\sqrt2\|d\|_2\}$. Under explicit realizability, geometry, and moment conditions, we then bound the best rank-$r$ error between an explicit multiple of $ψ(\sqrt{T_r})$ and $\min\{T_r/4,\sqrt{2T_r}\}$, where $T_r$ is the downstream-weighted tail energy of the target update. We also provide target-Fisher bounds when candidate scores remain within a fixed range of the target scores, and an unrestricted lower bound when a subset of tokens carries most of the probability mass. These spectral bounds describe finite-score approximation. We then construct explicit families in which softmax saturation makes the rank required to match the attention function strictly smaller than the rank required to match the finite logits. Finally, we extend the analysis to fused multi-head LoRA and joint query/key updates, exposing the effects of rank sharing and query/key factorization constraints.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_7619f8bec4943f223c545bdf