---
title: "κ-LoRA: Condition Numbers Reveal Which LoRA Matrices Worth Updating"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.22489v1"
permalink: "https://feed.astralor.com/items/itm_c9d5e239a9472d47bd78e11b"
published: "2026-07-24T17:00:40.000Z"
score: 79
channel: "tech"
tags: ["LoRA","κ-LoRA","Low-Rank Adaptation","Spectral Rebalancing","Parameter-Efficient Fine-Tuning","Condition Number"]
---

# κ-LoRA: Condition Numbers Reveal Which LoRA Matrices Worth Updating

## AI commentary

> This paper stands out because it doesn't just tweak LoRA engineering details; it uncovers a previously overlooked principle—the condition number distribution of LoRA matrices dictates update necessity. For engineers and researchers working on large model fine-tuning or edge deployment, κ-LoRA offers a simple and effective cost-reduction solution. The observation that condition numbers decrease during training hints at adaptive or dynamic strategies, potentially inspiring more efficient fine-tuning methods.

## At a glance

κ-LoRA analyzes the condition number (ratio of largest to smallest singular value) of LoRA matrices and finds that only matrices with large condition numbers contain meaningful update directions, while those with small condition numbers contribute little. By updating only the top 50% of matrices ranked by condition number, κ-LoRA halves trainable parameters, reduces fine-tuning time by 16.2% and memory cost by 4.5% on average, while matching standard LoRA accuracy. Experiments further show that condition numbers of selected matrices consistently decrease during training, indicating that the method works via targeted spectral rebalancing rather than mere parameter selection.

## Key points

- First to reveal that LoRA matrix condition numbers correlate positively with update importance: matrices with larger condition numbers span richer subspaces and drive performance gains.
- Proposes κ-LoRA, which updates only the top 50% of matrices by condition number, significantly reducing compute and memory costs.
- Achieves 16.2% average reduction in fine-tuning time and 4.5% memory reduction across multiple benchmarks, matching standard LoRA accuracy.
- Experimental finding that condition numbers of selected matrices decrease over training, suggesting κ-LoRA achieves efficient adaptation via spectral rebalancing.

## Full text

Low-Rank Adaptation (LoRA) has become a widely adopted technique for efficient neural network fine-tuning, decomposing model updates into low-rank matrices. However, LoRA remains computationally costly because it updates all matrices uniformly, regardless of their actual contribution to adaptation. This cost is especially prohibitive for large-scale models with billions of parameters and for resource-constrained settings such as edge deployment and on-device fine-tuning. We show for the first time that not all LoRA matrices are equally worth tuning: matrices with smaller condition numbers (the ratio of largest to smallest singular value) are already well-balanced across directions and contribute only marginally to adaptation, whereas matrices with larger condition numbers contain underdeveloped directions that span richer subspaces and drive most of the performance gains. This observation itself is a key contribution of our work, and it motivates a more selective approach to fine-tuning. Building on this insight, we propose \k{appa}-LoRA, a method that optimizes LoRA by focusing updates on the matrices with the largest condition numbers, which capture the most informative directions of change. By restricting LoRA updates to the top 50% of weight matrices ranked by condition number, \k{appa}-LoRA halves the trainable parameter count and correspondingly reduces compute and memory cost. Extensive experiments across multiple benchmarks show that this design cuts fine-tuning time by 16.2% on average while matching the accuracy of standard LoRA and reducing memory cost by 4.5%. Further analysis reveals that the condition numbers of the selected matrices consistently decrease over training, suggesting that \k{appa}-LoRA's effectiveness stems from targeted spectral rebalancing rather than parameter selection alone.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_c9d5e239a9472d47bd78e11b