---
title: "Super Weights in LLMs and the Failure of Selective Training"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.08733v1"
permalink: "https://feed.astralor.com/items/itm_64541ef176349ac37886909f"
published: "2026-07-09T17:35:00.000Z"
score: 77
channel: "ai"
tags: ["Ai2","LoRA","OLMo","Parameter Importance","Fine-tuning","Super Weights"]
---

# Super Weights in LLMs and the Failure of Selective Training

## AI commentary

> This paper delivers a much-needed wake-up call for those obsessed with parameter importance pruning and selective training: what is important is not necessarily trainable, and success in fine-tuning lies in structured decompositions rather than targeting individual extraordinary parameters. For researchers designing low-cost fine-tuning methods, the controlled experiments (random vs. Super Weight vs. LoRA) are highly informative, and they reaffirm LoRA's robustness in data efficiency.

## At a glance

Recent work identified 'Super Weights' in LLMs—individual parameters whose removal degrades model performance by orders of magnitude. This paper shows that such degradation is not universal across all LLMs. Additionally, training Super Weights in isolation (100–8,192 parameters) drops accuracy to random-guessing levels on OLMo-1B and OLMo-7B, and expanding to local neighborhoods of up to 36K parameters provides no improvement. In contrast, training an equal number of randomly chosen positions in the same layers improves performance. Standard LoRA, updating all attention weight matrix positions via low-rank structure with only 0.16% of parameters, succeeds. A 10-seed ablation confirms that constraining LoRA updates at Super Weight coordinates yields statistically indistinguishable results. These findings establish that parameter importance does not imply parameter trainability in isolation, and effective fine-tuning relies on structured decompositions over entire layers rather than targeting individually important weights.

## Key points

- The performance degradation caused by pruning Super Weights does not universally apply to all LLMs.
- Fine-tuning Super Weights in isolation (100–8,192 parameters) drops accuracy to random-guessing on OLMo-1B and OLMo-7B.
- Standard LoRA succeeds with only 0.16% of parameters by updating all attention weight matrix positions.
- Parameter importance does not imply parameter trainability; effective fine-tuning relies on structured decompositions over entire layers.

## Full text

Recent work identified Super Weights, individual parameters whose removal degrades model performance by orders of magnitude. We show that this degradation due to pruning Super Weights does not universally apply to all LLMs. Furthermore, if these parameters are so important, Super Weight-aware training should be effective. We show the opposite. Training Super Weights in isolation (100 to 8,192 parameters) drops accuracy to random-guessing levels on both OLMo-1B and OLMo-7B, and expanding to local neighborhoods of up to 36K parameters provides no improvement. The failure is specific to Super Weight coordinates: training an equal number of randomly chosen positions in the same down_proj layers instead improves over the baseline, so the collapse comes from targeting Super Weights, not from sparsity itself. Vanilla LoRA, updating every position in attention weight matrices through low-rank structure, succeeds with only 0.16% of parameters, and applying the same low-rank update to down_proj succeeds as well. A 10-seed ablation confirms that constraining LoRA updates at positions corresponding to Super Weight coordinates yields statistically indistinguishable results. These findings establish that parameter importance does not imply parameter trainability in isolation, and that effective fine-tuning relies on structured decompositions over entire layers rather than targeting individually important weights.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_64541ef176349ac37886909f