---
title: "大语言模型中的超级权重与选择性训练的失败"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.08733v1"
permalink: "https://feed.astralor.com/items/itm_64541ef176349ac37886909f"
published: "2026-07-09T17:35:00.000Z"
score: 77
channel: "ai"
tags: ["Ai2","LoRA","OLMo","参数重要性","微调","超级权重"]
---

# 大语言模型中的超级权重与选择性训练的失败

## AI 点评

> 这篇论文给了所有沉迷于参数重要性剪枝和选择性训练的从业者一剂清醒针：重要的不一定是可训练的，微调成功的关键在于结构化分解而非定位个别非凡参数。对于正在设计低成本微调方案的研究者来说，本文的对比实验（随机参数 vs 超级权重 vs LoRA）极具参考价值，也再次验证了LoRA在数据效率上的稳健性。

## 速览

近期研究发现大语言模型中存在所谓“超级权重”——个别参数删除后模型性能呈数量级下降。本文证明这种下降并非普遍适用，且尝试单独微调这些超级权重（100至8192个参数）会导致模型精度降至随机猜测水平，即使扩展到邻域参数也无改善。而随机选择同等数量参数进行微调则能提升性能。标准LoRA通过低秩结构更新所有注意力权重矩阵位置仅用0.16%参数即可成功。10次种子消融实验表明，将LoRA更新限制在超级权重坐标上不影响结果。结论：参数重要性并不等同于参数可训练性，有效微调依赖于对整层进行结构化分解而非定位个别重要权重。

## 关键点

- 超级权重删除导致模型性能骤降的现象并非适用于所有大模型。
- 单独微调超级权重（100至8192个参数）会使OLMo-1B和OLMo-7B精度降至随机水平。
- 标准LoRA以0.16%的参数更新所有注意力矩阵位置即可成功微调。
- 参数重要性不等于参数可训练性，有效微调需对整层进行结构化分解。

## 全文

Recent work identified Super Weights, individual parameters whose removal degrades model performance by orders of magnitude. We show that this degradation due to pruning Super Weights does not universally apply to all LLMs. Furthermore, if these parameters are so important, Super Weight-aware training should be effective. We show the opposite. Training Super Weights in isolation (100 to 8,192 parameters) drops accuracy to random-guessing levels on both OLMo-1B and OLMo-7B, and expanding to local neighborhoods of up to 36K parameters provides no improvement. The failure is specific to Super Weight coordinates: training an equal number of randomly chosen positions in the same down_proj layers instead improves over the baseline, so the collapse comes from targeting Super Weights, not from sparsity itself. Vanilla LoRA, updating every position in attention weight matrices through low-rank structure, succeeds with only 0.16% of parameters, and applying the same low-rank update to down_proj succeeds as well. A 10-seed ablation confirms that constraining LoRA updates at positions corresponding to Super Weight coordinates yields statistically indistinguishable results. These findings establish that parameter importance does not imply parameter trainability in isolation, and that effective fine-tuning relies on structured decompositions over entire layers rather than targeting individually important weights.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_64541ef176349ac37886909f