---
title: "当剪枝遇上可解释性：保持大语言模型中稀疏自编码器的鲁棒性"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.25941v1"
permalink: "https://feed.astralor.com/items/itm_fde05d1fdc7ff8e28686027e"
published: "2026-08-26T15:57:42.000Z"
score: 81
channel: "ai"
tags: ["SparseGPT","Wanda","可解释性","大语言模型","模型剪枝","稀疏自编码器"]
---

# 当剪枝遇上可解释性：保持大语言模型中稀疏自编码器的鲁棒性

## AI 点评

> 可解释性社区和压缩社区都在各自解决难题，这篇论文把两者真正连了起来。对从事模型压缩或需要审计模型行为的团队尤其有价值：它给出的不是单一技巧，而是一套判断剪枝方法是否“可解释友好”的评估框架。建议在复现时重点看中间层分配策略的收益是否在你自己的模型上同样成立。

## 速览

稀疏自编码器（SAE）用于解释大模型，但它在剪枝后的可靠性未知。新论文从理论上指出，剪枝影响由扰动能量决定；幅度剪枝忽略激活几何，破坏表示空间，而 Wanda、SparseGPT 等激活感知方法更稳健。中间层最脆弱，据此提出的逐层稀疏分配策略在四类模型上降低了困惑度。

## 关键点

- 从理论上证明剪枝对固定 SAE 的影响由扰动能量（协方差加权范数）控制。
- 幅度剪枝忽略激活几何，会破坏表示空间；Wanda、SparseGPT 等激活感知方法更稳健。
- 中间层对剪枝最敏感，是各类剪枝方法共有的结构性弱点。
- 提出逐层稀疏分配策略，在相同平均剪枝稀疏度下降低困惑度，并在四类模型架构上验证。

## 全文

Sparse autoencoders (SAEs) are widely used to interpret the internal representations of large language models (LLMs), yet their reliability under post-hoc model compression remains poorly understood. We present a systematic study of how pruning affects SAE behavior and theoretically show that, for a fixed SAE, its impact is governed by perturbation energy, a covariance-weighted norm. This perspective exposes a key limitation of magnitude pruning: by ignoring activation geometry, it distorts the learned representation space and degrades SAE functionality. Activation-aware methods such as Wanda and SparseGPT, in contrast, implicitly control perturbation energy and are therefore substantially more robust at preserving SAE behavior. We further reveal a consistent structural vulnerability across all pruning methods: middle layers are significantly more sensitive to pruning than early or late layers. Guided by this insight, we propose a layer-wise sparsity allocation strategy, achieving lower perplexity under the same average pruning sparsity. Experiments across four model architectures validate our theoretical findings. Code is publicly available at https://github.com/osu-srml/sae-robustness-under-pruning/tree/main.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_fde05d1fdc7ff8e28686027e