---
title: "When Pruning Meets Interpretability: Preserving Sparse Autoencoder Robustness in LLMs"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.25941v1"
permalink: "https://feed.astralor.com/items/itm_fde05d1fdc7ff8e28686027e"
published: "2026-08-26T15:57:42.000Z"
score: 81
channel: "ai"
tags: ["SparseGPT","Wanda","Interpretability","Large Language Models","Model Pruning","Sparse Autoencoder"]
---

# When Pruning Meets Interpretability: Preserving Sparse Autoencoder Robustness in LLMs

## AI commentary

> Interpretability and compression communities have been solving separate puzzles; this paper genuinely connects them. It is especially valuable for teams doing model compression or auditing model behavior, as it offers not just a trick but an evaluation framework for whether a pruning method is 'interpretability-friendly.' When reproducing the results, pay special attention to whether the middle-layer allocation benefit holds on your own models.

## At a glance

Sparse autoencoders (SAEs) are widely used to interpret LLM internals, but how model compression affects them is poorly understood. This paper systematically studies pruning's impact on SAE behavior. Theoretically, the effect of pruning on a fixed SAE is governed by perturbation energy, a covariance-weighted norm. Magnitude pruning ignores activation geometry and distorts the learned representation, while activation-aware methods like Wanda and SparseGPT implicitly control perturbation energy and preserve SAE behavior much better. The study also finds middle layers are consistently more sensitive than early or late layers across methods, and proposes a layer-wise sparsity allocation strategy that achieves lower perplexity under the same average pruning sparsity. Experiments on four model architectures confirm the findings. Code is available online.

## Key points

- Theoretically proves that pruning's impact on a fixed SAE is controlled by perturbation energy, a covariance-weighted norm.
- Magnitude pruning ignores activation geometry and distorts the representation space; activation-aware methods such as Wanda and SparseGPT are significantly more robust.
- Middle layers are consistently more sensitive to pruning than early or late layers across all methods.
- Proposes a layer-wise sparsity allocation strategy that lowers perplexity under the same average pruning sparsity, validated on four model architectures.

## Full text

Sparse autoencoders (SAEs) are widely used to interpret the internal representations of large language models (LLMs), yet their reliability under post-hoc model compression remains poorly understood. We present a systematic study of how pruning affects SAE behavior and theoretically show that, for a fixed SAE, its impact is governed by perturbation energy, a covariance-weighted norm. This perspective exposes a key limitation of magnitude pruning: by ignoring activation geometry, it distorts the learned representation space and degrades SAE functionality. Activation-aware methods such as Wanda and SparseGPT, in contrast, implicitly control perturbation energy and are therefore substantially more robust at preserving SAE behavior. We further reveal a consistent structural vulnerability across all pruning methods: middle layers are significantly more sensitive to pruning than early or late layers. Guided by this insight, we propose a layer-wise sparsity allocation strategy, achieving lower perplexity under the same average pruning sparsity. Experiments across four model architectures validate our theoretical findings. Code is publicly available at https://github.com/osu-srml/sae-robustness-under-pruning/tree/main.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_fde05d1fdc7ff8e28686027e