---
title: "Prune Gradually, Adapt Sparsely: Efficient Pruning of Large Language Models at Extreme Sparsity"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2610.41968v1"
permalink: "https://feed.astralor.com/items/itm_32f0afc8719a674263e1b417"
published: "2026-10-30T16:17:43.000Z"
score: 57
channel: "tech"
tags: ["LLM","推理效率","RAG","Prune Gradually","Adapt Sparsely","Efficient Pruning","Large Language Models","Extreme Sparsity Weight"]
---

# Prune Gradually, Adapt Sparsely: Efficient Pruning of Large Language Models at Extreme Sparsity

## 速览

Weight pruning can reduce the storage and inference cost of large language models (LLMs), but preserving quality at high sparsity levels remains challenging. Existing methods face a quality-cost trade-off in this regime…

## 关键点

- Weight pruning can reduce the storage and inference cost of large language models (LLMs), but preserving quality at high sparsity levels remains challenging.
- Existing methods face a quality-cost trade-off in this regime.
- One-shot pruning is cheap but degrades sharply as sparsity increases, even when followed by post-pruning recovery.

## 全文

Weight pruning can reduce the storage and inference cost of large language models (LLMs), but preserving quality at high sparsity levels remains challenging. Existing methods face a quality-cost trade-off in this regime. One-shot pruning is cheap but degrades sharply as sparsity increases, even when followed by post-pruning recovery. A growing line of work instead optimizes the mask directly from the dense model under the end-to-end objective; among these methods, [Lee et al.,2026], the recent state of the art at extreme sparsity, reaches high quality but at the cost of model-sized optimizer state and auxiliary variables. To address this gap, we propose ETAPS, which prunes gradually and adapts sparsely: it tightens a global sparsity constraint in progressive stages while training a sparse delta with a small, fixed budget. Our key observation is that gradual weight removal keeps each pruning step small enough for sparse adaptation to compensate, allowing the adapted model to provide end-to-end gradients that guide the next pruning decision without model-sized optimizer state. Across six models from 125M to 13B parameters, ETAPS achieves lower perplexity than ELSA at every evaluated sparsity from 50% to 90%, with perplexity reductions of up to 42%. ETAPS is also far cheaper: at 90% sparsity, it prunes LLaMA2-7B on a single H200, where ELSA uses four, with $5.0\times$ smaller total persistent state and 49% fewer GPU-hours. Together, our findings point to a promising direction for achieving high quality in demanding regimes such as extreme sparsity at substantially lower cost.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_32f0afc8719a674263e1b417