---
title: "Trainable Vector Quantization for Large Language Models"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2610.41822v1"
permalink: "https://feed.astralor.com/items/itm_0a4f3c62750b8f865b4313fc"
published: "2026-10-30T14:42:40.000Z"
score: 55
channel: "tech"
tags: ["LLM","推理效率","RAG","Trainable Vector Quantization","Large Language Models","Vector Quantization","Large Language Model","However"]
---

# Trainable Vector Quantization for Large Language Models

## 速览

Vector Quantization (VQ) is a rapidly evolving technique for compressing Large Language Model (LLM) weights, reducing memory usage, and speeding up inference. However, extreme compression deteriorates model performance,…

## 关键点

- Vector Quantization (VQ) is a rapidly evolving technique for compressing Large Language Model (LLM) weights, reducing memory usage, and speeding up inference.
- However, extreme compression deteriorates model performance, which is challenging to recover due to the discrete nature of VQ parameters.
- Existing methods employ training-free approaches that are limited to layer-wise reconstruction loss minimization.

## 全文

Vector Quantization (VQ) is a rapidly evolving technique for compressing Large Language Model (LLM) weights, reducing memory usage, and speeding up inference. However, extreme compression deteriorates model performance, which is challenging to recover due to the discrete nature of VQ parameters. Existing methods employ training-free approaches that are limited to layer-wise reconstruction loss minimization. To address this limitation, we propose TVQ, a novel method that leverages straight-through estimation to enable gradient descent-based optimization of vector-quantized LLMs. By reformulating VQ LLM optimization with trainable latent weights, TVQ enables arbitrary objective minimization. We constitute our approach by applying TVQ in a block-wise knowledge distillation paradigm, which balances quality and training efficiency, and propose several practical solutions to accelerate training. In experiments on modern LLMs, we demonstrate that TVQ is compatible with modern VQ schemes. As a result, TVQ achieves state-of-the-art quality in low-bit post-training quantization settings with a modest training budget and straightforward PyTorch implementation. For instance, TVQ achieves near-lossless 2-bit quantization for LLaMA2-70B with only 1% accuracy drop on popular Question-Answering benchmarks.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_0a4f3c62750b8f865b4313fc