---
title: "SLORR：简单高效的训练内低秩正则化框架"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.08754v1"
permalink: "https://feed.astralor.com/items/itm_4d760f0976b465d3343ffb1d"
published: "2026-07-09T17:51:50.000Z"
score: 78
channel: "ai"
tags: ["Hoyer稀疏度","ResNet-50","SLORR","低秩正则化","核范数"]
---

# SLORR：简单高效的训练内低秩正则化框架

## AI 点评

> 这篇论文的亮点在于其简洁性和高效性，无需复杂改动即可融入现有训练流程。对于从事模型压缩和边缘部署的研究者来说，SLORR提供了一种低成本的提升可压缩性的手段。特别值得注意的是其在LLM预训练上的表现——仅增加不到1%的开销，这在实际大规模训练中极具吸引力。

## 速览

SLORR是一种无状态、保架构的低秩正则化方法，基于Hoyer稀疏度和核范数两种变体，通过GPU友好的近似直接在原始权重矩阵上施加正则化，提升模型可压缩性。在ImageNet-1K上对ResNet-50、ViT等模型的短时持续训练和预训练中，引入低于8%的训练开销；在135M和560M参数规模的LLM预训练中，开销低于1%且压缩后模型性能显著优于无正则化模型。

## 关键点

- SLORR提出两种变体（Hoyer稀疏度和核范数），无需SVD或额外参数，保持原始架构。
- 通过GPU友好的前向/反向传播近似，训练开销低于1%～8%，并具有近似保证。
- 在图像分类和LLM预训练中验证，压缩后模型性能损失显著低于无正则化基线。

## 全文

Low-rank factorization is widely used to compress neural networks, but modern models are often not naturally amenable to aggressive factorization without significant accuracy loss. Existing training-time low-rank regularizers can improve compressibility, but they often require SVDs of large weight matrices, modify the model architecture (introducing additional trainable parameters), or rely on stateful cached quantities. To address these limitations, we introduce SLORR, a simple, stateless, and architecture-preserving framework for in-training low-rank regularization, instantiated with two main variants based on the Hoyer sparsity metric and the nuclear norm. SLORR directly regularizes the original weight matrices using GPU-friendly approximations for the forward and backward passes of the regularizers, for which we provide approximation guarantees. We first evaluate SLORR on ImageNet-1K across short-horizon continued training of ResNet-50, ViT-B/16, and ViT-L/16, and pretraining of ResNet-18, where SLORR induces compressibility while introducing less than 8% training overhead. We further evaluate SLORR-Hoyer in LLM pretraining at 135M and 560M scales: SLORR-trained compressed models preserve performance substantially better than unregularized models while adding less than 1% average training overhead.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_4d760f0976b465d3343ffb1d