---
title: "SLORR: Simple and Efficient In-Training Low-Rank Regularization"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.08754v1"
permalink: "https://feed.astralor.com/items/itm_4d760f0976b465d3343ffb1d"
published: "2026-07-09T17:51:50.000Z"
score: 78
channel: "ai"
tags: ["Hoyer Sparsity","ResNet-50","SLORR","Low-Rank Regularization","Nuclear Norm"]
---

# SLORR: Simple and Efficient In-Training Low-Rank Regularization

## AI commentary

> The strength of this paper lies in its simplicity and efficiency—it can be integrated into existing training pipelines without complex modifications. For researchers working on model compression and edge deployment, SLORR offers a low-cost way to improve compressibility. Particularly noteworthy is its performance in LLM pretraining with less than 1% overhead, which is highly attractive for real-world large-scale training.

## At a glance

SLORR is a stateless, architecture-preserving low-rank regularization framework instantiated with variants based on the Hoyer sparsity metric and the nuclear norm. It directly regularizes original weight matrices using GPU-friendly approximations, inducing compressibility with less than 8% training overhead on ImageNet-1K experiments (ResNet-50, ViT) and less than 1% overhead in LLM pretraining at 135M and 560M scales, while preserving significantly better compressed model performance than unregularized models.

## Key points

- SLORR introduces two variants (Hoyer sparsity and nuclear norm) with no SVD, extra parameters, or architecture modifications.
- GPU-friendly forward and backward approximations incur less than 1%–8% training overhead with approximation guarantees.
- Validated on image classification and LLM pretraining, compressed models suffer substantially less performance degradation than unregularized baselines.

## Full text

Low-rank factorization is widely used to compress neural networks, but modern models are often not naturally amenable to aggressive factorization without significant accuracy loss. Existing training-time low-rank regularizers can improve compressibility, but they often require SVDs of large weight matrices, modify the model architecture (introducing additional trainable parameters), or rely on stateful cached quantities. To address these limitations, we introduce SLORR, a simple, stateless, and architecture-preserving framework for in-training low-rank regularization, instantiated with two main variants based on the Hoyer sparsity metric and the nuclear norm. SLORR directly regularizes the original weight matrices using GPU-friendly approximations for the forward and backward passes of the regularizers, for which we provide approximation guarantees. We first evaluate SLORR on ImageNet-1K across short-horizon continued training of ResNet-50, ViT-B/16, and ViT-L/16, and pretraining of ResNet-18, where SLORR induces compressibility while introducing less than 8% training overhead. We further evaluate SLORR-Hoyer in LLM pretraining at 135M and 560M scales: SLORR-trained compressed models preserve performance substantially better than unregularized models while adding less than 1% average training overhead.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_4d760f0976b465d3343ffb1d