---
title: "LooM: Loop-Invariant Memory for Causal Encoder-Decoder Language Models"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2611.06835v1"
permalink: "https://feed.astralor.com/items/itm_69ea9035ac1861c72fa18091"
published: "2026-11-04T17:52:11.000Z"
score: 58
channel: "tech"
tags: ["LooM","Loop-Invariant Memory","Causal Encoder-Decoder Language","Models Looped Transformers","Looped Transformers","Loop-invariant Memory","CED"]
---

# LooM: Loop-Invariant Memory for Causal Encoder-Decoder Language Models

## 速览

Looped Transformers increase effective depth by re-executing shared layers, but three limitations hinder their adoption: memory grows with the loop count, compute can only be adjusted in whole-block steps, and the loop …

## 关键点

- Looped Transformers increase effective depth by re-executing shared layers, but three limitations hinder their adoption: memory grows with the loop count, compute can only be adjusted in whole-block steps, and the loop …
- We propose LooM (Loop-invariant Memory), which addresses each in turn.
- In a causal encoder-decoder (CED), LooM loops only the decoder layers after the first, which read a global memory written once per token, and each of them publishes the key-value (KV) entry for a token only on its first…

## 全文

Looped Transformers increase effective depth by re-executing shared layers, but three limitations hinder their adoption: memory grows with the loop count, compute can only be adjusted in whole-block steps, and the loop count is hard to change after training. We propose LooM (Loop-invariant Memory), which addresses each in turn. In a causal encoder-decoder (CED), LooM loops only the decoder layers after the first, which read a global memory written once per token, and each of them publishes the key-value (KV) entry for a token only on its first visit, so the cache does not grow with the loop count. These layers loop independently, providing every budget from 3 to 21 layer executions per token, and one model trained on randomly sampled budgets covers all of them without a router. With enough loops, LooM outperforms a single-pass baseline trained on the same data on language modeling, zero-shot accuracy and long-context perplexity, while using the same cache size. Our analysis further shows that the training budget distribution decides what extra depth buys. These results suggest that a CED is a natural place to loop: memory is written once, and depth becomes a budget chosen at deployment.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_69ea9035ac1861c72fa18091