---
title: "Going Deeper With Memories"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2611.05255v1"
permalink: "https://feed.astralor.com/items/itm_55f0e5a85436e669d9277e7f"
published: "2026-11-03T17:44:07.000Z"
score: 57
channel: "tech"
tags: ["推理效率","RAG","Going Deeper With","Memories Byte","DeepMem","Byte","Fast","Transformer"]
---

# Going Deeper With Memories

## 速览

Byte level language models avoid tokenizer dependence and provide universal coverage, but processing every byte at full resolution is expensive. We introduce DeepMem, a causal multiscale architecture that separates sema…

## 关键点

- Byte level language models avoid tokenizer dependence and provide universal coverage, but processing every byte at full resolution is expensive.
- We introduce DeepMem, a causal multiscale architecture that separates semantic compression from exact byte preservation.
- A learned invertible subband transform constructs a hierarchy of shorter sequences, while a lossless carrier and explicit detail sidebands preserve byte identity.

## 全文

Byte level language models avoid tokenizer dependence and provide universal coverage, but processing every byte at full resolution is expensive. We introduce DeepMem, a causal multiscale architecture that separates semantic compression from exact byte preservation. A learned invertible subband transform constructs a hierarchy of shorter sequences, while a lossless carrier and explicit detail sidebands preserve byte identity. Fast weight memory blocks operate along the encoder, with iterative shared state computation at the compressed bottleneck, while an attention based super resolution decoder restores byte resolution representations. DeepMem achieves stronger language modeling performance than a matched byte level Transformer on FineWeb-Edu and PG19, and obtains the best macro average performance across the evaluated FLORES languages. It also improves PIQA accuracy while remaining competitive on code modeling, multiple choice reasoning, and byte level corruption robustness. Bottleneck refinement requires approximately $7{,}600\times$ fewer counted FLOPs than equivalent full resolution reasoning, while maintaining competitive evaluation throughput.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_55f0e5a85436e669d9277e7f