---
title: "ProxyFormer：面向超长上下文与高分辨率生成的双流代理架构"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.23463v1"
permalink: "https://feed.astralor.com/items/itm_9459957c7c7400d3dbbd0ba4"
published: "2026-08-24T16:31:46.000Z"
score: 75
channel: "ai"
tags: ["KV 缓存","ProxyFormer","注意力机制","流匹配","超长上下文"]
---

# ProxyFormer：面向超长上下文与高分辨率生成的双流代理架构

## AI 点评

> 这篇论文让我眼前一亮，它没有在注意力机制上打补丁，而是用双流代理把全局交互挪到压缩空间，算是从架构层面解决二次复杂度。对做超长文本推理和高效生成的研究者尤其值得精读；哪怕只借鉴动态压缩比或仅代理 KV 缓存推理，也可能带来新的优化思路。

## 速览

ProxyFormer 提出基于代理 token 的双流架构：局部特征压缩为代理状态，全局交互仅在压缩空间进行，再解压回局部流，避免一次性压缩的不可逆信息丢失。压缩比 64 时，16GB GPU 可训练长度从 2 万扩至 70 万 token。64K 窗口模型在 104 万 token 检索中保持 92%-95% 准确率，8K 窗口可外推 256K 且超 94%。图像生成实验验证了像素/潜空间流匹配的可行性。

## 关键点

- 双流代理架构：局部流跨层保留，全局交互在压缩代理空间完成，减少注意力与 KV 缓存开销。
- 引入因子化多级压缩/解压、逐层动态压缩比、非对称双嵌入及仅代理 KV 缓存推理。
- 16GB GPU 上压缩比 64 可将可训练序列长度从约 20K 扩展到约 0.7M。
- 64K 窗口模型在 104 万 token 多针检索中保持 92%-95% 准确率，8K 窗口模型外推 256K 时超 94%。

## 全文

The quadratic growth of attention computation and key-value (KV) cache with respect to sequence length is a central bottleneck for ultra-long-context language models and high-resolution generative models. We propose ProxyFormer, a general dual-stream architecture built upon proxy tokens. In each layer, fine-grained local features are compressed bottom-up into a small set of proxy states; expensive global interactions are performed only in the compressed proxy space; the globally contextualized proxies are then decompressed and injected top-down back into the local stream. Because the local stream persists across layers, fine-grained information that is not captured by one compression step remains accessible for later refinement, alleviating the irreversible information loss of conventional one-shot compression. We further introduce factorized multi-level compression/decompression, layer-wise dynamic compression ratios, asymmetric dual embeddings, and a proxy-only KV-cache inference scheme. On a 16GB GPU with batch size 1, a standard decoder-only model can train sequences of only about 20K tokens, whereas ProxyFormer with a compression ratio of 64 extends the trainable sequence length to about 0.7M. A model trained with a 64K window retains 92%-95% retrieval accuracy on a multi-needle retrieval task with 1,048,576 tokens, and a model trained with an 8K window exceeds 94% accuracy when extrapolated to 256K tokens. Preliminary image-generation experiments demonstrate the feasibility of ProxyFormer for both pixel-space and latent-space flow matching.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_9459957c7c7400d3dbbd0ba4