---
title: "ProxyFormer: A Dual-Stream Proxy Architecture for Ultra-Long Context and High-Resolution Generation"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.23463v1"
permalink: "https://feed.astralor.com/items/itm_9459957c7c7400d3dbbd0ba4"
published: "2026-08-24T16:31:46.000Z"
score: 75
channel: "ai"
tags: ["KV cache","ProxyFormer","Attention Mechanism","Flow Matching","Ultra-long context"]
---

# ProxyFormer: A Dual-Stream Proxy Architecture for Ultra-Long Context and High-Resolution Generation

## AI commentary

> This paper stands out because it doesn't just patch attention; it moves global interactions into a compressed proxy space via a dual-stream design, tackling quadratic complexity at the architectural level. It's a must-read for researchers working on ultra-long text inference and efficient generation—even the dynamic compression ratio and proxy-only KV-cache inference alone could inspire new optimization directions.

## At a glance

ProxyFormer proposes a dual-stream architecture based on proxy tokens. Fine-grained local features are compressed bottom-up into a small set of proxy states; expensive global interactions are performed only in the compressed proxy space; then the context-rich proxies are decompressed and injected top-down back into the local stream. Because the local stream persists across layers, information missed by one compression can be refined later, mitigating irreversible loss. With a compression ratio of 64 on a 16GB GPU, ProxyFormer extends the trainable sequence length from about 20K to about 0.7M tokens. A model trained with a 64K window retains 92%-95% retrieval accuracy on a 1,048,576-token multi-needle task, while an 8K-window model exceeds 94% accuracy extrapolated to 256K tokens. Preliminary image generation experiments confirm feasibility in both pixel-space and latent-space flow matching.

## Key points

- Dual-stream proxy architecture keeps the local stream persistent and performs global interactions in compressed proxy space, reducing attention and KV cache overhead.
- Introduces factorized multi-level compression/decompression, layer-wise dynamic compression ratios, asymmetric dual embeddings, and proxy-only KV-cache inference.
- Extends trainable sequence length from ~20K to ~0.7M tokens with compression ratio 64 on a 16GB GPU.
- Trained with 64K window retains 92%-95% accuracy on 1M-token multi-needle retrieval; 8K-window model exceeds 94% at 256K extrapolation.

## Full text

The quadratic growth of attention computation and key-value (KV) cache with respect to sequence length is a central bottleneck for ultra-long-context language models and high-resolution generative models. We propose ProxyFormer, a general dual-stream architecture built upon proxy tokens. In each layer, fine-grained local features are compressed bottom-up into a small set of proxy states; expensive global interactions are performed only in the compressed proxy space; the globally contextualized proxies are then decompressed and injected top-down back into the local stream. Because the local stream persists across layers, fine-grained information that is not captured by one compression step remains accessible for later refinement, alleviating the irreversible information loss of conventional one-shot compression. We further introduce factorized multi-level compression/decompression, layer-wise dynamic compression ratios, asymmetric dual embeddings, and a proxy-only KV-cache inference scheme. On a 16GB GPU with batch size 1, a standard decoder-only model can train sequences of only about 20K tokens, whereas ProxyFormer with a compression ratio of 64 extends the trainable sequence length to about 0.7M. A model trained with a 64K window retains 92%-95% retrieval accuracy on a multi-needle retrieval task with 1,048,576 tokens, and a model trained with an 8K window exceeds 94% accuracy when extrapolated to 256K tokens. Preliminary image-generation experiments demonstrate the feasibility of ProxyFormer for both pixel-space and latent-space flow matching.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_9459957c7c7400d3dbbd0ba4