---
title: "ConvPE: Convolutional Position Embedding Improves Transformers"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2611.07749v1"
permalink: "https://feed.astralor.com/items/itm_fa780ec46530ac73a8f3169c"
published: "2026-11-05T09:55:15.000Z"
score: 54
channel: "tech"
tags: ["多模态","ConvPE","Convolutional Position Embedding","Improves Transformers ConvPE","Transformer"]
---

# ConvPE: Convolutional Position Embedding Improves Transformers

## 速览

ConvPE (Convolutional Position Embedding) improves both the performance and efficiency of Transformer models in language modeling, long-context extension, and vision. It injects a windowed, content-dependent bias into a…

## 关键点

- ConvPE (Convolutional Position Embedding) improves both the performance and efficiency of Transformer models in language modeling, long-context extension, and vision.
- It injects a windowed, content-dependent bias into attention scores through a lightweight causal convolution over a low-dimensional projection of the hidden states.
- As a lightweight plug-in, ConvPE is compatible with RoPE, attention masks, and KV concatenation.

## 全文

ConvPE (Convolutional Position Embedding) improves both the performance and efficiency of Transformer models in language modeling, long-context extension, and vision. It injects a windowed, content-dependent bias into attention scores through a lightweight causal convolution over a low-dimensional projection of the hidden states. As a lightweight plug-in, ConvPE is compatible with RoPE, attention masks, and KV concatenation. It also supports high-dimensional position embeddings via high-dimensional convolutions. In a reproduction, ConvPE outperforms five 1D baselines and eight 2D baselines. The code is publicly available at https://github.com/wzhagi/convpe.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_fa780ec46530ac73a8f3169c