---
title: "CHARM+: Cross-Hardware Attention with Re-Merge Multistream Mechanism"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2611.05906v1"
permalink: "https://feed.astralor.com/items/itm_c332b9277eb54a938a7bf472"
published: "2026-11-04T03:57:56.000Z"
score: 57
channel: "tech"
tags: ["LLM","RAG","CHARM","Cross-Hardware Attention","Re-Merge Multistream Mechanism","Ring-Attention","LLMs","Nevertheless"]
---

# CHARM+: Cross-Hardware Attention with Re-Merge Multistream Mechanism

## 速览

As sequence lengths continue to grow, distributed attention based on sequence parallel (e.g., Ring-Attention) has become a mainstream solution for large language models (LLMs). Nevertheless, inherent communication depen…

## 关键点

- As sequence lengths continue to grow, distributed attention based on sequence parallel (e.g., Ring-Attention) has become a mainstream solution for large language models (LLMs).
- Nevertheless, inherent communication dependencies limit computation-communication overlap, reducing GPU utilization.
- Moreover, in cross-hardware (NUMA or node) architectures, due to imbalanced communication bandwidth, frequent communication synchronization will limit overall performance.

## 全文

As sequence lengths continue to grow, distributed attention based on sequence parallel (e.g., Ring-Attention) has become a mainstream solution for large language models (LLMs). Nevertheless, inherent communication dependencies limit computation-communication overlap, reducing GPU utilization. Moreover, in cross-hardware (NUMA or node) architectures, due to imbalanced communication bandwidth, frequent communication synchronization will limit overall performance. Based on these insights, we propose CHARM+, Cross-Hardware Attention with Re-Merge Multistream Mechanism. We reorganize sequential communications into several levels and achieve intra-level parallelism and computation overlap. Furthermore, we decouple cross-hardware communication from intra-hardware communication, eliminating synchronization overhead caused by bandwidth imbalance. Forward and backward algorithms were evaluated on multiple hardware architectures and consistently outperformed state-of-the-art methods. Compared to Ring-Attention, our forward and backward algorithms achieve average speedups of 1.3x and 1.7x on a dual-NUMA PCIe architecture, and 2.8x and 2.9x on a dual-node NVLink architecture, respectively.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_c332b9277eb54a938a7bf472