---
title: "Causal Attention on CPU: A Cache-Aware and Load-Balanced Scheduling Kernel for Prefill"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2610.38809v1"
permalink: "https://feed.astralor.com/items/itm_04923c0fb7b573f36df2ce17"
published: "2026-10-28T20:08:51.000Z"
score: 50
channel: "tech"
tags: ["AI Agent","推理效率","Causal Attention","CPU","Cache-Aware","Load-Balanced Scheduling Kernel","Prefill Agentic","GPU"]
---

# Causal Attention on CPU: A Cache-Aware and Load-Balanced Scheduling Kernel for Prefill

## At a glance

Agentic AI drives significant productivity gains across diverse domains, but concurrent long-context prefills from multiple agents can saturate GPU compute resources. Although CPUs offer a viable alternative, existing C…

## Key points

- Agentic AI drives significant productivity gains across diverse domains, but concurrent long-context prefills from multiple agents can saturate GPU compute resources.
- Although CPUs offer a viable alternative, existing CPU attention kernels fall short in adapting to multi-tier cache hierarchies and lack effective thread scheduling.
- To address this, we introduce CACPU, a high-performance attention kernel featuring tiered cache blocking and an in-kernel task dispatching mechanism designed for causal full-sequence prefilling.

## Full text

Agentic AI drives significant productivity gains across diverse domains, but concurrent long-context prefills from multiple agents can saturate GPU compute resources. Although CPUs offer a viable alternative, existing CPU attention kernels fall short in adapting to multi-tier cache hierarchies and lack effective thread scheduling. To address this, we introduce CACPU, a high-performance attention kernel featuring tiered cache blocking and an in-kernel task dispatching mechanism designed for causal full-sequence prefilling. CACPU achieves up to a 2.01x speedup over PyTorch SDPA, FlexAttention, and SGLang attention kernels. Integrated into SGLang, it reduces Time to First Token (TTFT) of Llama-3.1-8B by up to 1.74x while preserving model accuracy.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_04923c0fb7b573f36df2ce17