---
title: "RealisticTritonBench: A Benchmark for Triton-Kernel Generation in Real-World AI Frameworks"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.12004v1"
permalink: "https://feed.astralor.com/items/itm_59a96d5ead2f21caad3030ed"
published: "2026-08-12T12:38:17.000Z"
score: 79
channel: "tech"
tags: ["LLM","RealisticTritonBench","Triton","Benchmark"]
---

# RealisticTritonBench: A Benchmark for Triton-Kernel Generation in Real-World AI Frameworks

## AI commentary

> What stands out here is shifting evaluation from translation-like tasks to real engineering problems; using real PRs forces models to handle contextual constraints, not just syntax conversion. For teams working on AI compilers or automatic code generation, this benchmark could be a better yardstick for progress, and it reminds us that current models are far from production-ready in this area.

## At a glance

Existing benchmarks for evaluating LLM-generated Triton kernels suffer from three key limitations: restricting tasks to PyTorch-to-Triton translation, evaluating only individual-kernel performance rather than end-to-end performance, and relying on manually written evaluation scripts that models can exploit to bypass correctness checks. To address these, the authors introduce RealisticTritonBench, the first benchmark that derives Triton kernel generation tasks from real-world pull requests in popular AI frameworks. It preserves concrete engineering contexts, integrates generated kernels back into their original frameworks, and evaluates them using end-to-end tests. Experiments show that leading LLMs still struggle with real-world Triton kernel generation tasks.

## Key points

- RealisticTritonBench extracts Triton kernel modifications from real pull requests in open-source AI frameworks, converting them into generation tasks with engineering context, covering more diverse scenarios than PyTorch-to-Triton translation.
- It integrates generated kernels back into their original frameworks and evaluates with end-to-end tests, replacing previous individual-kernel performance checks to better reflect real deployment standards.
- Evaluation shows that leading LLMs still perform poorly on real-world Triton kernel generation, indicating significant room for improvement.

## Full text

In modern AI frameworks, GPU kernels are key to overall system performance. Combining usability, portability, and near-handwritten CUDA performance, Triton is widely adopted for implementing GPU kernels. Recent advances show the potential of large language models (LLMs) to automatically generate Triton kernels, reducing the manual effort required from expert kernel developers. Several benchmarks evaluate LLM-generated Triton kernels. However, they suffer from three key limitations: (1) they restrict tasks to PyTorch-to-Triton translation, failing to reflect the diversity and complexity of real-world Triton tasks; (2) they evaluate only individual-kernel performance rather than end-to-end performance, the core criterion for real-world deployment in AI frameworks; and (3) they rely on manually written evaluation scripts for individual kernels, which may contain flaws that models can exploit to bypass correctness checks and obtain inflated scores. To address these limitations, we introduce RealisticTritonBench, the first benchmark to derive Triton kernel generation tasks from real-world pull requests in popular AI frameworks, enabling realistic, production-like evaluation. RealisticTritonBench systematically extracts PRs that modify Triton kernels from popular open-source AI frameworks and transforms them into generation tasks with concrete engineering contexts. Each task takes a natural language requirement as input and requires a corresponding Triton kernel implementation, with a complete and reproducible evaluation environment. Unlike prior benchmarks focused on isolated kernel performance, RealisticTritonBench integrates generated kernels into their original frameworks and evaluates them using end-to-end tests, enabling a more faithful assessment. We evaluate leading LLMs on RealisticTritonBench and find that they still struggle with real-world Triton kernel generation tasks.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_59a96d5ead2f21caad3030ed