---
title: "RealisticTritonBench：面向真实世界 AI 框架的 Triton 内核生成基准"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.12004v1"
permalink: "https://feed.astralor.com/items/itm_59a96d5ead2f21caad3030ed"
published: "2026-08-12T12:38:17.000Z"
score: 79
channel: "tech"
tags: ["LLM","RealisticTritonBench","Triton","基准测试"]
---

# RealisticTritonBench：面向真实世界 AI 框架的 Triton 内核生成基准

## AI 点评

> 这份工作最值得关注的是把评测从“翻译题”变成了“工程题”，真实 PR 的引入让模型必须面对上下文约束，而不仅仅是语法转换。对做 AI 编译器或自动代码生成的团队来说，这个基准可能是衡量进展的更好标尺，也提示我们现有模型的泛化能力还远未达到生产可用。

## 速览

现有针对大模型生成 Triton 内核的评测基准存在三大局限：任务仅限 PyTorch 到 Triton 的转换、只评估单核性能而非端到端性能、依赖可能被模型钻空子的手工评测脚本。为此，研究者提出 RealisticTritonBench，首个从主流 AI 框架真实 Pull Request 中提取 Triton 内核生成任务的基准。它保留具体工程上下文，将生成的内核集成回原框架，并通过端到端测试进行验证。实验表明，当前领先大模型在真实世界的 Triton 内核生成上仍面临挑战。

## 关键点

- RealisticTritonBench 从开源 AI 框架的真实 Pull Request 中提取 Triton 内核修改，转化为带工程背景的生成任务，覆盖比 PyTorch 到 Triton 转换更多样的场景。
- 该基准将生成的内核集成回原始框架，使用端到端测试评估，替代过去只测单核性能的做法，更贴近实际部署标准。
- 评测发现当前领先大模型在真实世界 Triton 内核生成任务上表现仍不理想，说明该方向存在较大提升空间。

## 全文

In modern AI frameworks, GPU kernels are key to overall system performance. Combining usability, portability, and near-handwritten CUDA performance, Triton is widely adopted for implementing GPU kernels. Recent advances show the potential of large language models (LLMs) to automatically generate Triton kernels, reducing the manual effort required from expert kernel developers. Several benchmarks evaluate LLM-generated Triton kernels. However, they suffer from three key limitations: (1) they restrict tasks to PyTorch-to-Triton translation, failing to reflect the diversity and complexity of real-world Triton tasks; (2) they evaluate only individual-kernel performance rather than end-to-end performance, the core criterion for real-world deployment in AI frameworks; and (3) they rely on manually written evaluation scripts for individual kernels, which may contain flaws that models can exploit to bypass correctness checks and obtain inflated scores. To address these limitations, we introduce RealisticTritonBench, the first benchmark to derive Triton kernel generation tasks from real-world pull requests in popular AI frameworks, enabling realistic, production-like evaluation. RealisticTritonBench systematically extracts PRs that modify Triton kernels from popular open-source AI frameworks and transforms them into generation tasks with concrete engineering contexts. Each task takes a natural language requirement as input and requires a corresponding Triton kernel implementation, with a complete and reproducible evaluation environment. Unlike prior benchmarks focused on isolated kernel performance, RealisticTritonBench integrates generated kernels into their original frameworks and evaluates them using end-to-end tests, enabling a more faithful assessment. We evaluate leading LLMs on RealisticTritonBench and find that they still struggle with real-world Triton kernel generation tasks.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_59a96d5ead2f21caad3030ed