---
title: "WorldCup Arena: Prospective, Leakage-Free Evaluation of Frontier LLMs on a Live Tournament"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.04008v1"
permalink: "https://feed.astralor.com/items/itm_4c9e04fa69cd16ece1d000d1"
published: "2026-08-04T17:59:55.000Z"
score: 76
channel: "ai"
tags: ["2026 FIFA World Cup","arXiv","WorldCup Arena","Large Language Models","Leakage-free Evaluation"]
---

# WorldCup Arena: Prospective, Leakage-Free Evaluation of Frontier LLMs on a Live Tournament

## AI commentary

> As an editor, what I find most valuable is not the accuracy numbers themselves but the demonstration that real-world forecasting evaluation cannot rely on post-hoc filtering—it must let time do the work. For people building model evals, agent decision systems, or sports data products, this benchmark is a strong reference.

## At a glance

Most benchmarks measuring forecasting ability of large language models are retrospective: the event has already happened, so evaluations must defend against memorisation. This paper flips the design. During the 39 days of the 2026 FIFA World Cup, six frontier LLMs—all with extended thinking and native server-side web search—were asked before each kickoff to fill a seven-market prediction card for all 104 matches, 12 group winners and an outright pool. Since no answer existed when asked, the evaluation is leakage-free by construction. With 4,494 scored predictions, the systems average 63.9% on match outcomes, matching the strategy of backing the bookmaker's favourite. They agree with one another far more often than they are right, so majority vote adds nothing; they under-commit to draws and goals, and crowd scoreline picks onto a single prototypical result. The benchmark, dossiers and scoring code are released.

## Key points

- The evaluation is prospective: questions were asked before kickoff, so no answer existed and leakage is impossible by construction.
- Across 4,494 predictions from six frontier LLMs on 104 matches, average outcome accuracy is 63.9%, matching a strategy of backing the bookmaker's favourite.
- Models agree with each other more often than they are right; majority voting adds no value, and scoreline picks collapse onto a single prototype.
- Leaderboard positions are stable at the top and bottom but churn in the middle, with narrow margins; the dataset and code are released.

## Full text

Benchmarks that measure the forecasting ability of large language models are almost always retrospective: the event has happened, the answer is somewhere on the Web, and the evaluation must defend itself against memorisation. We report the opposite design. Over the 39 days of the 2026 FIFA World Cup, six frontier LLMs -- all with extended thinking and native server-side web search -- were asked before every kickoff, one match at a time, to fill in a seven-market prediction card for all 104 matches, plus 12 group winners and a pre-tournament outright pool; no answer existed when the question was asked, so the evaluation is leakage-free by construction rather than by filtering, and the frozen archive holds 4,494 scored predictions. What the tournament establishes is a set of behaviours the six systems share. On match outcome they average 63.9%, level with backing the bookmaker's favourite -- which is in fact what they usually do. They agree with one another far more often than they are right, so a majority vote adds nothing. They under-commit to draws and to goals, and crowd their scoreline picks onto a single prototypical result. Accuracy tracks how lopsided a fixture is rather than how much is known about it: it collapses in the closest ties, where the dossiers are richest, while questions about the tournament as a whole are answered well. On this task the current generation of frontier systems is not sharply differentiated: the standings hold up at the top and the bottom across the run and churn in the middle, and the margins stay narrow throughout. The briefing dossiers, fixtures and official results are released as a benchmark, together with the scoring code.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_4c9e04fa69cd16ece1d000d1