---
title: "Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.14051v1"
permalink: "https://feed.astralor.com/items/itm_ac1c3092126470dd4207e722"
published: "2026-07-15T17:21:43.000Z"
score: 75
channel: "ai"
tags: ["Hindcast","LLM Evaluation","Polymarket","Information Leakage","Prediction Markets"]
---

# Hindcast: Replaying Prediction Markets to Evaluate LLM Forecasters

## AI commentary

> This paper tackles the core flaw of LLM forecasting evaluation—information leakage—with an elegantly simple solution, Hindcast. If you work on AI forecasting, AI safety, or market analysis, this is a must-read. For those studying LLM capabilities, it helps distinguish genuine reasoning from mere recall.

## At a glance

This paper introduces Hindcast, a method to fairly evaluate LLMs' forecasting ability. Traditional backtesting suffers from information leakage: LLMs can surface post-event reports or exploit training data containing future knowledge, turning foresight into recall. Hindcast closes both leaks by freezing a Reddit snapshot at a chosen past date t₀, allowing the model to read only posts before t₀. It replays resolved Polymarket prediction markets and compares forecasts against both outcomes and market prices at t₀. Experiments show that once leaks are closed, retrieval helps only when Reddit discussed the event beforehand; otherwise, it hurts performance.

## Key points

- Traditional LLM forecasting evaluation suffers from two leakage channels: retrieving post-event reports and training data containing future knowledge.
- Hindcast eliminates both leaks by freezing a Reddit snapshot and setting per-market cutoffs, ensuring the test measures genuine foresight.
- Experiments show that with leaks closed, retrieval helps only when Reddit had prior discussion; otherwise it degrades accuracy.
- The evaluation is reusable on new markets without going stale.

## Full text

Forecasters are evaluated by backtesting, which replays resolved questions and grades the probability the system would have assigned before the outcome was known. For LLMs, two channels leak the answer into this test. A model that retrieves can surface reports written after the event, turning forecasting into a lookup, and each new model is trained on data closer to the event, so a question that lay in the future for last year's models sits inside this year's training data. Either way, the test grades recall while claiming to grade foresight. We introduce Hindcast, which closes both leaks by grading a model as if it stood at a chosen past date $t_0$, before the outcome existed in either channel. Hindcast replays resolved Polymarket prediction markets against a frozen snapshot of public Reddit, lets the model read only posts written before $t_0$, and scores each forecast against both what happened and the market's own price at $t_0$, itself a human forecast made from the same past information. Because the cutoff is set per market and the snapshot never changes, the evaluation re-runs on new markets as models improve, without going stale. Once the leak is closed, retrieval still helps most models, but only where Reddit discussed the event beforehand. Where the archive carried only speculation, retrieval hurts.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_ac1c3092126470dd4207e722