---
title: "Hindcast：通过回放预测市场评估LLM预测器"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.14051v1"
permalink: "https://feed.astralor.com/items/itm_ac1c3092126470dd4207e722"
published: "2026-07-15T17:21:43.000Z"
score: 75
channel: "ai"
tags: ["Hindcast","LLM评估","Polymarket","信息泄漏","预测市场"]
---

# Hindcast：通过回放预测市场评估LLM预测器

## AI 点评

> 这篇论文直击LLM预测评估的核心痛点——信息泄漏，提出的Hindcast方法简单而巧妙。如果你正在做AI预测、AI安全或市场分析，这篇值得细读。尤其对研究LLM能力边界的朋友，它帮我们区分了真正的推理和记忆。

## 速览

论文提出Hindcast方法，用于公平评估LLM的预测能力。传统回测存在信息泄漏：LLM可通过检索事后报告或利用训练数据中的事后知识来“回忆”答案，而非真正“预测”。Hindcast通过冻结指定过去日期的Reddit快照，仅允许模型读取该日期前的帖子，从而消除两种泄漏途径。该方法对每个市场设定截止日期，并利用Polymarket市场的人类报价作为基准。实验发现，一旦泄漏被关闭，检索仅在Reddit事先讨论过事件时才有帮助，否则反而降低表现。

## 关键点

- 传统LLM预测评估存在两种信息泄漏：检索到事后报告和训练数据包含未来知识。
- Hindcast通过固定Reddit快照和按市场设定截止日期消除泄漏，保证测试的是真实预测能力。
- 实验表明，关闭泄漏后，检索仅在Reddit事先有讨论时有益，否则会损害预测准确度。
- 该方法可重复运行于新市场，不会因时间推移而失效。

## 全文

Forecasters are evaluated by backtesting, which replays resolved questions and grades the probability the system would have assigned before the outcome was known. For LLMs, two channels leak the answer into this test. A model that retrieves can surface reports written after the event, turning forecasting into a lookup, and each new model is trained on data closer to the event, so a question that lay in the future for last year's models sits inside this year's training data. Either way, the test grades recall while claiming to grade foresight. We introduce Hindcast, which closes both leaks by grading a model as if it stood at a chosen past date $t_0$, before the outcome existed in either channel. Hindcast replays resolved Polymarket prediction markets against a frozen snapshot of public Reddit, lets the model read only posts written before $t_0$, and scores each forecast against both what happened and the market's own price at $t_0$, itself a human forecast made from the same past information. Because the cutoff is set per market and the snapshot never changes, the evaluation re-runs on new markets as models improve, without going stale. Once the leak is closed, retrieval still helps most models, but only where Reddit discussed the event beforehand. Where the archive carried only speculation, retrieval hurts.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_ac1c3092126470dd4207e722