---
title: "LLMs Don't Know What They Know: Benchmarking Temporal Self-Knowledge for Financial Backtesting"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2611.06808v1"
permalink: "https://feed.astralor.com/items/itm_5965a4d4c755e8c0ada42e82"
published: "2026-11-04T17:24:02.000Z"
score: 60
channel: "finance"
tags: ["LLM","OpenAI","LLMs Don","Know What They","Know","Benchmarking Temporal Self-Knowledge","Financial Backtesting Financial","LLMs"]
---

# LLMs Don't Know What They Know: Benchmarking Temporal Self-Knowledge for Financial Backtesting

## 速览

Financial backtests of large language models (LLMs) are meaningful only if the model did not already know the outcomes it is asked to predict. Practitioners guard against this by trusting a vendor's stated knowledge cut…

## 关键点

- Financial backtests of large language models (LLMs) are meaningful only if the model did not already know the outcomes it is asked to predict.
- Practitioners guard against this by trusting a vendor's stated knowledge cutoff, by asking the model what it knows, or by instructing it to answer as of a past date.
- To verify whether any of these safeguards can be trusted, we introduce FinDendro, a benchmark of 2,226 dated U.S.

## 全文

Financial backtests of large language models (LLMs) are meaningful only if the model did not already know the outcomes it is asked to predict. Practitioners guard against this by trusting a vendor's stated knowledge cutoff, by asking the model what it knows, or by instructing it to answer as of a past date. To verify whether any of these safeguards can be trusted, we introduce FinDendro, a benchmark of 2,226 dated U.S. facts from 1999 to 2026, covering monthly macroeconomic series, month-end stock prices and Federal Reserve decisions. Each fact yields four prompts: a value question, Recall and Predict questions that pair each candidate value with a claim of knowing or guessing without a declared date, and a Predict question with a historical declared date. Controls for persistence and dominant outcomes keep chance accuracy near 0.27. Results show that for most frontier models, self-reports understate knowledge. GPT-5 answers 76% of controlled facts correctly but claims to know 38%, and answers 61% of the facts it says it does not know. Knowledge claims can persist beyond the measured knowledge boundary, by up to seven quarters, and a declared date removes most of the claims of prior knowledge while leaving accuracy unchanged. Backtests inside a model's training window therefore cannot be certified leak-free by self-report, instruction or stated cutoff. These findings motivate behavioral measurement of temporal knowledge boundaries, and FinDendro provides a benchmark and procedure for selecting backtest periods based on demonstrated factual knowledge.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_5965a4d4c755e8c0ada42e82