---
title: "World Models Need Not Be Right to Rank Policies Right: Relative Bounds and Audit Limits"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2611.01139v1"
permalink: "https://feed.astralor.com/items/itm_0a47e768ee0ee368570ff265"
published: "2026-10-31T17:30:15.000Z"
score: 57
channel: "ai"
tags: ["LLM","RAG","World Models Need","Not","Right","Rank Policies Right","Relative Bounds","Audit Limits Selecting"]
---

# World Models Need Not Be Right to Rank Policies Right: Relative Bounds and Audit Limits

## At a glance

Selecting a reinforcement-learning (RL) policy usually requires additional environment interaction, while learned world models offer cheap but potentially biased evaluations. This raises two distinct questions: can shar…

## Key points

- Selecting a reinforcement-learning (RL) policy usually requires additional environment interaction, while learned world models offer cheap but potentially biased evaluations.
- This raises two distinct questions: can shared model errors preserve pairwise rankings despite poor absolute values, and do they make a small number of real audits more informative?
- We derive an exact pairwise simulation identity and a relative bound separating common-mode Bellman residuals from policy-specific error.

## Full text

Selecting a reinforcement-learning (RL) policy usually requires additional environment interaction, while learned world models offer cheap but potentially biased evaluations. This raises two distinct questions: can shared model errors preserve pairwise rankings despite poor absolute values, and do they make a small number of real audits more informative? We derive an exact pairwise simulation identity and a relative bound separating common-mode Bellman residuals from policy-specific error. The analysis motivates RAVEL (Relative Auditing of Value Errors in Learned models), a selective policy tournament whose eliminations use held-out real outcomes. We test 45 SAC, TD3, and PPO policies per task in DMC cheetah-run and walker-walk. At rollout horizon 500, model rankings remain strong (Kendall tau = 0.962 and 0.800), and same-run differential-error ratios are 0.674 and 0.106. Yet mean paired audit-error correlations are only 0.007 and 0.090, and pairing changes standard deviation by -2.5 percent and +1.9 percent. Simultaneous intervals attain 99.98 percent and 99.75 percent coverage, but certify only 13.2 percent and 0 percent of decisions at eight audits. RAVEL improves low-budget walker-walk regret, whereas real-only successive halving is better at the largest budget, rejecting the precommitted twofold efficiency target. Frozen-model optimization lowers real return without increasing predicted return on average. Thus ranking fidelity, useful audit covariance, and upward model exploitation are distinct properties, separated in part by closed-loop trajectory divergence.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_0a47e768ee0ee368570ff265