---
title: "AV-AIVAT: 74x Cheaper Agent Evaluation with Certified Anytime-Valid Stopping in Imperfect-Information Games"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.06362v1"
permalink: "https://feed.astralor.com/items/itm_109def1169d75773db287d44"
published: "2026-08-06T17:57:11.000Z"
score: 76
channel: "ai"
tags: ["Agent Evaluation","AIVAT","AV-AIVAT","Confidence Sequences","Imperfect-Information Games"]
---

# AV-AIVAT: 74x Cheaper Agent Evaluation with Certified Anytime-Valid Stopping in Imperfect-Information Games

## AI commentary

> What I appreciate most here is that the paper does not stop at theory: over 70k HUNL hands, it turns saved samples into earlier decisions. For teams that compare dozens of agent versions daily, this can visibly compress evaluation cycles; for online-statistics researchers, the structural bound for EB-CS offers a useful framework.

## At a glance

Determining which of two agents is stronger requires playing games until evidence outweighs luck, and every game costs money, inference, or expert time. Fixed-budget evaluations either waste resources after the result is settled or stop too early to distinguish agents, while naive optional stopping with ordinary confidence intervals invalidates the guarantee. AV-AIVAT combines AIVAT's variance reduction with continuously monitored confidence sequences to deliver certified anytime-valid stopping in imperfect-information games. Across 15 LLM agent configurations spanning 71,439 paired Heads-Up No-Limit Hold'em hands, AIVAT reduces variance by a median 54x; at nominal 95% level and ±1 Big Blind precision, raw outcomes need a median 74x as many hands to stop. The Empirical-Bernstein CS provides exact finite-sample certification with a structurally justified bound on corrected payoffs, making the evaluation auditable and reproducible at the stopping time.

## Key points

- AIVAT cuts variance by a median 54x across HUNL hands but provides no stopping rule.
- AV-AIVAT pairs AIVAT with anytime-valid confidence sequences, reducing required hands to about 1/74 of raw evaluation at 95% confidence.
- EB-CS gives exact finite-sample certification, with a structural payoff bound established for Leduc hold'em.
- Evaluations can stop as soon as evidence suffices, leaving an auditable and reproducible record.

## Full text

Deciding which of two agents is stronger means playing games until skill outweighs luck, and every game costs money, model inference, or expert time. Since the number of games needed is unknown, fixed-budget evaluations either keep paying after the result is settled or stop before the agents can be told apart, while naive optional stopping with an ordinary confidence interval invalidates the stated level. We make such an evaluation stop as soon as its evidence suffices, with the guarantee intact. The Action-Informed Value Assessment Tool (AIVAT) reduces variance in imperfect-information games through conditional mean-zero corrections, by a median $54\times$ across 15 LLM agent configurations spanning 71,439 paired Heads-Up No-Limit Hold'em (HUNL) hands, but does not say when to stop. We combine AIVAT with continuously monitored Confidence Sequences (CSs) into anytime-valid AIVAT (AV-AIVAT), whose online value model learns only from past games so that no game scores its own correction. At the nominal 95\% level and a target precision of $\pm1$ Big Blind, raw outcomes need a median $74\times$ as many hands as AIVAT-corrected outcomes to stop under the Asymptotic CS (AsympCS). Exact finite-sample certification uses the Empirical-Bernstein CS (EB-CS), which needs an independently justified bound on corrected payoffs. We establish such a bound structurally for Leduc hold'em and characterize a width floor set by the CS's bet cap and that bound, which governs how much of a variance gain becomes earlier stopping; the descriptive HUNL EB-CS runs show a median $1.37\times$ stopping-time ratio. AV-AIVAT turns variance reduction into efficient, auditable early stopping while separating asymptotic screening from exact certification, so an evaluation can stop the moment its evidence suffices and hand a third party everything needed to recheck the verdict at that very stopping time.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_109def1169d75773db287d44