---
title: "A Statistical Audit of Physical AI Benchmark Redundancy"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.25940v1"
permalink: "https://feed.astralor.com/items/itm_f0fc879d487fa993dbda4d36"
published: "2026-08-26T15:54:15.000Z"
score: 76
channel: "ai"
tags: ["arXiv","Bradley-Terry Model","Benchmark","Physical AI"]
---

# A Statistical Audit of Physical AI Benchmark Redundancy

## AI commentary

> This paper is a wake-up call for the physical AI community: we may be repeatedly testing the same capabilities while thinking we get diverse evaluations. For teams running evaluations, this greedy selection method can directly help prune redundant benchmarks, saving time and compute; for benchmark researchers, it offers a reusable statistical toolkit that transfers to other domains. Do look closely at the Bradley-Terry fitting part, as it is key to ranking stability.

## At a glance

Physical AI models are evaluated on suites of benchmarks that differ across model reports, leaving the model-by-benchmark matrix sparse and the relationship between benchmarks unmeasured. This study constructs a matrix of 51 models on 12 physical AI benchmarks, demonstrating significant benchmark redundancy: collapsing two substitute pairs moves 22 of 51 models by three or more places under an equally weighted average. A greedy selection obtains a four-benchmark subset retaining 78.5% of the utility of all 12, on which a Bradley-Terry ranking is fitted. The procedure requires only benchmark-level scores with sufficient overlap and is not specific to physical AI.

## Key points

- Constructs a score matrix of 51 models on 12 physical AI benchmarks and measures shared information to quantify redundancy.
- Collapsing two substitute pairs moves 22 of 51 models by three or more places, showing redundancy affects reported rankings.
- A greedy selection yields a four-benchmark subset retaining 78.5% of the utility of all 12, on which a Bradley-Terry ranking is fitted.
- The procedure requires only benchmark-level scores with sufficient overlap and is broadly applicable beyond physical AI.

## Full text

Physical AI models are evaluated on suites of benchmarks that differ across model reports, leaving the model-by-benchmark matrix sparse and the relationship between benchmarks unmeasured. We construct a matrix of 51 models on 12 physical AI benchmarks, selected from a registry of 51 benchmarks and 152 models by reporting density, combining scores from model cards and benchmark papers with our own evaluation runs under each benchmark's official protocol. We measure how much information the benchmarks share and show quantitative evidence of Redundancy. Redundancy affects reported rankings: collapsing the two substitute pairs into single columns moves 22 of 51 models by three or more places under an equally weighted average. We then select benchmarks greedily under a utility combining score dispersion with variance not explained by the already-selected set, and obtain a four-benchmark subset retaining 78.5\% of the utility of all 12, on which we fit a Bradley--Terry ranking. The procedure requires only benchmark-level scores with sufficient overlap and is not specific to physical AI.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_f0fc879d487fa993dbda4d36