---
title: "将随机机器分组：精确性而非能力，应成为 AI 系统的前沿评估指标"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.19140v1"
permalink: "https://feed.astralor.com/items/itm_ccabf2f345fb8b505e1f886b"
published: "2026-08-19T17:29:47.000Z"
score: 66
channel: "ai"
tags: ["AI系统评测","arXiv","前沿语言模型","精度指标"]
---

# 将随机机器分组：精确性而非能力，应成为 AI 系统的前沿评估指标

## AI 点评

> 这篇论文值得关注的不是它给了最终答案，而是它指出了评测体系的一个盲区：我们一直在比谁打得准，却忘了比弹着点是否聚拢。对正在做模型选型或 Agent 稳定性优化的团队，这个分组指标可以当作一个免费的工作坊练习，花几小时就能跑出自己系统的精度画像。

## 速览

前沿语言模型在能力指标上已趋饱和，平均输出都能命中目标。本文提出，真正区分系统的已不是能力（平均落点），而是精度（重复相同请求后输出围绕目标的集中程度）。作者借鉴射击术语，给出可低成本测量的分组指标：固定温度下多次运行确定性评分任务，计算一致性。该指标能区分集中偏移（可调准）与散乱失败（需换模型），并已在实测中被单一规则完全改变结果。

## 关键点

- 前沿模型的能力已饱和，均值无法区分系统，精度（输出围绕目标的集中程度）成为关键差异。
- 提出分组指标：固定温度下重复运行确定性评分任务，计算单任务结果一致性，无需模型作为评分器。
- 指标可指导决策：集中偏移可视为准星调整问题，散乱失败则需更换模型或采样方式。
- 实测发现，单一规则即可完全修复某个精度差距，而由规则自身构造的任务无法测出价值。

## 全文

Frontier language models are compared, marketed, and benchmarked on capability -- what their best or average output can achieve. I argue this measures the wrong axis. The models have saturated accuracy: their mean output lands on the target. What now separates one system from another in practice is precision: how tightly concentrated their outputs are around that target across repeated, identical requests. Borrowing the marksman's distinction, capability is where the average shot lands; reliability is the size of the group. I make three claims. First, precision, not capability, is the frontier differentiator between systems, and benchmark culture systematically fails to measure it, reporting central tendency rather than spread. Second, precision is measurable, cheaply and without circularity, by running a fixed suite of deterministically scored tasks many times at fixed temperature and computing the per-task consistency of outcomes -- no model-in-the-loop grader required. Third, the measurement is not merely descriptive but decision-guiding: it separates consistent failures (a tight group off-centre, correctable by the operating discipline of Paper 1 -- a sight adjustment) from scattered failures (a wide group, correctable only by changing the model or its sampling -- a rifle problem). I define a grouping metric, specify a harness, and show how tracking a human-AI pair's grouping over time yields the compounding signal that Paper 1's field study requires. A first real run, since replicated, illustrates both the method and its most important limit: one measured gap was closed completely by a single rule (0/5 -> 5/5), while a suite of tasks authored from the rules themselves found no value, because a frontier model already embodies explicit good practice -- establishing that a discipline's worth is found by measurement on real work, not constructed from its own rulebook.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_ccabf2f345fb8b505e1f886b