---
title: "Phantom Gains: Auditing Self-Improvement Against a Measured Null"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.20290v1"
permalink: "https://feed.astralor.com/items/itm_b3f2ae3bdf7b8addacc15bba"
published: "2026-08-20T17:30:14.000Z"
score: 78
channel: "ai"
tags: ["LoRA","Qwen3-8B","Measurement Artifacts","Knowledge Distillation","Self-training"]
---

# Phantom Gains: Auditing Self-Improvement Against a Measured Null

## AI commentary

> This paper is essential for teams working on model self-training or comparative evaluation. It reminds us that without proper controls, many 'capability gains' may be phantom artifacts from inference settings or statistical noise. I recommend adopting its null-audit approach as a standard check in evaluation pipelines.

## At a glance

Auditing self-improvement in language models requires distinguishing real gains from measurement artifacts. This paper examines three rounds of rank-32 LoRA self-training on Qwen3-8B against a frozen control through an identical pipeline, identifying seven measurement failures that invert reported findings when controls are absent. A single greedy-decode ledger fabricates capability changes on an untrained model due to inference batching; the expansion statistic assigns that model a rate of 0.280. The authors propose a per-problem exact test under false-discovery-rate control, which finds nothing on held-out replicates and remains stable under multiple-testing rules. Applied to a matched ladder of arms, external distillation improves problems rarely reached by the base model, while three self-training forms do not; a regression rejects the asymmetry as a byproduct of distillation's larger overall gain (p<10^-8). Self-training corrupts baseline-solved problems at high rates. The study concludes that transition-level auditing demands a separately measured null for every statistic, built from baseline replicates already owned by multi-arm studies.

## Key points

- Proposes auditing self-training evaluation with frozen controls, identifying seven measurement failures, several of which are standard practice.
- Shows that single greedy decoding and inference batching can fabricate capability gains on an untrained model, with an expansion statistic misreporting a rate of 0.280.
- Introduces a per-problem exact test under FDR control that detects nothing on held-out replicates and is robust across multiple-testing rules.
- Finds external distillation outperforms three self-training forms, while self-training corrupts baseline-solved problems at high rates.

## Full text

Whether a language model has improved itself is increasingly judged not by mean accuracy but by which individual problems it gains and loses. Tracking these transitions means differencing two noisy estimates, leaving them vulnerable to measurement artifacts. Auditing three rounds of rank-$32$ LoRA self-training on Qwen3-8B against a frozen control pushed through the identical pipeline, we identify seven measurement failures, each of which inverts a reported finding when its control is absent. Several are standard practice. A ledger built on a single greedy decode manufactures capability changes on an untrained model, largely an artifact of inference batching; the expansion statistic separating acquisition from sharpening assigns that same model a rate of $0.280$. The natural threshold repair does not survive replication: estimated across the frozen comparisons such a design already contains, its null stays non-zero. We replace it with a per-problem exact test against a pooled baseline under false-discovery-rate control, which detects nothing on any held-out replicate and is unchanged under the multiple-testing rule, error rate and pool size. Applied to a ladder of arms matched in stream, volume and evaluation, the audit finds that external distillation improves problems the base model rarely reaches while three forms of self-training do not; a regression rejects this asymmetry as a by-product of distillation's larger overall gain ($p < 10^{-8}$). On the far smaller set of problems the base model never reaches, the evidence is inconclusive, while self-training corrupts problems solved at baseline at rates well above the measured floor. Transition-level auditing therefore requires a separately measured null for every statistic it reports: nulls that cost no new experiments, built from baseline replicates a multi-arm study already owns, though not from as few as most possess.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_b3f2ae3bdf7b8addacc15bba