---
title: "幻影增益：审计自我改进需对照实测零假设"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.20290v1"
permalink: "https://feed.astralor.com/items/itm_b3f2ae3bdf7b8addacc15bba"
published: "2026-08-20T17:30:14.000Z"
score: 78
channel: "ai"
tags: ["LoRA","Qwen3-8B","测量伪影","知识蒸馏","自训练"]
---

# 幻影增益：审计自我改进需对照实测零假设

## AI 点评

> 这篇论文对正在做模型自训练或对比评估的团队非常重要。它提醒我们，如果不设控制组，很多“能力提升”可能只是推理配置或统计噪声造成的幻影。我建议把其中的零假设审计方法作为评估流水线的标准检查项。

## 速览

该论文针对语言模型自我改进评估中的测量伪影问题，以 Qwen3-8B 三轮 LoRA 自训练为对象，用冻结对照识别出七种可使结论反转的测量失败。作者提出基于 FDR 的逐问题精确检验，发现外部蒸馏能改进基础模型罕见触及的问题，而三种自训练均不能；自训练反而以高比率破坏基线已解问题。研究强调转换级审计必须采用独立实测的零假设。

## 关键点

- 提出用冻结控制审计自训练评估，识别出七种测量失败，其中多种是常规做法。
- 发现单次贪婪解码与推理批处理可在未训练模型上制造虚假能力提升，扩展统计量误报率为 0.280。
- 构建基于 FDR 控制的逐问题精确检验，在保留复制上无检出，且对多重检验规则稳定。
- 外部蒸馏优于三种自训练，而自训练会以高比率破坏基线已解问题。

## 全文

Whether a language model has improved itself is increasingly judged not by mean accuracy but by which individual problems it gains and loses. Tracking these transitions means differencing two noisy estimates, leaving them vulnerable to measurement artifacts. Auditing three rounds of rank-$32$ LoRA self-training on Qwen3-8B against a frozen control pushed through the identical pipeline, we identify seven measurement failures, each of which inverts a reported finding when its control is absent. Several are standard practice. A ledger built on a single greedy decode manufactures capability changes on an untrained model, largely an artifact of inference batching; the expansion statistic separating acquisition from sharpening assigns that same model a rate of $0.280$. The natural threshold repair does not survive replication: estimated across the frozen comparisons such a design already contains, its null stays non-zero. We replace it with a per-problem exact test against a pooled baseline under false-discovery-rate control, which detects nothing on any held-out replicate and is unchanged under the multiple-testing rule, error rate and pool size. Applied to a ladder of arms matched in stream, volume and evaluation, the audit finds that external distillation improves problems the base model rarely reaches while three forms of self-training do not; a regression rejects this asymmetry as a by-product of distillation's larger overall gain ($p < 10^{-8}$). On the far smaller set of problems the base model never reaches, the evidence is inconclusive, while self-training corrupts problems solved at baseline at rates well above the measured floor. Transition-level auditing therefore requires a separately measured null for every statistic it reports: nulls that cost no new experiments, built from baseline replicates a multi-arm study already owns, though not from as few as most possess.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_b3f2ae3bdf7b8addacc15bba