SFGA: A Statistics-First Gating Architecture with Adjudicative Escalation for Trustworthy SFT Data Procurement

Quick overview

This paper proposes SFGA, a statistics-first gating architecture that treats SFT data procurement as a cost-aware routing problem. The system computes cheap blind measurements across three quality axes (diversity, utility, redundancy) and accepts a decision only when confidence intervals are tight, sample sizes adequate, and axes agree; otherwise, it escalates to a buy/reject adjudicative debate. On a benchmark of 12 datasets, the gate achieves 0.90 accuracy and 0.83 F1 at $0.017 per unit, undercutting the always-escalate baseline ($0.020). Experiments further reveal a con-side win rate of 0.80 and a 52% position-flip rate under advocate swapping, exposing negativity and positional biases that naive LLM judges would hide.

• Proposes SFGA, a statistics-first gating architecture combining cheap measurements and adjudicative escalation for cost-effective SFT data procurement.
• Makes cost-aware decisions on diversity, utility, and redundancy axes, auto-passing only under high confidence, otherwise triggering debate.
• Benchmark achieves 0.90 accuracy and 0.83 F1, with cost lower than always-escalate and close to oracle upper bound.
• Reveals significant negativity bias (80% con-side win rate) and 52% position-flip rate in debate, warning of hidden biases in LLM judges.