The Test Oracle Problem in Synthetic LLM-as-Judge Corpora: Disappearance, Distortion and a Validation Protocol

In LLM-as-judge evaluations, researchers typically construct synthetic corpora by prompting an LLM to generate a hallucinated answer paired with a factual one. This paper reports a case where a shared decoding-budget parameter (max_tokens) truncated hallucinated answers to a few words in a Turkish/English faithfulness-judgment corpus, causing a spurious cross-lingual accuracy drop of 32 percentage points. Through three-layer mechanistic analysis and producer-swap experiments, the authors demonstrate this effect is entirely due to data contamination. Crucially, traditional aggregate statistical metrics (accuracy, confidence intervals) cannot detect this fault; only manual reading of raw generations reveals it. The authors term this the 'test oracle problem': for LLM-generated negatives, there is no mechanical way to verify item integrity. In contrast, corpora built by deterministic perturbation (e.g., entity substitution) carry an item-level oracle for free. A positive control experiment shows that a zero-cost string comparison catches analogous faults with 100% accuracy. The paper concludes with a validation protocol for most contemporary multilingual LLM-as-judge corpora, including manual sampling, parameter sensitivity analysis, and deterministic perturbation robustness checks.