---
title: "How Robust Are Automated Fact-Checking Systems? A Cross-Benchmark Evaluation Reveals Domain Shifts and Retrieval Bottlenecks"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.25934v1"
permalink: "https://feed.astralor.com/items/itm_325e79ac6bacff73c84ebe4a"
published: "2026-08-26T15:50:10.000Z"
score: 77
channel: "ai"
tags: ["AVeriTeC","ClimateCheck","SciFact","Large Language Models","Retrieval-Augmented Generation","Automated Fact-Checking"]
---

# How Robust Are Automated Fact-Checking Systems? A Cross-Benchmark Evaluation Reveals Domain Shifts and Retrieval Bottlenecks

## AI commentary

> This paper's value lies in hard data shattering the illusion of leaderboard universality—the same model can drop dramatically across domains. Beyond technical conclusions, it's a wake-up call for RAG and fact-checking engineering: if your retrieved evidence is noisy, even the strongest verifier won't save you. Teams doing real deployments should seriously study its domain-shift analysis and gold-evidence comparisons.

## At a glance

This paper presents the first comprehensive cross-benchmark evaluation of automated fact-checking (AFC) systems, spanning four datasets across scientific, open-web, and climate domains, comparing nine models from random/sparse baselines to fine-tuned transformers, zero-shot LLMs, and top AVeriTeC 2025 systems. Findings show fine-tuned models outperform zero-shot LLMs and top systems on ClimateCheck; rankings are highly domain- and metric-dependent; replacing retrieved evidence with gold annotations improves veracity accuracy by 14-22 points, confirming retrieval remains the primary bottleneck. Code and datasets are released.

## Key points

- Benchmarked nine AFC models across four datasets (scientific, open-web, climate), from baselines to AVeriTeC 2025 top systems.
- On ClimateCheck, fine-tuned models outperform zero-shot LLMs and AVeriTeC 2025 winners; noisy evidence degrades veracity prediction.
- Rankings are strongly domain- and metric-dependent: the best model on SciFact (macro-F1 0.70) drops to 0.31 on ClimateCheck.
- Replacing retrieved evidence with gold annotations improves veracity accuracy by 14-22 points, confirming retrieval as the primary bottleneck.

## Full text

Automated fact-checking (AFC) systems retrieve evidence and predict claim veracity, yet evaluations omit simple baselines, systems are developed for a single benchmark and cannot be trusted to generalise across domains. No prior work cross-evaluates the full two-stage retrieve-then-verify pipeline across diverse datasets, complementing retrieval-only studies (Thakur et al., 2021) and single-stage benchmarking studies (Calamai et al., 2025). We benchmark nine models, ranging from random and sparse baselines to fine-tuned transformers, zero-shot LLMs, and the two highest-ranked systems from the AVeriTeC 2025 shared task, across four datasets spanning scientific, open-web, and climate domains. Three findings stand out: (1) on ClimateCheck claim-only and fine-tuned models outperform zero-shot LLM and top-performing AVeriTeC 2025 systems, highlighting that noisy evidence can degrade veracity prediction; (2) system rankings are strongly domain- and metric-dependent: the best model on SciFact (macro-F1 0.70) drops to 0.31 on ClimateCheck, while the AVeriTeC 2025 winner and runner-up swap rankings based on evaluation metrics and datasets; (3) replacing retrieved evidence with gold annotations improves veracity accuracy by 14-22 points across models, confirming retrieval remains primary bottleneck. We release code, pre-processed datasets, and all results to support reproducible AFC research.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_325e79ac6bacff73c84ebe4a