---
title: "Candidate Supply and Answer Selection Shape the Value of LLM Judging in Multi-Agent Systems"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.25937v1"
permalink: "https://feed.astralor.com/items/itm_dbc53d9aee941a8c3b3a2892"
published: "2026-08-26T15:52:20.000Z"
score: 73
channel: "ai"
tags: ["arXiv","LLM Judging","Candidate Generation","Multi-agent Systems","Answer Selection"]
---

# Candidate Supply and Answer Selection Shape the Value of LLM Judging in Multi-Agent Systems

## AI commentary

> This paper cleanly decomposes failure causes in multi-agent systems, especially relevant for those building agent frameworks and voting/selection mechanisms. It reminds us that simply increasing model count is not enough; when the correct answer already exists in the candidate pool, the real bottleneck may lie in the selection rule.

## At a glance

This paper conceptualizes multi-agent reasoning as an evolutionary pipeline of candidate generation, peer communication, and terminal selection, showing that correct candidates often already exist yet the system still reports wrong answers. Based on 16,278 questions, 81,390 replayed candidate pools, and 15,336 judge reliability tests, they propose a combined rule using answer frequency plus judge evaluation, lifting accuracy from 63.82% to 70.82-70.95%, mainly by rescuing correct answers outnumbered by popular errors. Judge reliability is not a fixed trait but varies with task, generator, and answer rarity. The findings establish a diagnostic basis for designing multi-agent architectures that prevent loss of generated correct answers.

## Key points

- Correct candidates are often already generated, yet the multi-agent system still converges to a wrong answer;
- LLM judge reliability varies with task, generator, and answer rarity, not a fixed trait;
- Combining answer frequency with judge evaluation improves accuracy by about 7 points across benchmarks;
- The value of generating more candidates depends on whether extra samples make correct answers present, frequent, or recognizable.

## Full text

Multi-agent systems (MAS) sometimes already have the potential to answer correctly, but still report a wrong answer. Explaining this outcome is difficult because generation, communication and final answer-selection rules usually change simultaneously. We conceptualize multi-agent reasoning as an evolutionary pipeline of candidate generation, peer communication and terminal selection, wherein consensus without quality control can exhibit patterns of memetic drift. We study two questions: (1) when an LLM judge provides effective selection pressure by supplying a signal of answer correctness for candidates generated in a multi-agent system, and (2) when using that signal improves the reported answer. To map judge reliability, we analysed 15,336 questions from MMLU-Pro, GPQA, MedXpertQA and MuSR, with Humanity's Last Exam analysed separately. To test these rules, we replayed 81,390 fixed candidate pools drawn from 16,278 questions across five benchmarks. We report three findings. (1) A correct answer is often already present among the generated candidates, but the system can still converge on and report a wrong answer. (2) Judge reliability is not a fixed trait of the model, but varies with the task, the generator and how rare the correct answer is. (3) Combining answer frequency with the judge's evaluation changed only the final answer-selection rule and raised accuracy from 63.82% to 70.82-70.95%, primarily by rescuing correct answers that were outnumbered by popular errors. In the systems studied here, the value of generating more candidates depends on whether those extra samples make correct answers present, frequent or recognisable. By isolating generation, recognition and selection, these findings establish a diagnostic basis for designing multi-agent architectures that protect generated correct answers from being lost.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_dbc53d9aee941a8c3b3a2892