---
title: "Agon: Competitive Cross-Model RL with Implicit Rival Grading of Reasoning"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.07690v1"
permalink: "https://feed.astralor.com/items/itm_1e8c4b0ab67fe3218908c2fd"
published: "2026-07-08T17:49:14.000Z"
score: 79
channel: "ai"
tags: ["Agon","DeepMath","GRPO","Qwen3","Adversarial Training"]
---

# Agon: Competitive Cross-Model RL with Implicit Rival Grading of Reasoning

## AI commentary

> This paper introduces Agon, a clever method that implicitly improves reasoning chain quality through adversarial dual-model training, requiring no human or external reward model. For researchers and engineers working on RLHF or reasoning post-training, this is a promising new direction. The consistent gains across multiple settings demonstrate strong generality.

## At a glance

Agon is an innovative competitive cross-model reinforcement learning method that improves reasoning by having two models serve as each other's graders. Unlike traditional GRPO which only grades final answers, Agon implicitly judges the reasoning process during training without process labels or a reward model. The two models alternate roles as 'solver' and 'reviewer', aiming to outperform each other in a dynamic adversarial training setup. At inference, it deploys a cascade: one model drafts, the other reads and answers. On the hard split of DeepMath, Agon doubles the pass@1 of Qwen3, roughly eight times the gain of an untrained Mixture-of-Agents approach. The improvement replicates on competitive programming and across model families (Qwen3.5, Gemma 4), demonstrating the potential of adversarial co-training for reasoning reinforcement.

## Key points

- Agon makes two competing models grade each other during training, implicitly evaluating reasoning without process labels or a reward model.
- Training uses alternating roles: one model acts as solver, the other as reviewer, each rewarded for outperforming the other.
- At inference, a cascade is used: one model drafts a solution, the other reads it and produces the final answer.
- On the hard split of DeepMath, Agon doubles Qwen3's pass@1, significantly outperforming GRPO and untrained Mixture-of-Agents.

## Full text

Reinforcement learning from verifiable rewards (e.g. GRPO) is the engine behind today's reasoning models, yet it grades only the final answer. On hard problems this trains models to write more rather than to think better, since the trace itself is never graded and no label for good thinking exists. We introduce Agon, which makes two competing models each other's graders. Both attempt the same problem; in alternating roles, one drafts a solution and the other reads it while solving, and each is rewarded for out-solving the other. To win, a model must out-reason a rival that has seen its work, so reasoning is judged implicitly during training, with no process labels and no reward model. Because both models are optimized, each faces a progressively stronger rival, which single-model RL cannot provide. The two need only be comparably strong and behaviorally different. At inference the pair deploys as it trains, a two-stage cascade in which one model drafts and the other answers after reading the draft. On the hard split of DeepMath with Qwen3, this doubles GRPO's pass@1, roughly eight times the gain of an untrained Mixture-of-Agents pass over the same base. The ordering replicates on competitive-programming code and across model families (Qwen3.5, Gemma 4). For now the models talk in text; the next step is to let them reason together in latent space.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_1e8c4b0ab67fe3218908c2fd