---
title: "RRC: Unlocking Generative Reward Models in LLM Reinforcement Learning via Ranking-Based Reward Construction"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.06310v1"
permalink: "https://feed.astralor.com/items/itm_c8e3c5071127ec3eb1684443"
published: "2026-08-06T17:24:36.000Z"
score: 74
channel: "ai"
tags: ["Ranking-based Reward Construction","Large Language Models","Reinforcement Learning","Generative Reward Models","Self-competitive Ranking","Anchor-guided Ranking"]
---

# RRC: Unlocking Generative Reward Models in LLM Reinforcement Learning via Ranking-Based Reward Construction

## AI commentary

> We think this paper finally pinpoints why generative reward models underperform in RL: the scalar scoring paradigm is inherently incompatible with comparative outputs. RRC's ranking-based reward construction is simple yet effective, and the anchor-guided strategy is especially friendly for teams with limited compute. We recommend RLHF/DPO practitioners and reward model researchers check the open-source code directly—it's more valuable than reading the abstract.

## At a glance

Recent advances in reward modeling show a paradigm shift from discriminative to generative reward models. However, generative reward models underperform in reinforcement learning (RL) despite strong ranking capabilities. This paper identifies a mismatch between the comparative nature of generative reward modeling and the scalar scoring paradigm of existing RL algorithms. To bridge this gap, they propose Ranking-based Reward Construction (RRC), which derives rewards from relative preference rankings via two complementary strategies: self-competitive ranking and anchor-guided ranking. Experiments on open-ended chat and reasoning benchmarks show consistent gains over existing methods. Code is open-sourced.

## Key points

- Identifies the mismatch between generative reward modeling and RL's scalar scoring paradigm, explaining poor RL performance.
- Proposes RRC with two complementary strategies: self-competitive ranking and anchor-guided ranking.
- Demonstrates consistent gains over existing reward construction methods on chat and reasoning benchmarks.
- Releases open-source code for reproducibility.

## Full text

Recent advances in reward modeling show a paradigm shift from discriminative reward models to generative reward models. However, despite their strong capabilities in response ranking, generative reward models have not realized their potential in reinforcement learning (RL). Our analysis reveals that this limitation arises from a mismatch between the comparative nature of generative reward modeling and the scalar scoring paradigm adopted by existing RL algorithms. To bridge this gap, we propose a Ranking-based Reward Construction (RRC) approach, which enables generative reward models to provide more effective RL learning signals by deriving rewards from relative preference rankings. RRC introduces two complementary strategies: self-competitive ranking, which exploits comparisons among sampled responses, and anchor-guided ranking, which enables scalable ranking-based reward construction with a small set of reference responses. Experiments across open-ended chat and reasoning benchmarks demonstrate that RRC substantially improves RL training with generative reward models, achieving consistent gains over existing reward construction approaches. Our code can be found at https://github.com/wangclnlp/RRC.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_c8e3c5071127ec3eb1684443