---
title: "RRC：基于排序奖励构建释放生成式奖励模型在LLM强化学习中的潜力"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.06310v1"
permalink: "https://feed.astralor.com/items/itm_c8e3c5071127ec3eb1684443"
published: "2026-08-06T17:24:36.000Z"
score: 74
channel: "ai"
tags: ["RRC","大语言模型","强化学习","生成式奖励模型","自竞争排序","锚点引导排序"]
---

# RRC：基于排序奖励构建释放生成式奖励模型在LLM强化学习中的潜力

## AI 点评

> 我们认为，生成式奖励模型在RL中难用的症结终于被点破——标量评分范式本质上与比较式输出不兼容。RRC用排序信号做奖励，思路简单但有效，尤其是锚点引导排序对算力受限的团队格外友好。推荐给正在做RLHF、DPO或奖励模型训练的工程师和研究者，直接去看开源代码，比读摘要更有收获。

## 速览

近期奖励模型研究从判别式转向生成式，但生成式奖励模型在强化学习（RL）中表现不佳。本文指出，根因在于生成式奖励的比较特性与现有RL标量评分范式不匹配。为此提出基于排序的奖励构建（RRC）方法，通过自竞争排序和锚点引导排序从相对偏好中提取奖励信号，在开放域聊天和推理基准上显著提升RL训练效果，代码已开源。

## 关键点

- 揭示生成式奖励模型与强化学习标量评分范式之间的不匹配，解释其在RL中表现不佳的原因。
- 提出RRC方法，包含自竞争排序与锚点引导排序两种互补策略，从相对偏好中构建奖励。
- 在开放域聊天和推理基准上验证RRC显著优于现有奖励构建方法。
- 开源代码，便于复现和进一步研究。

## 全文

Recent advances in reward modeling show a paradigm shift from discriminative reward models to generative reward models. However, despite their strong capabilities in response ranking, generative reward models have not realized their potential in reinforcement learning (RL). Our analysis reveals that this limitation arises from a mismatch between the comparative nature of generative reward modeling and the scalar scoring paradigm adopted by existing RL algorithms. To bridge this gap, we propose a Ranking-based Reward Construction (RRC) approach, which enables generative reward models to provide more effective RL learning signals by deriving rewards from relative preference rankings. RRC introduces two complementary strategies: self-competitive ranking, which exploits comparisons among sampled responses, and anchor-guided ranking, which enables scalable ranking-based reward construction with a small set of reference responses. Experiments across open-ended chat and reasoning benchmarks demonstrate that RRC substantially improves RL training with generative reward models, achieving consistent gains over existing reward construction approaches. Our code can be found at https://github.com/wangclnlp/RRC.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_c8e3c5071127ec3eb1684443