---
title: "正途正道：结合可验证奖励与人类示范的语言模型训练"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.01181v1"
permalink: "https://feed.astralor.com/items/itm_a251971f604fd2e793053b96"
published: "2026-07-01T17:13:35.000Z"
score: 59
channel: "ai"
tags: ["人类示范","可验证奖励强化学习","对抗训练","生成器-判别器框架","语言模型"]
---

# 正途正道：结合可验证奖励与人类示范的语言模型训练

## 速览

本文提出一种对抗式生成器-判别器框架，在可验证奖励强化学习（RLVR）中融入人类示范信号。生成器通过RL同时优化任务准确性和来自判别器的对抗奖励，判别器则学习区分人类与模型输出。该方法在代码修复、故事生成等任务中显著提升非可验证属性（如风格、多样性），同时保持RLVR的准确率，并几乎消除奖励破解行为。

## 关键点

- 提出对抗式框架，将可验证奖励与人类示范结合，弥补RLVR忽视主观属性的缺陷。
- 生成器优化任务准确性与对抗奖励；判别器作为人类分布代理，提供非形式化反馈。
- 在代码修复中降低编辑距离，在故事生成中提高胜率与多样性，在奖励破解基准中几乎消除异常行为。

## 全文

RL with verifiable rewards (RLVR) has emerged as a powerful paradigm for training LMs on tasks with well-defined success metrics, such as code generation and mathematical reasoning. However, current RLVR methods optimize only what can be objectively scored, often neglecting subjective, non-verifiable aspects of human-like outputs, such as style and structure. This limitation leads to well-documented failure modes such as diversity collapse, unnatural-sounding responses, and reward hacking. We propose an adversarial generator-discriminator framework that augments verifiable rewards with a learned signal from human demonstrations. A generator model is trained using RL to maximize both task accuracy and an adversarial reward derived from a discriminator. The discriminator, trained alongside the generator policy, learns to distinguish human-written outputs from model-generated ones. The discriminator serves as a learned proxy for the human output distribution, providing feedback on aspects of generation that are difficult to formalize as scalar rewards. Across diverse domains, including bug fixing and open-ended generation, our approach consistently improves non-verifiable properties while preserving the accuracy gains of RLVR. In bug fixing, our method produces solutions with significantly lower edit distance compared to RLVR baselines while matching end performance. In story generation, our method significantly improves win rate while producing stories that are diverse and more human-like. And in a simple reward hacking benchmark, our method nearly eliminates model misbehavior while maintaining high benchmark scores. Together, these results show that our approach bridges RL and SFT, offering a scalable path toward jointly optimizing the verifiable and non-verifiable properties of a task.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_a251971f604fd2e793053b96