---
title: "Right in the Right Way: LM Training with Verifiable Rewards and Human Demonstrations"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.01181v1"
permalink: "https://feed.astralor.com/items/itm_a251971f604fd2e793053b96"
published: "2026-07-01T17:13:35.000Z"
score: 59
channel: "ai"
tags: ["human demonstration","RLVR","Adversarial Training","generator-discriminator framework","Language Model"]
---

# Right in the Right Way: LM Training with Verifiable Rewards and Human Demonstrations

## At a glance

This paper proposes an adversarial generator-discriminator framework that augments verifiable rewards with a learned signal from human demonstrations. The generator is trained via RL to maximize both task accuracy and an adversarial reward from the discriminator, which learns to distinguish human-written outputs from model-generated ones. Across bug fixing, story generation, and a reward hacking benchmark, the method consistently improves non-verifiable properties while preserving RLVR accuracy gains and nearly eliminating reward hacking.

## Key points

- Proposes an adversarial framework combining verifiable rewards with human demonstrations to address RLVR's neglect of subjective attributes.
- Generator optimizes task accuracy and adversarial reward; discriminator serves as proxy for human distribution, providing unformalized feedback.
- Reduces edit distance in bug fixing, improves win rate and diversity in story generation, and nearly eliminates reward hacking.

## Full text

RL with verifiable rewards (RLVR) has emerged as a powerful paradigm for training LMs on tasks with well-defined success metrics, such as code generation and mathematical reasoning. However, current RLVR methods optimize only what can be objectively scored, often neglecting subjective, non-verifiable aspects of human-like outputs, such as style and structure. This limitation leads to well-documented failure modes such as diversity collapse, unnatural-sounding responses, and reward hacking. We propose an adversarial generator-discriminator framework that augments verifiable rewards with a learned signal from human demonstrations. A generator model is trained using RL to maximize both task accuracy and an adversarial reward derived from a discriminator. The discriminator, trained alongside the generator policy, learns to distinguish human-written outputs from model-generated ones. The discriminator serves as a learned proxy for the human output distribution, providing feedback on aspects of generation that are difficult to formalize as scalar rewards. Across diverse domains, including bug fixing and open-ended generation, our approach consistently improves non-verifiable properties while preserving the accuracy gains of RLVR. In bug fixing, our method produces solutions with significantly lower edit distance compared to RLVR baselines while matching end performance. In story generation, our method significantly improves win rate while producing stories that are diverse and more human-like. And in a simple reward hacking benchmark, our method nearly eliminates model misbehavior while maintaining high benchmark scores. Together, these results show that our approach bridges RL and SFT, offering a scalable path toward jointly optimizing the verifiable and non-verifiable properties of a task.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_a251971f604fd2e793053b96