Right in the Right Way: LM Training with Verifiable Rewards and Human Demonstrations

Computer Science > Machine Learning

arXiv:2607.01181v1 (cs) [Submitted on 1 Jul 2026]

Title:Right in the Right Way: LM Training with Verifiable Rewards and Human Demonstrations

Authors:Mehul Damani, Isha Puri, Idan Shenfeld, Jacob Andreas View a PDF of the paper titled Right in the Right Way: LM Training with Verifiable Rewards and Human Demonstrations, by Mehul Damani and 3 other authors View PDF HTML (experimental)

Abstract:RL with verifiable rewards (RLVR) has emerged as a powerful paradigm for training LMs on tasks with well-defined success metrics, such as code generation and mathematical reasoning. However, current RLVR methods optimize only what can be objectively scored, often neglecting subjective, non-verifiable aspects of human-like outputs, such as style and structure. This limitation leads to well-documented failure modes such as diversity collapse, unnatural-sounding responses, and reward hacking. We propose an adversarial generator-discriminator framework that augments verifiable rewards with a learned signal from human demonstrations. A generator model is trained using RL to maximize both task accuracy and an adversarial reward derived from a discriminator. The discriminator, trained alongside the generator policy, learns to distinguish human-written outputs from model-generated ones. The discriminator serves as a learned proxy for the human output distribution, providing feedback on aspects of generation that are difficult to formalize as scalar rewards. Across diverse domains, including bug fixing and open-ended generation, our approach consistently improves non-verifiable properties while preserving the accuracy gains of RLVR. In bug fixing, our method produces solutions with significantly lower edit distance compared to RLVR baselines while matching end performance. In story generation, our method significantly improves win rate while producing stories that are diverse and more human-like. And in a simple reward hacking benchmark, our method nearly eliminates model misbehavior while maintaining high benchmark scores. Together, these results show that our approach bridges RL and SFT, offering a scalable path toward jointly optimizing the verifiable and non-verifiable properties of a task.

Subjects:	Machine Learning (cs.LG); Artificial Intelligence (cs.AI); Computation and Language (cs.CL)
Cite as:	arXiv:2607.01181 [cs.LG]
	(or arXiv:2607.01181v1 [cs.LG] for this version)
	https://doi.org/10.48550/arXiv.2607.01181 Focus to learn more arXiv-issued DOI via DataCite (pending registration)

Submission history

From: Mehul Damani [view email]
[v1] Wed, 1 Jul 2026 17:13:35 UTC (1,028 KB)
Full-text links:

Access Paper:

View a PDF of the paper titled Right in the Right Way: LM Training with Verifiable Rewards and Human Demonstrations, by Mehul Damani and 3 other authors
• View PDF
• HTML (experimental)
• TeX Source

view license

Current browse context:

cs.LG < prev   |   next >
new | recent | 2026-07 Change to browse by: cs
cs.AI
cs.CL

References & Citations

• NASA ADS
• Google Scholar
• Semantic Scholar

export BibTeX citation Loading...

BibTeX formatted citation

× Data provided by:

Bookmark

Bibliographic Tools

Bibliographic and Citation Tools

Bibliographic Explorer Toggle Bibliographic Explorer (What is the Explorer?) Connected Papers Toggle Connected Papers (What is Connected Papers?) Litmaps Toggle Litmaps (What is Litmaps?) scite.ai Toggle scite Smart Citations (What are Smart Citations?) Code, Data, Media

Code, Data and Media Associated with this Article

alphaXiv Toggle alphaXiv (What is alphaXiv?) Links to Code Toggle CatalyzeX Code Finder for Papers (What is CatalyzeX?) DagsHub Toggle DagsHub (What is DagsHub?) GotitPub Toggle Gotit.pub (What is GotitPub?) Huggingface Toggle Hugging Face (What is Huggingface?) ScienceCast Toggle ScienceCast (What is ScienceCast?) Demos

Demos

Replicate Toggle Replicate (What is Replicate?) Spaces Toggle Hugging Face Spaces (What is Spaces?) Spaces Toggle TXYZ.AI (What is TXYZ.AI?) Related Papers

Recommenders and Search Tools

Link to Influence Flower Influence Flower (What are Influence Flowers?) Core recommender toggle CORE Recommender (What is CORE?) IArxiv recommender toggle IArxiv Recommender (What is IArxiv?)

• Author
• Venue
• Institution
• Topic

About arXivLabs

arXivLabs: experimental projects with community collaborators

arXivLabs is a framework that allows collaborators to develop and share new arXiv features directly on our website.

Both individuals and organizations that work with arXivLabs have embraced and accepted our values of openness, community, excellence, and user data privacy. arXiv is committed to these values and only works with partners that adhere to them.

Have an idea for a project that will add value for arXiv's community? Learn more about arXivLabs.

Which authors of this paper are endorsers? | Disable MathJax (What is MathJax?)