---
title: "Density-Aware Reward Scaling: A Reinforcement Learning Framework for Reward-Proportional Sampling"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2611.04792v1"
permalink: "https://feed.astralor.com/items/itm_16776115b663064c4a1aadae"
published: "2026-11-03T12:29:58.000Z"
score: 58
channel: "tech"
tags: ["安全","RAG","多模态","Density-Aware Reward Scaling","Reinforcement Learning Framework","Reward-Proportional Sampling Reinforcement","Reinforcement","DARS"]
---

# Density-Aware Reward Scaling: A Reinforcement Learning Framework for Reward-Proportional Sampling

## 速览

Reinforcement learning objectives that maximize expected return can concentrate probability mass on a small subset of high-reward outcomes, limiting diversity when multiple distinct solutions are desirable. We introduce…

## 关键点

- Reinforcement learning objectives that maximize expected return can concentrate probability mass on a small subset of high-reward outcomes, limiting diversity when multiple distinct solutions are desirable.
- We introduce Density-Aware Reward Scaling (DARS), a simple reward-processing mechanism that promotes reward-proportional sampling in continuous outcome spaces while retaining standard policy-gradient training.
- DARS estimates the local density of outcomes within each on-policy rollout group and inversely scales rewards by this density.

## 全文

Reinforcement learning objectives that maximize expected return can concentrate probability mass on a small subset of high-reward outcomes, limiting diversity when multiple distinct solutions are desirable. We introduce Density-Aware Reward Scaling (DARS), a simple reward-processing mechanism that promotes reward-proportional sampling in continuous outcome spaces while retaining standard policy-gradient training. DARS estimates the local density of outcomes within each on-policy rollout group and inversely scales rewards by this density. The method requires no learned density model, auxiliary objective, critic, or modification to the underlying policy-gradient objective. In a controlled multimodal navigation task, DARS-GRPO avoids the single-mode collapse exhibited by GRPO and closely recovers the target mode allocation. Applied to text-to-image prompt adaptation, DARS-GRPO consistently improves image and prompt diversity across datasets, Policy LMs, and reward settings while maintaining good semantic alignment. These results show that density-aware reward scaling provides a lightweight way to improve coverage of diverse high-reward outcomes within existing reinforcement-learning pipelines.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_16776115b663064c4a1aadae