---
title: "面向高效采样的扩散模型RLHF：选择性时间步加权与基于优势的回放"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.07693v1"
permalink: "https://feed.astralor.com/items/itm_545e2302a57de1284757ff4f"
published: "2026-07-08T17:49:49.000Z"
score: 81
channel: "ai"
tags: ["PPO","RLHF","对齐","扩散模型","采样效率"]
---

# 面向高效采样的扩散模型RLHF：选择性时间步加权与基于优势的回放

## AI 点评

> 这篇论文直击扩散模型RLHF的痛点——反馈效率，提出的STW和ABR方法简单且有效，6倍的采样效率提升非常惊艳。对于任何从事扩散模型对齐或RLHF研究的开发者，这是必读之作，尤其是那些受限于高昂人类反馈成本的团队。

## 速览

扩散模型RLHF面临反馈效率低下的问题，现有方法需要大量人工或奖励模型评估。本文提出两种互补策略：每时间步加权方案，利用奖励信息在去噪轨迹中的不均匀分布，重新加权策略优化中的去噪步骤；以及基于优势的回放机制，优先使用信息丰富的历史轨迹，减少对新奖励查询的依赖。在相同超参数下，该方法相比基线实现最高6倍的采样效率提升，且能泛化到未见提示。

## 关键点

- 观察到扩散轨迹中奖励信息分布不均，并非所有去噪步或轨迹对学习贡献相同。
- 提出每时间步加权方法，从理论上联系到PPO的最优收敛性质，并近似加权趋势。
- 引入基于优势的回放机制，优先重用信息丰富的轨迹，减少奖励查询次数。
- 在相同超参数下，采样效率提升高达6倍。

## 全文

Reinforcement learning from human feedback (RLHF) has emerged as a powerful paradigm for aligning generative models with human preferences. However, applying RLHF to diffusion models remains highly feedback inefficient, as existing approaches typically require large amounts of human or reward model evaluations. This limitation reduces the practicality of diffusion RLHF in realworld settings where feedback is the primary bottleneck. In this paper, we propose two complementary strategies that substantially improve the feedback efficiency of diffusion RLHF while preserving generalization to unseen prompts. Our key observation is that reward information in diffusion trajectories is unevenly distributed: not all denoising timesteps or trajectories contribute equally to learning from a reward signal. By emphasizing informative timesteps and trajectories during optimization, we obtain more effective gradient updates. First, we introduce a per-timestep weighting scheme that reweights denoising steps during policy optimization. We theoretically connect this weighting to the optimal convergence properties of proximal policy optimization (PPO) and approximate the resulting weighting trend empirically. Second, we introduce a replay mechanism that prioritizes informative trajectories, enabling the model to reuse past samples instead of repeatedly querying new rewards. Together, these strategies significantly improve the feedback efficiency of diffusion RLHF. Under identical hyperparameter settings, our approach achieves up to a 6$\times$ improvement in sample efficiency compared to widely used diffusion RLHF baselines.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_545e2302a57de1284757ff4f