---
title: "Selective Timestep Weighting and Advantage-Based Replay for Sample-Efficient Diffusion RLHF"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.07693v1"
permalink: "https://feed.astralor.com/items/itm_545e2302a57de1284757ff4f"
published: "2026-07-08T17:49:49.000Z"
score: 81
channel: "ai"
tags: ["PPO","RLHF","Alignment","Diffusion Models","Sample Efficiency"]
---

# Selective Timestep Weighting and Advantage-Based Replay for Sample-Efficient Diffusion RLHF

## AI commentary

> This paper hits the pain point of diffusion RLHF—feedback efficiency. The STW and ABR methods are simple yet effective, with a 6× sample efficiency gain that is impressive. A must-read for any developer working on diffusion model alignment or RLHF, especially teams constrained by high human feedback costs.

## At a glance

Diffusion RLHF suffers from low feedback efficiency, requiring extensive human or reward model evaluations. This paper proposes two complementary strategies: a per-timestep weighting scheme that leverages the uneven distribution of reward information across denoising steps to reweight policy optimization; and an advantage-based replay mechanism that prioritizes informative past trajectories, reducing reliance on new reward queries. Under identical hyperparameters, the approach achieves up to 6× sample efficiency improvement over baselines while generalizing to unseen prompts.

## Key points

- Observes that reward information is unevenly distributed across diffusion trajectories—not all timesteps or trajectories contribute equally.
- Proposes a per-timestep weighting scheme, theoretically connected to optimal convergence of PPO, with empirical approximation of weighting trends.
- Introduces an advantage-based replay mechanism that prioritizes informative trajectories, reducing the need for new reward queries.
- Achieves up to 6× improvement in sample efficiency under identical hyperparameters.

## Full text

Reinforcement learning from human feedback (RLHF) has emerged as a powerful paradigm for aligning generative models with human preferences. However, applying RLHF to diffusion models remains highly feedback inefficient, as existing approaches typically require large amounts of human or reward model evaluations. This limitation reduces the practicality of diffusion RLHF in realworld settings where feedback is the primary bottleneck. In this paper, we propose two complementary strategies that substantially improve the feedback efficiency of diffusion RLHF while preserving generalization to unseen prompts. Our key observation is that reward information in diffusion trajectories is unevenly distributed: not all denoising timesteps or trajectories contribute equally to learning from a reward signal. By emphasizing informative timesteps and trajectories during optimization, we obtain more effective gradient updates. First, we introduce a per-timestep weighting scheme that reweights denoising steps during policy optimization. We theoretically connect this weighting to the optimal convergence properties of proximal policy optimization (PPO) and approximate the resulting weighting trend empirically. Second, we introduce a replay mechanism that prioritizes informative trajectories, enabling the model to reuse past samples instead of repeatedly querying new rewards. Together, these strategies significantly improve the feedback efficiency of diffusion RLHF. Under identical hyperparameter settings, our approach achieves up to a 6$\times$ improvement in sample efficiency compared to widely used diffusion RLHF baselines.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_545e2302a57de1284757ff4f