---
title: "PBPO: Prefix-Block Policy Optimization for Diffusion Language Models"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2611.07783v1"
permalink: "https://feed.astralor.com/items/itm_600ea4f7b66ed1ae1b12cd35"
published: "2026-11-05T10:20:41.000Z"
score: 59
channel: "ai"
tags: ["LLM","PBPO","Prefix-Block Policy Optimization","Diffusion Language Models","Diffusion Large Language","Models"]
---

# PBPO: Prefix-Block Policy Optimization for Diffusion Language Models

## 速览

Diffusion Large Language Models (dLLMs) commonly decode responses by proceeding autoregressively across blocks while iteratively denoising multiple tokens within each block. This structure makes block-level policy optim…

## 关键点

- Diffusion Large Language Models (dLLMs) commonly decode responses by proceeding autoregressively across blocks while iteratively denoising multiple tokens within each block.
- This structure makes block-level policy optimization a natural adaptation of autoregressive policy optimization.
- However, the exact likelihood of a completed block is intractable because it marginalizes over the intermediate states and reveal orders of the denoising process.

## 全文

Diffusion Large Language Models (dLLMs) commonly decode responses by proceeding autoregressively across blocks while iteratively denoising multiple tokens within each block. This structure makes block-level policy optimization a natural adaptation of autoregressive policy optimization. However, the exact likelihood of a completed block is intractable because it marginalizes over the intermediate states and reveal orders of the denoising process. To realize the block-level formulation, we construct a prefix-block state and use a single model evaluation to define a factorized one-step surrogate for the block likelihood. The resulting block surrogate ratio is a product of its token surrogate ratios, allowing variation at individual positions to propagate across the entire block before clipping. We therefore introduce Prefix-Block Policy Optimization (PBPO), which preserves block-level conditioning but applies surrogate importance weighting and clipping separately to each token. PBPO further diversifies repeated updates by resampling corrupted views of the prompt and revealed prefix, a design motivated by masked-dLLM pre-training. Under our analysis assumptions, block-product weighting amplifies the variance of gradient contributions relative to token-local weighting. Across two LLaDA-family initializations, six benchmarks, and three generation lengths, PBPO achieves competitive performance against existing dLLM RL methods.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_600ea4f7b66ed1ae1b12cd35