---
title: "掩码感知策略梯度用于扩散语言模型"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.15200v1"
permalink: "https://feed.astralor.com/items/itm_06291c4b697f29c22c46b224"
published: "2026-07-16T16:57:34.000Z"
score: 83
channel: "ai"
tags: ["GSM8K","MBPP","强化学习","掩码扩散语言模型","策略梯度"]
---

# 掩码感知策略梯度用于扩散语言模型

## AI 点评

> 这篇论文的价值在于它精准点出了扩散语言模型在RL训练中被长期忽视的‘掩码顺序’问题，并给出了优雅的理论解。对于正在探索扩散模型后训练或推理增强的团队，这篇文章比单纯调优采样的工作更具根本意义。它证明了‘选什么’和‘什么时候选’必须一起优化，这很可能也会启发其他扩散生成任务。

## 速览

强化学习已有效提升了大语言模型的推理能力，但将其扩展到掩码扩散语言模型（MDLM）时，对数似然估计的难解性带来了挑战。现有方法仅建模 token 预测，忽略了生成过程中位置解掩码的顺序。本文观察到 MDLM 的生成步骤涉及两个决策：在每个掩码位置放置什么 token 以及哪些位置需要重新掩码。作者将其形式化为两阶段动作 MDP，证明策略梯度自然地分解为 token 项和 masking 项。联合优化两项后，在 GSM8K 和 MBPP 上分别达到 87.1% 和 53.4% 的最新水平。

## 关键点

- 现有方法忽略了解掩码顺序，仅按 token 预测近似对数似然。
- 将 MDLM 生成形式化为两阶段动作 MDP，证明策略梯度分解为 token 项和 masking 项。
- 联合优化两者在数学推理 GSM8K 和代码生成 MBPP 上达到 SOTA。

## 全文

Reinforcement learning has proven effective for improving reasoning in large language models, but extending it to Masked Diffusion Language Models (MDLMs) remains challenging due to the intractability of the log-likelihood estimation. Existing approaches approximate this log-likelihood by modeling only the token predictions, ignoring the order in which positions are unmasked during generation. We observe that MDLM generation involves two decisions at each step: what tokens to place at each masked position and which positions to remask. We formalize this as a two-stage action MDP, showing that the policy gradient naturally decomposes into a token term and a masking term. Combining optimization of both terms leads to state-of-the-art outcomes on mathematical reasoning and coding benchmarks, with scores of 87.1% on GSM8K and 53.4% on MBPP.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_06291c4b697f29c22c46b224