---
title: "Reward Budgeting Reduces Premature Convergence in Reinforcement Learning for LLM Reasoning"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2611.05942v1"
permalink: "https://feed.astralor.com/items/itm_141ec0368c451ad7a3f1c41c"
published: "2026-11-04T04:32:08.000Z"
score: 59
channel: "ai"
tags: ["LLM","推理效率","Reward Budgeting Reduces","Premature Convergence","Reinforcement Learning","LLM Reasoning","Recent"]
---

# Reward Budgeting Reduces Premature Convergence in Reinforcement Learning for LLM Reasoning

## At a glance

RL for LLM reasoning often reaches an early performance plateau as the policy distribution becomes increasingly sharp. Recent work has studied policy entropy as both a diagnostic and a control signal for this behavior. …

## Key points

- RL for LLM reasoning often reaches an early performance plateau as the policy distribution becomes increasingly sharp.
- Recent work has studied policy entropy as both a diagnostic and a control signal for this behavior.
- In our experiments, however, increasing entropy alone does not reliably change the eventual plateau.

## Full text

RL for LLM reasoning often reaches an early performance plateau as the policy distribution becomes increasingly sharp. Recent work has studied policy entropy as both a diagnostic and a control signal for this behavior. In our experiments, however, increasing entropy alone does not reliably change the eventual plateau. We analyze RL through its implicit target policy and derive a sharpness factor $κ$, together with an effective counterpart $κ_{\mathrm{eff}}$, that describes how strongly reward or advantage gaps are translated into probability-ratio gaps. Motivated by this view, we introduce \emph{reward budgeting}, a plug-in advantage construction that assigns a fixed total reward budget to each prompt. For binary rewards and independent on-policy rollouts, its unclipped score-function estimator follows the gradient of an exact finite-$G$ concave objective with diminishing returns, while its advantage gap decreases as more rollouts become correct. For general rewards, a shifted log-mean interpretation holds asymptotically under the stated regularity conditions. These results motivate, rather than establish exact objective equivalence for, the clipped implementations. On Qwen3-8B and Llama3.2-3B-Instruct, reward budgeting consistently improves a range of RL baselines, with gains up to +12.50 on AMC23 and +10.00 on AIME24 and AIME25. Overall, our results support an update-dynamics view of premature convergence and show that strong gains can be obtained without explicitly preserving entropy.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_141ec0368c451ad7a3f1c41c