---
title: "Bayes-Adaptive MDPs for Rollout Allocation to Enhance GRPO Training"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2610.37429v1"
permalink: "https://feed.astralor.com/items/itm_9bd359a9f259cdf15c57404a"
published: "2026-10-27T15:41:57.000Z"
score: 53
channel: "ai"
tags: ["LLM","RAG","Bayes-Adaptive MDPs","Rollout Allocation","Enhance GRPO Training","Group Relative Policy","Optimization","GRPO"]
---

# Bayes-Adaptive MDPs for Rollout Allocation to Enhance GRPO Training

## 速览

Group Relative Policy Optimization (GRPO) has demonstrated strong performance in LLM post-training. However, it allocates a uniform rollout budget to every training question, ignoring variations in question difficulty. …

## 关键点

- Group Relative Policy Optimization (GRPO) has demonstrated strong performance in LLM post-training.
- However, it allocates a uniform rollout budget to every training question, ignoring variations in question difficulty.
- As a result, highly difficult questions may receive too few rollouts to generate informative positive reward signals, limiting their contribution to training, while moderately difficult questions often receive more roll…

## 全文

Group Relative Policy Optimization (GRPO) has demonstrated strong performance in LLM post-training. However, it allocates a uniform rollout budget to every training question, ignoring variations in question difficulty. As a result, highly difficult questions may receive too few rollouts to generate informative positive reward signals, limiting their contribution to training, while moderately difficult questions often receive more rollouts than necessary. In this work, we formulate the problem of difficulty-aware rollout budget allocation to maximize training utility under the same total rollout budget as standard GRPO. We propose AdaRoll-GRPO, which reformulates rollout allocation as a sequential decision-making problem and leverages a Bayes-adaptive Markov decision process (MDP) to learn an adaptive allocation policy. The learned policy is computed offline and applied during GRPO training without introducing additional online optimization overhead. Extensive experiments on multiple reasoning datasets and LLM architectures demonstrate that AdaRoll-GRPO consistently improves GRPO training performance under a fixed rollout budget, highlighting the effectiveness of adaptive rollout allocation for GRPO in LLM post-training.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_9bd359a9f259cdf15c57404a