---
title: "Aligning Training and Rollout Policies for End-to-End FP4 RL on Mixture-of-Experts LLMs"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2611.04376v1"
permalink: "https://feed.astralor.com/items/itm_025722fecac12fc3baa04803"
published: "2026-11-03T06:03:38.000Z"
score: 58
channel: "tech"
tags: ["LLM","推理效率","安全","Aligning Training","Rollout Policies","End-to-End FP4","Mixture-of-Experts LLMs","FP4"]
---

# Aligning Training and Rollout Policies for End-to-End FP4 RL on Mixture-of-Experts LLMs

## At a glance

In this work, we study end-to-end FP4 reinforcement learning (RL) post-training for Mixture-of-Experts (MoE) large language models. Recent work on FP4 RL has introduced effective recovery recipes that restore accuracy u…

## Key points

- In this work, we study end-to-end FP4 reinforcement learning (RL) post-training for Mixture-of-Experts (MoE) large language models.
- Recent work on FP4 RL has introduced effective recovery recipes that restore accuracy under aggressive low-precision quantization for dense models, but we find that the same approach yields no improvement over the naive…
- We conduct an in-depth study to understand why.

## Full text

In this work, we study end-to-end FP4 reinforcement learning (RL) post-training for Mixture-of-Experts (MoE) large language models. Recent work on FP4 RL has introduced effective recovery recipes that restore accuracy under aggressive low-precision quantization for dense models, but we find that the same approach yields no improvement over the naive FP4 baseline in the MoE setting. We conduct an in-depth study to understand why. Our analysis identifies the root cause in the routing mechanism unique to MoE: because the router makes discrete expert-selection decisions, training-inference policy alignment becomes acutely sensitive to whether the same recovery technique is applied symmetrically on the training and inference sides. When an asymmetric recipe is used-including the dense-model approach, which corrects only the rollout-training-inference alignment deteriorates sharply, as measured by the Pearson correlation coefficient between training and inference probabilities, and router selection matching rates. Building on this finding, we propose MoeResQ, a quantization-recovery recipe for MoE RL post-training that enforces symmetry between the training- and inference-side corrections. MoeResQ restores training-inference alignment and substantially closes the accuracy gap to full precision, establishing symmetric recovery as the key principle in FP4 RL for MoE models.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_025722fecac12fc3baa04803