---
title: "MoE Proxy Models for Low-Cost Failure Reproduction and Diagnosis in LLM RL Post-Training"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.10823v1"
permalink: "https://feed.astralor.com/items/itm_d1d2d502256d5276dc475152"
published: "2026-08-11T11:50:48.000Z"
score: 70
channel: "tech"
tags: ["MoE","Surrogate Model","Huawei Ascend","RL Post-training","Failure Diagnosis"]
---

# MoE Proxy Models for Low-Cost Failure Reproduction and Diagnosis in LLM RL Post-Training

## AI commentary

> If you are an engineer working on LLM alignment or RL post-training, this is worth reading—it cuts the compute cost of failure reproduction by an order of magnitude, giving you a cheap surrogate for debugging. The paper is based on Ascend, but the method applies to GPU clusters too.

## At a glance

This paper from arXiv addresses the high cost of failure reproduction in RL post-training of LLMs by proposing a MoE-based proxy model construction method. By preserving the backbone architecture, routing mechanism, and clustering-based expert pruning, it reduces accelerator requirements by 50%-87.5% and per-step NPU-hour cost by up to 33.3x, while maintaining training dynamics and reproducing failure responses. Based on large-scale RL training on Huawei Ascend, the study systematically analyzes failure types and model-side factors, offering a low-cost solution for failure investigation and auxiliary diagnosis.

## Key points

- Systematically summarizes typical failure types in LLM RL post-training on Ascend and identifies three model-side factors.
- Proposes a structure-preserving, clustering-based expert pruning method for proxy models, reducing accelerator requirements by 50%-87.5%.
- Proxy models cut per-step NPU-hour cost by up to 33.3x while preserving major training dynamics and reproducing failure behaviors.

## Full text

Reinforcement learning (RL) post-training of large language models (LLMs) is computationally intensive and involves complex system pipelines with substantial debugging overhead. In practice, factors such as framework adaptation, numerical precision, and operator implementation can cause failures, including gradient overflow and loss divergence. Reproducing such failures directly on large models requires considerable time and computational resources. This paper systematically analyzes failures encountered during large-scale RL training on the Huawei Ascend platform, summarizes representative failure types, and identifies three model-side factors relevant to fault reproduction. Based on these factors, we propose a proxy-model construction method for low-cost fault investigation and auxiliary diagnosis. It employs structure-preserving, clustering-based expert pruning to select representative experts while retaining the model's backbone architecture, routing mechanism, and basic task capabilities. Our experimental results show that the proxy models reduce accelerator requirements by 50%-87.5% and achieve up to a 33.3x reduction in per-step NPU-hour cost, while preserving major training dynamics and reproducing fault responses consistent with the original models. Overall, the proxy models can serve as low-cost surrogates for fault reproduction, targeted validation, and auxiliary diagnosis in RL post-training.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_d1d2d502256d5276dc475152