---
title: "MoE 代理模型：低成本复现 LLM 强化学习后训练故障的新方法"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.10823v1"
permalink: "https://feed.astralor.com/items/itm_d1d2d502256d5276dc475152"
published: "2026-08-11T11:50:48.000Z"
score: 70
channel: "tech"
tags: ["MoE","代理模型","华为昇腾","强化学习后训练","故障诊断"]
---

# MoE 代理模型：低成本复现 LLM 强化学习后训练故障的新方法

## AI 点评

> 如果你是做 LLM 对齐或 RL 后训练的工程师，这篇值得看——它把故障复现的算力成本砍了一个量级，等于给你发了个便宜的调试替身。论文基于昇腾平台，但方法对云上 GPU 集群同样有借鉴意义。

## 速览

本论文来自 arXiv，针对 LLM 强化学习后训练中故障复现成本高的问题，提出基于 MoE 的代理模型构建方法。该方法通过保持主干结构、路由机制和聚类专家剪枝，在降低 50%-87.5% 加速器需求和最高 33.3 倍每步 NPU 时成本的同时，保留训练动态并复现故障。研究基于华为昇腾平台的大规模 RL 训练实践，系统分析了故障类型和模型侧因素，为低成本故障复现与辅助诊断提供了可行方案。

## 关键点

- 系统总结了昇腾平台上 LLM RL 后训练的典型故障类型，并提炼出三个模型侧影响因素。
- 提出保持结构、基于聚类的专家剪枝代理模型构建法，加速器需求降低 50%-87.5%。
- 代理模型可将每步 NPU 时成本降低最高 33.3 倍，同时保留主要训练动态并复现故障行为。

## 全文

Reinforcement learning (RL) post-training of large language models (LLMs) is computationally intensive and involves complex system pipelines with substantial debugging overhead. In practice, factors such as framework adaptation, numerical precision, and operator implementation can cause failures, including gradient overflow and loss divergence. Reproducing such failures directly on large models requires considerable time and computational resources. This paper systematically analyzes failures encountered during large-scale RL training on the Huawei Ascend platform, summarizes representative failure types, and identifies three model-side factors relevant to fault reproduction. Based on these factors, we propose a proxy-model construction method for low-cost fault investigation and auxiliary diagnosis. It employs structure-preserving, clustering-based expert pruning to select representative experts while retaining the model's backbone architecture, routing mechanism, and basic task capabilities. Our experimental results show that the proxy models reduce accelerator requirements by 50%-87.5% and achieve up to a 33.3x reduction in per-step NPU-hour cost, while preserving major training dynamics and reproducing fault responses consistent with the original models. Overall, the proxy models can serve as low-cost surrogates for fault reproduction, targeted validation, and auxiliary diagnosis in RL post-training.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_d1d2d502256d5276dc475152