---
title: "代理探索与可复用引导：一种基于代理引导更新信号的模块化LLM后训练范式"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.11505v1"
permalink: "https://feed.astralor.com/items/itm_869526bdfacdc0b0512c7586"
published: "2026-07-13T12:56:21.000Z"
score: 81
channel: "ai"
tags: ["PUST","Qwen3","代理引导更新信号传输","后训练","弱到强泛化"]
---

# 代理探索与可复用引导：一种基于代理引导更新信号的模块化LLM后训练范式

## AI 点评

> 这篇论文提出的PUST框架让我眼前一亮——通过代理模型提取相对改进信号并转移，巧妙解决了后训练中探索成本高、信号无法复用的痛点。对于从事LLM训练和调优的研究者或工程师，这篇工作提供了一种新的模块化思路，尤其是弱到强泛化的能力有望加速大模型的迭代实验。

## 速览

本文提出PUST框架，将LLM后训练中的策略探索与分布对齐解耦。通过轻量级代理模型代替主模型进行高奖励行为探索，提取初始与优化状态间的相对改进信号并转移给主模型。该范式显著降低计算开销，支持信号的异步生成、缓存和跨模型复用，实现弱到强提升。在Qwen3系列模型上的数学和代码任务评估表明，从较弱代理提取的信号能稳健增强更强模型。

## 关键点

- 提出代理引导更新信号传输（PUST）框架，解耦策略探索与分布对齐。
- 使用轻量级代理模型进行高效探索，提取相对改进信号而非绝对策略分布。
- 支持异步生成、缓存和跨模型转移优化信号，降低计算成本。
- 在Qwen3模型上验证，较弱代理的信号能稳健增强较强模型，实现弱到强提升。

## 全文

Post-training is essential for refining the domain-specific capabilities of large language models (LLMs), yet existing reward optimization and distribution matching methods tightly couple policy exploration with distribution alignment. This coupling forces expensive exploration directly on the policy model and severely hinders the asynchronous generation, reuse, and cross-model transfer of optimization signals. In this paper, we propose Proxy-guided Update Signal Transfer (PUST), a novel post-training framework that fundamentally decouples update-signal exploration from distribution alignment. Instead of utilizing the primary model for costly exploration, PUST employs a lightweight proxy model as an efficient testbed to discover high-reward behaviors. We extract the relative improvement signal between the proxy's initial and optimized states, transferring this directional update to the primary model to guide its policy alignment. This decoupled pipeline, comprising proxy exploration, update-signal extraction, and signal transfer, significantly reduces computational overhead and enables optimization signals to be asynchronously generated, cached, and reused. Crucially, by transferring relative improvements rather than absolute policy distributions, PUST naturally supports weak-to-strong improvement and seamless cross-model transfer. Systematic evaluations on Qwen3-family models across math and code domains demonstrate that update signals extracted from substantially weaker proxies can robustly and adjustably enhance stronger primary models. Ultimately, PUST transforms post-training from a monolithic online optimization process into a highly modular, reusable, and cost-efficient paradigm.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_869526bdfacdc0b0512c7586