---
title: "Proxy Exploration and Reusable Guidance: A Modular LLM Post-Training Paradigm via Proxy-Guided Update Signals"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.11505v1"
permalink: "https://feed.astralor.com/items/itm_869526bdfacdc0b0512c7586"
published: "2026-07-13T12:56:21.000Z"
score: 81
channel: "ai"
tags: ["PUST","Qwen3","Proxy-guided Update Signal Transfer","Post-training","Weak-to-strong generalization"]
---

# Proxy Exploration and Reusable Guidance: A Modular LLM Post-Training Paradigm via Proxy-Guided Update Signals

## AI commentary

> This paper's PUST framework caught my eye—by extracting relative improvement signals from a proxy model and transferring them, it cleverly addresses the high exploration cost and non-reusable signals in post-training. For researchers and engineers working on LLM training and finetuning, this work offers a new modular approach, and the weak-to-strong capability is promising for accelerating iterative experiments.

## At a glance

This paper proposes PUST, a framework that decouples policy exploration from distribution alignment in LLM post-training. It uses a lightweight proxy model to discover high-reward behaviors, extracts relative improvement signals between the proxy's initial and optimized states, and transfers them to the primary model. This paradigm significantly reduces computational overhead, supports asynchronous generation, caching, and cross-model reuse of signals, enabling weak-to-strong improvement. Evaluations on Qwen3-family models in math and code domains show that signals from weaker proxies robustly enhance stronger models.

## Key points

- Proposes Proxy-guided Update Signal Transfer (PUST) to decouple policy exploration from distribution alignment.
- Uses a lightweight proxy model for efficient exploration, extracting relative improvement signals instead of absolute policy distributions.
- Enables asynchronous generation, caching, and cross-model transfer of optimization signals, reducing compute overhead.
- Validated on Qwen3 models, showing signals from weaker proxies robustly enhance stronger models (weak-to-strong improvement).

## Full text

Post-training is essential for refining the domain-specific capabilities of large language models (LLMs), yet existing reward optimization and distribution matching methods tightly couple policy exploration with distribution alignment. This coupling forces expensive exploration directly on the policy model and severely hinders the asynchronous generation, reuse, and cross-model transfer of optimization signals. In this paper, we propose Proxy-guided Update Signal Transfer (PUST), a novel post-training framework that fundamentally decouples update-signal exploration from distribution alignment. Instead of utilizing the primary model for costly exploration, PUST employs a lightweight proxy model as an efficient testbed to discover high-reward behaviors. We extract the relative improvement signal between the proxy's initial and optimized states, transferring this directional update to the primary model to guide its policy alignment. This decoupled pipeline, comprising proxy exploration, update-signal extraction, and signal transfer, significantly reduces computational overhead and enables optimization signals to be asynchronously generated, cached, and reused. Crucially, by transferring relative improvements rather than absolute policy distributions, PUST naturally supports weak-to-strong improvement and seamless cross-model transfer. Systematic evaluations on Qwen3-family models across math and code domains demonstrate that update signals extracted from substantially weaker proxies can robustly and adjustably enhance stronger primary models. Ultimately, PUST transforms post-training from a monolithic online optimization process into a highly modular, reusable, and cost-efficient paradigm.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_869526bdfacdc0b0512c7586