---
title: "RL后训练构建组合推理策略：从原始技能到高阶策略的重组机制"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.07646v1"
permalink: "https://feed.astralor.com/items/itm_fb33127327caf094cfe4dcc9"
published: "2026-07-08T17:04:42.000Z"
score: 76
channel: "ai"
tags: ["arXiv","Transformer","强化学习后训练","拒绝微调","组合推理"]
---

# RL后训练构建组合推理策略：从原始技能到高阶策略的重组机制

## AI 点评

> 这篇论文是我近期看到对RL后训练机制解释最透明的工作之一。它用简单的符号环境剥离了其他干扰因素，直接展示了RL如何从碎片化的原始技能中“编织”出可复用的高阶策略。如果你在做推理模型的训练策略优化，尤其是对比RL和SFT/拒绝微调的效果差异，这篇文章提供了非常坚实的实验证据：关键不在于探索多少，而在于探索什么。预训练组织方式的重要性也被量化证明，这对设计预训练任务有直接启示。

## 速览

本研究通过可控重写文法环境，探究强化学习后训练能否将基础模型的原始技能组合成高阶推理策略。实验表明，RL不仅提升了原始能力，还通过相位组合机制（先强化原始归约，再发现有效组合过程，如顺序组合与并行组合）构建出可复用的策略库。与拒绝微调相比，RL的关键优势在于探索选择性而非数量：后者产生大量无效捷径，而RL集中探索有效可复用结构。预训练必须将原始能力组织为可压缩的归约过程，RL才能将其构建为可靠策略。

## 关键点

- RL后训练能将原始符号重写技能组合为顺序与并行两种高阶策略，且策略被稳定复用
- 拒绝微调早期有效但之后停滞，产生大量无效捷径，而RL通过选择性探索发现可复用结构
- 组合策略的涌现依赖预训练将原始能力组织为可压缩的归约过程，而非仅暴露于原始技能

## 全文

Does RL post-training merely amplify primitive skills already latent in a base model, or can it compose primitive skills into new higher-level strategies? We study this question in a fully observable rewrite-grammar environment where the pretraining distribution is known and every generated rewrite can be audited. A Transformer is pretrained on primitive symbol-rewrite chains and post-trained on a Trace-based reasoning task with only a binary final-answer reward. RL solves held-out problems that remain rarely solved by the pretrained model even under much larger sampling budgets, while rejection fine-tuning improves early but plateaus. Trace analysis shows that RL reorganizes primitive competence through a phased compositional mechanism: it first strengthens primitive reductions, then discovers valid composed procedures. These include sequential compositions, which collapse ordered chains of primitive contractions, and parallel compositions, which combine independent primitive contractions in a single step. The composed procedures are not isolated samples; they are reused and consolidated into a stable repertoire. Comparing RL with rejection fine-tuning shows that the key difference is not exploration volume but selectivity: RFT produces many shortcut-like rewrites, much of them invalid, whereas RL concentrates exploration into valid reusable structure. Pretraining ablations show that the emergence of compositional strategies is gated not by primitive exposure alone, but by whether pretraining organizes primitive competence into reduction procedures that RL can later compress. The base model provides weak procedural ingredients; RL builds them into reliable higher-level strategies.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_fb33127327caf094cfe4dcc9