---
title: "RL Post-Training Builds Compositional Reasoning Strategies: Reorganizing Primitive Skills into Higher-Level Strategies"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.07646v1"
permalink: "https://feed.astralor.com/items/itm_fb33127327caf094cfe4dcc9"
published: "2026-07-08T17:04:42.000Z"
score: 76
channel: "ai"
tags: ["arXiv","Transformer","RL Post-training","Rejection Fine-Tuning","Compositional Reasoning"]
---

# RL Post-Training Builds Compositional Reasoning Strategies: Reorganizing Primitive Skills into Higher-Level Strategies

## AI commentary

> This paper is one of the most transparent mechanistic explanations of RL post-training I've seen recently. By using a simple symbolic environment that strips away confounding factors, it directly shows how RL 'weaves' reusable higher-level strategies from fragmented primitive skills. If you're optimizing training strategies for reasoning models, especially comparing RL with SFT/rejection fine-tuning, this paper provides solid experimental evidence: the key is not how much you explore, but what you explore. The quantifiable importance of pretraining organization also offers direct insights for designing pretraining tasks.

## At a glance

This study investigates whether RL post-training can compose primitive skills of a base model into higher-level reasoning strategies using a fully observable rewrite-grammar environment. Experiments show that RL not only amplifies primitive abilities but reorganizes them through a phased compositional mechanism: first strengthening primitive reductions, then discovering valid composed procedures (sequential and parallel compositions) that are reused and consolidated into a stable repertoire. Compared to rejection fine-tuning, the key advantage of RL is selectivity rather than exploration volume: RFT produces many shortcut-like invalid rewrites while RL concentrates on valid reusable structures. Pretraining must organize primitive competence into reduction procedures that RL can later compress for strategy emergence.

## Key points

- RL post-training composes primitive symbol-rewrite skills into sequential and parallel higher-level strategies that are stably reused
- Rejection fine-tuning improves early but plateaus, producing many invalid shortcuts, while RL concentrates exploration on valid reusable structures
- Emergence of compositional strategies depends on pretraining organizing primitive competence into compressible reduction procedures, not just primitive exposure

## Full text

Does RL post-training merely amplify primitive skills already latent in a base model, or can it compose primitive skills into new higher-level strategies? We study this question in a fully observable rewrite-grammar environment where the pretraining distribution is known and every generated rewrite can be audited. A Transformer is pretrained on primitive symbol-rewrite chains and post-trained on a Trace-based reasoning task with only a binary final-answer reward. RL solves held-out problems that remain rarely solved by the pretrained model even under much larger sampling budgets, while rejection fine-tuning improves early but plateaus. Trace analysis shows that RL reorganizes primitive competence through a phased compositional mechanism: it first strengthens primitive reductions, then discovers valid composed procedures. These include sequential compositions, which collapse ordered chains of primitive contractions, and parallel compositions, which combine independent primitive contractions in a single step. The composed procedures are not isolated samples; they are reused and consolidated into a stable repertoire. Comparing RL with rejection fine-tuning shows that the key difference is not exploration volume but selectivity: RFT produces many shortcut-like rewrites, much of them invalid, whereas RL concentrates exploration into valid reusable structure. Pretraining ablations show that the emergence of compositional strategies is gated not by primitive exposure alone, but by whether pretraining organizes primitive competence into reduction procedures that RL can later compress. The base model provides weak procedural ingredients; RL builds them into reliable higher-level strategies.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_fb33127327caf094cfe4dcc9