---
title: "TREK: Distill to Explore, Reinforce to Refine – Boosting GRPO on Hard Reasoning Tasks"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.05339v1"
permalink: "https://feed.astralor.com/items/itm_e940216a94ad0f63717dcdba"
published: "2026-07-06T17:21:16.000Z"
score: 80
channel: "ai"
tags: ["agentic tasks","DeepSeek","GRPO","Qwen3","TREK","Mathematical Reasoning"]
---

# TREK: Distill to Explore, Reinforce to Refine – Boosting GRPO on Hard Reasoning Tasks

## AI commentary

> This paper is a must-read for teams working on reinforcement learning reasoning optimization. The core insight of TREK is simple yet effective: repurposing distillation from imitation to exploration, directly addressing GRPO's coverage issue on hard problems. Most practically, it doesn't require teacher internals and can even use self-context, lowering deployment barriers. If you're training reasoning models with GRPO or similar methods, TREK is likely a plug-and-play improvement.

## At a glance

This paper proposes TREK (Teacher-Routed Exploration via Forward KL), a method that addresses GRPO's stall on hard prompts where correct solution modes lie outside the student's on-policy support. Instead of using distillation for imitation, TREK expands student support. It first identifies prompts with low student pass rate, queries a teacher to generate verified candidates, selects top-r by student likelihood, applies a short forward-KL phase to pull those modes into support, then resumes standard GRPO refinement. On math reasoning, TREK with DeepSeek-V4 improves Qwen3-8B on AIME 2025 from 36.9 to 40.3 and AIME 2024 from 47.9 to 51.1 (avg@16); self-context variant reaches 38.5 and 49.6 without external teacher. On agentic tasks, TREK raises ALFWorld success rate from 75.8 to 82.8 and ScienceWorld from 12.5 to 26.7, notably achieving high success rates early on hardest tasks while unaided GRPO requires many more steps.

## Key points

- TREK expands student policy support via forward-KL distillation, not imitation.
- Works with only verified output trajectories, compatible with black-box, white-box teacher, or self-context.
- Significantly improves Qwen3-8B on AIME 2024/2025 math reasoning benchmarks.
- Substantially boosts success rates on agentic tasks (ALFWorld, ScienceWorld), especially early on hardest task types.

## Full text

Group Relative Policy Optimization (GRPO) is effective when the current policy already samples useful reasoning trajectories, but it stalls on hard prompts whose correct solution modes lie outside the student's on-policy support. We propose TREK (Teacher-Routed Exploration via Forward KL), a simple staged procedure that uses distillation not for imitation but for exploration support expansion. A key advantage of TREK is its generality: because it only consumes verified output trajectories, it can use an external black-box teacher, a white-box teacher, or the same model given additional inference-time context, and it can efficiently identify which hard-prompt samples are most worth consolidating even when teacher internals are unavailable. TREK first identifies prompts where the unaided student has very low pass rate, queries a proposal source to produce verified candidate solutions, keeps the top-$r$ proposals ranked by current student likelihood, applies a short forward-KL phase to pull those verified modes into the student's support, and then returns to standard on-policy GRPO refinement. On mathematical reasoning, TREK with DeepSeek-V4 proposals improves Qwen3 models across all tested scales on AIME 2024 and AIME 2025; for Qwen3-8B, it improves AIME 2025 from 36.9 to 40.3 and AIME 2024 from 47.9 to 51.1 (avg@16), while the self-context variant reaches 38.5 and 49.6 without an external teacher. On agentic tasks, TREK raises ALFWorld success rate from 75.8 to 82.8 and ScienceWorld success rate from 12.5 to 26.7; notably, on the hardest task types, TREK achieves high success rates early in training while unaided GRPO requires substantially more optimization steps to reach comparable levels.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_e940216a94ad0f63717dcdba