---
title: "TREK：利用蒸馏探索、强化精炼——提升GRPO在困难推理任务上的表现"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.05339v1"
permalink: "https://feed.astralor.com/items/itm_e940216a94ad0f63717dcdba"
published: "2026-07-06T17:21:16.000Z"
score: 80
channel: "ai"
tags: ["Agent任务","DeepSeek","GRPO","Qwen3","TREK","数学推理"]
---

# TREK：利用蒸馏探索、强化精炼——提升GRPO在困难推理任务上的表现

## AI 点评

> 这篇论文对正在做强化学习推理优化的团队非常值得一读。TREK的核心洞察简单但有效——把蒸馏从模仿工具变成探索工具，直接解决了GRPO在困难问题上的覆盖不足问题。最实用的是它不需要教师模型内部信息，甚至可以用同一个模型自上下文，部署门槛很低。如果你正在用GRPO或类似方法训练推理模型，TREK很可能是一个即插即用的改进方案。

## 速览

本文提出TREK（Teacher-Routed Exploration via Forward KL）方法，针对GRPO在困难提示上采样不到正确推理路径的问题，通过蒸馏（而非模仿）扩展学生策略的支持域。TREK首先识别学生低通过率的提示，利用外部或内部教师生成候选解，基于学生似然保留最优r个，经短前向KL阶段将其拉入学生支持域，再回到标准GRPO微调。在数学推理（AIME 2024/2025）上，TREK搭配DeepSeek-V4将Qwen3-8B的AIME 2025成绩从36.9提升至40.3，AIME 2024从47.9提升至51.1；无外部教师时自上下文变体达到38.5和49.6。在Agent任务上，ALFWorld成功率从75.8升至82.8，ScienceWorld从12.5升至26.7，且最难任务类型上TREK早期即达高成功率。

## 关键点

- TREK通过前向KL蒸馏扩展学生策略支持域，而非用于模仿。
- 仅需推理轨迹即可工作，兼容黑盒、白盒教师或同一模型自上下文。
- 在数学推理基准上显著提升Qwen3-8B在AIME 2024/2025上的成绩。
- 在Agent任务（ALFWorld、ScienceWorld）上成功率大幅提升，尤其在最难任务上早期高效。

## 全文

Group Relative Policy Optimization (GRPO) is effective when the current policy already samples useful reasoning trajectories, but it stalls on hard prompts whose correct solution modes lie outside the student's on-policy support. We propose TREK (Teacher-Routed Exploration via Forward KL), a simple staged procedure that uses distillation not for imitation but for exploration support expansion. A key advantage of TREK is its generality: because it only consumes verified output trajectories, it can use an external black-box teacher, a white-box teacher, or the same model given additional inference-time context, and it can efficiently identify which hard-prompt samples are most worth consolidating even when teacher internals are unavailable. TREK first identifies prompts where the unaided student has very low pass rate, queries a proposal source to produce verified candidate solutions, keeps the top-$r$ proposals ranked by current student likelihood, applies a short forward-KL phase to pull those verified modes into the student's support, and then returns to standard on-policy GRPO refinement. On mathematical reasoning, TREK with DeepSeek-V4 proposals improves Qwen3 models across all tested scales on AIME 2024 and AIME 2025; for Qwen3-8B, it improves AIME 2025 from 36.9 to 40.3 and AIME 2024 from 47.9 to 51.1 (avg@16), while the self-context variant reaches 38.5 and 49.6 without an external teacher. On agentic tasks, TREK raises ALFWorld success rate from 75.8 to 82.8 and ScienceWorld success rate from 12.5 to 26.7; notably, on the hardest task types, TREK achieves high success rates early in training while unaided GRPO requires substantially more optimization steps to reach comparable levels.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_e940216a94ad0f63717dcdba