---
title: "超越前缀局部性的混合RL rollout调度策略MISA-T"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.11152v1"
permalink: "https://feed.astralor.com/items/itm_e5d2ca5b48eb91c2d528c149"
published: "2026-08-11T17:10:50.000Z"
score: 76
channel: "ai"
tags: ["KV缓存调度","MISA-T","rollout调度","vLLM","强化学习后训练"]
---

# 超越前缀局部性的混合RL rollout调度策略MISA-T

## AI 点评

> 这篇论文值得做RL基础设施的团队关注，尤其是正在跑大规模RL训练并遇到推理瓶颈的人。MISA-T把KV缓存容量调度和负载混合控制放在一起，解决的是真实系统里很痛的‘rollout抢KV’问题，而不是单纯理论优化。如果你在用vLLM或类似框架支撑多任务RL后训练，这个思路可以直接启发路由策略改造。

## 速览

现代大语言模型的RL后训练流水线常混合多种rollout工作负载，如RLVR、RLHF和Agentic rollout。现有前缀感知路由虽能优化缓存复用与负载均衡，但未控制异构会话对KV缓存容量的竞争。本文提出MISA-T，一种路由层准入策略，包含自适应会话准入、工作负载感知的KV容量分配和驻留时间感知的KV记账。在Step3.7和Qwen3.6-35B-A3B上的实验表明，相比调优的vLLM路由器，MISA-T分别提升rollout吞吐量53.3%和43.6%，并在50轮匹配实验中提升35.6%吞吐量、降低22.8%平均迭代时间，同时保持工作负载混合接近训练目标。

## 关键点

- MISA-T 是面向混合RL rollout的路由层准入策略，专门处理异构会话对KV缓存容量的竞争。
- 自适应会话准入与工作负载感知的KV容量分配结合，避免扭曲训练器指定的负载混合。
- 在Step3.7上提升吞吐量53.3%，在Qwen3.6-35B-A3B上提升43.6%，均优于调优过的vLLM Router。
- 在50轮完整实验中，吞吐量提升35.6%，平均迭代时间下降22.8%，任务分数与基线相当。

## 全文

Modern reinforcement learning (RL) post-training pipelines for large language models (LLMs) increasingly combine rollout workloads across multiple domains and feedback paradigms. Prefix-aware routing improves inference efficiency through cache reuse and load balancing, but it does not control how heterogeneous rollout sessions compete for KV-cache capacity. When reinforcement learning with verifiable rewards (RLVR), reinforcement learning from human feedback (RLHF), and agentic rollouts share an asynchronous inference service, their distinct sequence structures, interaction patterns, and KV-residency times create substantially different serving demands. Rollout scheduling must account for this heterogeneity without distorting the workload mixture specified by the trainer. We present MISA-T, a routing-layer admission policy for mixed rollout serving. MISA-T combines adaptive session admission, workload-aware KV-capacity allocation, and residency-time-aware KV accounting. In rollout-only ablations on Step3.7 and Qwen3.6-35B-A3B, MISA-T improves rollout throughput over a sweep-tuned cache-aware vLLM Router by 53.3% and 43.6%, respectively, while maintaining high prefix-cache hit rates. In a matched 50-iteration Step3.7 experiment, it increases rollout throughput by 35.6% and reduces mean iteration time by 22.8%, while keeping the consumed workload mixture close to the trainer target and achieving comparable task scores.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_e5d2ca5b48eb91c2d528c149