---
title: "PyroDash: Cost-Efficient Token-Level Small-Large Language Model Collaborative Inference"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.20327v1"
permalink: "https://feed.astralor.com/items/itm_95c5a09999cbcfc43335d2fc"
published: "2026-07-22T16:14:26.000Z"
score: 78
channel: "ai"
tags: ["PyroDash","SLM-LLM collaborative inference"]
---

# PyroDash: Cost-Efficient Token-Level Small-Large Language Model Collaborative Inference

## AI commentary

> PyroDash impresses me with its clever control-token mechanism that lets small models learn when to ask for help, dramatically cutting LLM costs. For engineers working on LLM inference deployment and cost optimization, this is a must-read. The fact that it requires no LLM modification nor a separate router makes it highly practical.

## At a glance

PyroDash is a cost-aware framework for token-level small-large language model collaborative inference. It enables the SLM to decide during generation whether to hand off partial reasoning traces to a frozen LLM via a control token, requiring no separate router, LLM retraining, or access to LLM logits. Through three-stage training—control-token embedding learning, offloading-oriented supervised fine-tuning, and cost-aware alignment with GRPO—it achieves flexible accuracy-cost trade-offs across five math reasoning benchmarks. With as low as 1.90% LLM token ratio, it reduces total inference cost from $49.36 to $1.78 while preserving competitive accuracy.

## Key points

- Proposes token-level SLM-LLM collaborative inference where the SLM uses a control token to decide when to request LLM assistance, without separate routers or LLM fine-tuning.
- Three-stage training: control-token embedding learning, offloading-oriented SFT, and cost-aware alignment with GRPO to balance accuracy and inference cost.
- On five math reasoning benchmarks, PyroDash achieves 54.55% accuracy with only 1.90% LLM token ratio, reducing cost by 96.4%.

## Full text

Large language models (LLMs) provide strong reasoning capabilities but are expensive to serve at scale, whereas small language models (SLMs) are cheaper but less reliable on difficult problems. We introduce PyroDash, a cost-aware framework for token-level SLM-LLM collaborative inference. During generation, the SLM decides whether to request assistance by emitting a control token. A Collaborate Engine then sends the query and partial reasoning trace to a frozen LLM for completion through a single handoff. The policy is internalized in the SLM, requiring neither a separate router, LLM retraining, nor access to LLM logits. PyroDash trains the SLM in three stages: control-token embedding learning, offloading-oriented supervised fine-tuning, and cost-aware alignment with Group Relative Policy Optimization. Its reward balances answer accuracy against inference cost normalized by LLM-only inference. Across five mathematical reasoning benchmarks, PyroDash supports different accuracy-cost operating points. With $λ=0.05$, it achieves 64.04 percent average accuracy, 6.36 percentage points above the LLM-only baseline, while reducing cost by 20.4 percent. With $λ=0.6$, it achieves 54.55 percent accuracy with a 1.90 percent LLM token ratio and 0.012 LLM calls per example, reducing total cost from USD 49.36 to USD 1.78. These results show that learned token-level handoffs can reduce LLM use while preserving strong reasoning performance.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_95c5a09999cbcfc43335d2fc