---
title: "HiLo-RD: Cross-Budget Self-Distillation for Masked Diffusion Language Models"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2610.36901v1"
permalink: "https://feed.astralor.com/items/itm_994181b9314cf171d7968cdc"
published: "2026-10-27T09:34:08.000Z"
score: 57
channel: "ai"
tags: ["RAG","多模态","HiLo-RD","Cross-Budget Self-Distillation","Masked Diffusion Language","Models","High-to-Low Budget Representation","Distillation"]
---

# HiLo-RD: Cross-Budget Self-Distillation for Masked Diffusion Language Models

## 速览

We propose HiLo-RD (High-to-Low Budget Representation Distillation) to improve masked diffusion language models (MDLMs) under a limited generation budget. Unlike autoregressive models, MDLMs choose both what to generate…

## 关键点

- We propose HiLo-RD (High-to-Low Budget Representation Distillation) to improve masked diffusion language models (MDLMs) under a limited generation budget.
- Unlike autoregressive models, MDLMs choose both what to generate and where to commit it.
- This creates two coupled challenges: predicting useful content before reasoning is complete, and deciding when to fix answer tokens that shape later predictions.

## 全文

We propose HiLo-RD (High-to-Low Budget Representation Distillation) to improve masked diffusion language models (MDLMs) under a limited generation budget. Unlike autoregressive models, MDLMs choose both what to generate and where to commit it. This creates two coupled challenges: predicting useful content before reasoning is complete, and deciding when to fix answer tokens that shape later predictions. We identify budget-induced premature commitment: even when a correct solution fits, the model can fix answer fragments near the window boundary before resolving intermediate steps. These fragments become context for later predictions and redirect subsequent reasoning. Existing post-training methods do not directly transfer the stronger trajectories that the same model follows with a larger window. HiLo-RD addresses this gap by (1) pairing large-window teacher states with cropped small-window views and (2) matching their hidden representations at committed positions. This supervision guides both token prediction and commitment order without an external teacher. Across mathematics benchmarks, HiLo-RD improves low-budget accuracy and shortens responses; these gains extend to code generation without code-domain training. On GSM8K, accuracy rises by 9.4 percentage points as mean readable length falls by about 25%. Attention diagnostics and fixed-budget order interventions support the proposed mechanism. Together, these results show that large-window reasoning can improve small-window decoding.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_994181b9314cf171d7968cdc