---
title: "In-Place Tokenizer Expansion for Pre-trained LLMs"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.15232v1"
permalink: "https://feed.astralor.com/items/itm_440442f20bb4352fc75b254f"
published: "2026-07-16T17:32:38.000Z"
score: 74
channel: "ai"
tags: ["BPE","LFM2-8B-A1B","Tokenizer Expansion","Multilingual","Large Language Models","On-device Deployment"]
---

# In-Place Tokenizer Expansion for Pre-trained LLMs

## AI commentary

> This paper offers a highly practical engineering solution for multilingual on-device deployment, especially valuable for teams concerned with inference cost and language coverage. The method is simple and effective, and the open-source release lowers the barrier to reproduction. While incremental, it addresses a real pain point with significant impact.

## At a glance

A tokenizer fixed during pre-training allocates vocabulary proportionally to the training corpus. When priorities shift, later-added languages suffer from more tokens per word, increasing latency, compute, and energy. Cloud models can afford large vocabularies, but on-device models use small vocabularies due to bandwidth constraints, causing fragmentation for non-default languages. This paper introduces an in-place tokenizer expansion method: continue BPE merges on a multilingual corpus, keep source tokens unchanged, and initialize new token embeddings as the mean of their source sub-token embeddings. A two-stage adaptation (embedding-only then full-model continued pre-training) recovers source checkpoint quality. Applied to LFM2-8B-A1B, it reduces tokens for Hindi and Vietnamese by ~2.4× and 2.6× (up to 4.0× for Thai), with estimated 2.2-3.7× per-character decode speedup. Model weights and the expanded tokenizer are released.

## Key points

- Proposes an in-place tokenizer expansion method that preserves source tokens and initializes new token embeddings as mean of their source sub-token embeddings.
- Two-stage adaptation: embedding-only training followed by full-model continued pre-training recovers source checkpoint quality.
- Applied to LFM2-8B-A1B (8B MoE model), expands to 128K vocabulary, reducing tokens for Hindi and Vietnamese by 2.4× and 2.6× (up to 4.0× for Thai).
- Estimated 2.2-3.7× per-character decode speedup on reference devices; model weights and expanded tokenizer are open-sourced.

## Full text

A tokenizer fixed at the start of pre-training allocates vocabulary in proportion to the pre-training corpus, reflecting the deployment priorities at that time. When those priorities shift, languages added later are split into many more tokens per word, which can raise latency, compute, and energy consumption for users of those languages. Cloud models can afford a broad vocabulary because the embedding and LM-head matrices are a small fraction of their parameters. On a compact model those matrices are a material share of per-token decode bandwidth, so on-device models ship small vocabularies and accept fragmentation outside a fixed language set. We present tokenizer expansion, an in-place recipe for upgrading a pre-trained model's tokenizer when the model producer controls its design. We continue the existing tokenizer's BPE merges on a multilingual corpus, so most source tokens carry over unchanged as single tokens and every new token has an exact decomposition into source tokens. We copy the carried-over embedding rows unchanged and initialize new rows as the mean of their source sub-token embeddings. A two-stage adaptation, embedding-only training then full-model continued pre-training, recovers source-checkpoint quality. We apply the recipe to a continued pre-trained checkpoint of LFM2-8B-A1B, an 8B-parameter Mixture-of-Experts model, to help produce LFM2.5-8B-A1B with a 128K tokenizer. The expanded tokenizer encodes Hindi and Vietnamese in roughly $2.4\times$ and $2.6\times$ fewer tokens than the source (up to $4.0\times$ on Thai). Combining these reductions with the measured per-token cost of the larger vocabulary, we estimate a $2.2$-$3.7\times$ per-character decode speedup for these languages across our reference devices. We release the model weights and the expanded tokenizer, and report the negative findings that shaped the recipe.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_440442f20bb4352fc75b254f