---
title: "预训练大语言模型的原地分词器扩展方法"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.15232v1"
permalink: "https://feed.astralor.com/items/itm_440442f20bb4352fc75b254f"
published: "2026-07-16T17:32:38.000Z"
score: 74
channel: "ai"
tags: ["BPE","LFM2-8B-A1B","分词器扩展","多语言","大语言模型","端侧部署"]
---

# 预训练大语言模型的原地分词器扩展方法

## AI 点评

> 这篇论文为端侧模型的多语言部署提供了一个非常实用的工程方案，尤其对关注模型推理成本和语言覆盖的团队很有价值。方法简单有效，开源实现降低了复现门槛。虽然是增量工作，但针对的痛点真实，效果显著。

## 速览

预训练时固定的分词器会按语料比例分配词汇，当部署优先级变化时，后续加入的语言每个词被切分为更多token，增加延迟、计算和能耗。云模型可承担大词表，但端侧模型因传输带宽限制只能用小词表，导致非预设语言碎片化。本文提出一种原地分词器扩展方法：在已有BPE merges基础上继续合并多语言语料，保留原有token不变，新增token通过子token嵌入均值初始化。经过两阶段适应训练（先只训练嵌入层，再全模型继续预训练），恢复原始检查点质量。在LFM2-8B-A1B模型上应用后，印地语和越南语编码token数分别减少约2.4倍和2.6倍（泰语达4.0倍），预估每字符解码速度提升2.2-3.7倍。已发布模型权重和扩展后的分词器。

## 关键点

- 提出原地分词器扩展方法，在不改变已有token的前提下扩充分词器词表，新token通过已有子token嵌入均值初始化。
- 两阶段训练策略：先只训练嵌入层，再全模型继续预训练，可恢复原始检查点质量。
- 在8B参数的MoE模型LFM2-8B-A1B上应用，扩展至128K词表，印地语和越南语token数分别减少2.4倍和2.6倍，泰语达4.0倍。
- 预估在多款参考设备上每字符解码速度提升2.2-3.7倍，已开源模型权重和扩展分词器。

## 全文

A tokenizer fixed at the start of pre-training allocates vocabulary in proportion to the pre-training corpus, reflecting the deployment priorities at that time. When those priorities shift, languages added later are split into many more tokens per word, which can raise latency, compute, and energy consumption for users of those languages. Cloud models can afford a broad vocabulary because the embedding and LM-head matrices are a small fraction of their parameters. On a compact model those matrices are a material share of per-token decode bandwidth, so on-device models ship small vocabularies and accept fragmentation outside a fixed language set. We present tokenizer expansion, an in-place recipe for upgrading a pre-trained model's tokenizer when the model producer controls its design. We continue the existing tokenizer's BPE merges on a multilingual corpus, so most source tokens carry over unchanged as single tokens and every new token has an exact decomposition into source tokens. We copy the carried-over embedding rows unchanged and initialize new rows as the mean of their source sub-token embeddings. A two-stage adaptation, embedding-only training then full-model continued pre-training, recovers source-checkpoint quality. We apply the recipe to a continued pre-trained checkpoint of LFM2-8B-A1B, an 8B-parameter Mixture-of-Experts model, to help produce LFM2.5-8B-A1B with a 128K tokenizer. The expanded tokenizer encodes Hindi and Vietnamese in roughly $2.4\times$ and $2.6\times$ fewer tokens than the source (up to $4.0\times$ on Thai). Combining these reductions with the measured per-token cost of the larger vocabulary, we estimate a $2.2$-$3.7\times$ per-character decode speedup for these languages across our reference devices. We release the model weights and the expanded tokenizer, and report the negative findings that shaped the recipe.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_440442f20bb4352fc75b254f