---
title: "One Form to Transfer Them All: Pretraining Multilingual Language Models Beyond Native Orthography"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.25904v1"
permalink: "https://feed.astralor.com/items/itm_7b6d2d1aa28a8d22c788b2e0"
published: "2026-08-26T15:23:18.000Z"
score: 75
channel: "ai"
tags: ["IPA","Multilingual Language Model","Romanization","Autoregressive Pretraining","Cross-lingual Transfer"]
---

# One Form to Transfer Them All: Pretraining Multilingual Language Models Beyond Native Orthography

## AI commentary

> This paper deserves attention from anyone training multilingual models, elevating romanization from a preprocessing trick to a core design choice. If your team is still pretraining with native orthography, you may already be starting behind the curve.

## At a glance

This study systematically compares three input representations in autoregressive multilingual pretraining: native orthographic text, IPA transcription, and romanization. Across parameter scales from 467M to 1.03B, eight languages, and four typologically motivated pairs, romanized pretraining yields the strongest cross-lingual transfer, with the advantage widening as scale increases. IPA outperforms text in most settings but trails romanization. Surprisingly, finetuning a text-pretrained model on romanized data hurts performance on languages already covered by the base model. The authors recommend romanization as a core pretraining design choice rather than a post hoc fix.

## Key points

- Controlled comparison of orthographic text, IPA, and romanized inputs across 8 languages and 4 typologically motivated pairs
- Romanized pretraining yields the strongest cross-lingual transfer, with advantages widening as scale increases
- IPA outperforms native text in most settings but trails romanization
- Finetuning a text-pretrained model on romanized data hurts performance on covered languages and only marginally helps when script coverage is absent

## Full text

Multilingual language models transfer knowledge across languages through shared subword vocabulary, a mechanism that breaks down when related languages use different writing systems. Prior work addresses this via script equalization (romanization or IPA transcription), but direct comparisons are rare; the focus has been on encoder-only models, with most work adapting existing pretrained models. We systematically compare different input representations in autoregressive multilingual pretraining, comparing orthographic text, IPA, and romanization in a controlled setup across three scales (467M, 709M, and 1.03B) on eight languages in four typologically motivated pairs. Across a wide range of downstream tasks on seen and unseen languages, romanized pretraining yields the strongest cross-lingual transfer, and the advantage over text widens with scale. IPA improves over text in most settings but trails romanization. Surprisingly, finetuning a text-pretrained model on romanized data hurts performance on languages already covered by the base model, only marginally helping when the model lacks script coverage. Our results indicate that for multilingual models spanning typologically diverse scripts, to obtain maximum benefits, romanization should be treated as a core design choice applied at pretraining rather than a post hoc fix.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_7b6d2d1aa28a8d22c788b2e0