---
title: "Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.05000v1"
permalink: "https://feed.astralor.com/items/itm_89c8d730d4c8fba878913333"
published: "2026-08-05T16:09:25.000Z"
score: 77
channel: "ai"
tags: ["arXiv","MoE","Multimodal Pre-training","Knowledge Flow"]
---

# Towards Physics of Multimodal Pretraining: Knowledge Flow, Modality Synergy, Early Unification, and Recipes

## AI commentary

> This paper pushes multimodal pretraining from 'alchemy' toward 'physics', with four solid empirical findings. The 'vision laziness' phenomenon is a particularly useful warning that late fusion may lead models to lazily rely on language priors—a caution for teams seeking shortcuts. Highly recommended for algorithm engineers designing multimodal architectures and researchers pursuing efficiency.

## At a glance

This paper systematically explores the design space and fundamental mechanisms of multimodal pretraining through controlled experiments on synthetic and large-scale real-world datasets. Four key insights emerge: knowledge flow across modalities shows distinct asymmetric influence patterns; data complexity determines whether modalities are synergistic or competitive, with shared attention and modality-specific feed-forward layers promoting synergy; early joint training outperforms late alignment or sequential training, revealing a 'vision laziness' phenomenon; and derived recipes achieve strong generative performance using only 5% of the compute budget. These findings are validated at scale by training multiple 13.5B MoE models on 2T tokens.

## Key points

- Knowledge Flow: Language, visual understanding, and visual generation transfer knowledge across modalities with distinct asymmetric influence patterns.
- Synergy vs. Competition: Data complexity determines modality relationships; shared attention with modality-specific feed-forward layers promotes synergy, generalizing across visual tokenizer designs.
- Early Unification: Joint training from the very early stages outperforms late alignment or sequential training, uncovering a 'vision laziness' phenomenon.
- Efficient Recipes: Strong generative performance is achieved with only 5% of the compute budget, validated on 13.5B MoE models trained on 2T tokens.

## Full text

Vision offers a critical axis for advancing foundation models, driving a shift towards natively unified multimodal pretraining. Despite this momentum, the design space and the fundamental mechanisms of how modalities interact during unified training remain underexplored. We provide empirical clarity through a systematic exploration of multimodal pretraining. Our controlled experiments on both synthetic and large-scale real-world datasets yield four key insights into the physics of multimodal pretraining: (i) Knowledge Flow: We disentangle how language, visual understanding, and visual generation transfer knowledge across modalities, revealing distinct patterns of influence and asymmetry; (ii) Synergy vs. Competition: We show that data "complexity" largely determines whether modalities are synergistic, identify architectural choices that promote synergy: such as shared attention and normalization with modality-specific feed-forward layers, and find that these behaviors generalize across different visual tokenizer designs; (iii) Early Unification: Unifying modalities from the very early stages and training them jointly is shown to be more effective than late alignment or sequential training. This process uncovers a vision laziness phenomenon, where delayed integration leads models to rely on language priors; (iv) Recipes: We derive efficient pretraining recipes that achieve strong generative performance using only 5% of the compute budget. These core findings are subsequently validated at scale by training multiple 13.5B MoE models on 2T tokens. We hope this study provides a principled foundation for understanding and scaling multimodal pretraining.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_89c8d730d4c8fba878913333