---
title: "多模态预训练的物理原理：知识流动、模态协同、早期统一与配方"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.05000v1"
permalink: "https://feed.astralor.com/items/itm_89c8d730d4c8fba878913333"
published: "2026-08-05T16:09:25.000Z"
score: 77
channel: "ai"
tags: ["arXiv","MoE","多模态预训练","知识流动"]
---

# 多模态预训练的物理原理：知识流动、模态协同、早期统一与配方

## AI 点评

> 这篇论文把多模态预训练从“玄学”推向“物理”，四类发现都很扎实。尤其“视觉懒惰”现象提醒我们，晚融合方案可能让模型偷懒地依赖语言先验——这对想省事的团队是个警示。强烈推荐给正在设计多模态架构的算法工程师和追求效率的研究者。

## 速览

本文通过受控实验系统探索多模态预训练的设计空间与基础机制，在合成和真实大规模数据集上揭示四个核心发现：知识流动呈现非对称影响模式；数据复杂度决定模态协同还是竞争，共享注意力与专用前馈层可促进协同；早期联合训练优于晚期对齐或顺序训练，并发现“视觉懒惰”现象；基于上述洞察可推导高效配方，仅用5%计算预算即可获得强生成性能。这些结论在13.5B MoE模型、2T token规模下得到验证。

## 关键点

- 知识流动：语言、视觉理解与视觉生成之间的知识迁移呈现非对称影响模式。
- 协同与竞争：数据复杂度决定模态关系，共享注意力与模态专用前馈层可增强协同，且跨视觉 tokenizer 泛化。
- 早期统一：从一开始联合训练多模态优于晚期对齐或顺序训练，并发现“视觉懒惰”现象。
- 高效配方：仅用 5% 计算预算即可达到强生成性能，并在 13.5B MoE、2T token 规模上验证。

## 全文

Vision offers a critical axis for advancing foundation models, driving a shift towards natively unified multimodal pretraining. Despite this momentum, the design space and the fundamental mechanisms of how modalities interact during unified training remain underexplored. We provide empirical clarity through a systematic exploration of multimodal pretraining. Our controlled experiments on both synthetic and large-scale real-world datasets yield four key insights into the physics of multimodal pretraining: (i) Knowledge Flow: We disentangle how language, visual understanding, and visual generation transfer knowledge across modalities, revealing distinct patterns of influence and asymmetry; (ii) Synergy vs. Competition: We show that data "complexity" largely determines whether modalities are synergistic, identify architectural choices that promote synergy: such as shared attention and normalization with modality-specific feed-forward layers, and find that these behaviors generalize across different visual tokenizer designs; (iii) Early Unification: Unifying modalities from the very early stages and training them jointly is shown to be more effective than late alignment or sequential training. This process uncovers a vision laziness phenomenon, where delayed integration leads models to rely on language priors; (iv) Recipes: We derive efficient pretraining recipes that achieve strong generative performance using only 5% of the compute budget. These core findings are subsequently validated at scale by training multiple 13.5B MoE models on 2T tokens. We hope this study provides a principled foundation for understanding and scaling multimodal pretraining.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_89c8d730d4c8fba878913333