---
title: "自回归EHR基础模型的多模态输入框架：潜在压缩与门控交叉注意力"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.22264v1"
permalink: "https://feed.astralor.com/items/itm_c7f3100a3003cfa4309e12cc"
published: "2026-07-24T12:59:20.000Z"
score: 79
channel: "tech"
tags: ["EHR基础模型","MIMIC-IV","多模态融合","潜在压缩","自回归模型","门控交叉注意力"]
---

# 自回归EHR基础模型的多模态输入框架：潜在压缩与门控交叉注意力

## AI 点评

> 这篇论文对正在开发多模态临床预测模型的研究者和工程师非常有价值。它不仅给出了具体的架构选择（潜在压缩优于无压缩），还敲响了警钟：堆叠模态不一定提升性能，甚至可能有害。我特别欣赏作者没有回避负面结果，这恰恰是实际部署中最需要关注的。如果你在尝试将影像或文本融入EHR模型，这篇是必读。

## 速览

该论文提出一种框架，使自回归电子健康记录（EHR）基础模型能够融合ECG波形、胸部X光图像和临床笔记等辅助模态，通过模态特定潜在压缩与时间对齐的门控交叉注意力实现多模态条件化。在MIMIC-IV上的消融实验表明，最佳潜在压缩配置优于无压缩交叉注意力与均值池化，且编码器选择对模态内效果影响显著。然而，简单添加模态并不保证在ICU死亡率预测上超越纯EHR基线，提示需要精心设计融合架构并进行临床场景评估。

## 关键点

- 提出一种多模态框架，利用潜在压缩和门控交叉注意力扩展自回归EHR模型
- 对ECG、X光和临床笔记三种模态进行消融，发现最优潜在压缩配置超越传统方法
- 编码器质量显著影响模态内性能，但多模态融合不必然提升下游任务
- 强调多模态EHR模型的设计需要针对临床场景进行谨慎评估

## 全文

Autoregressive foundation models trained on tokenized electronic health records (EHRs) can support zero-shot clinical prediction, yet most operate on structured event codes alone, and do not incorporate multiple modalities in a principled way. We present a framework for conditioning such models on auxiliary clinical modalities, including ECG waveforms, chest X-ray images, and clinical notes, using modality-specific latent compression and gated cross-attention with temporal alignment. We investigate two key design choices: (1) how to compress long per-modality sequences (e.g., ECG time series) before they enter the multi-modal cross-attention. This feature may be essential to reduce compute overheads and may be beneficial for generalization; (2) how the choice of pretrained encoder for each modality impacts downstream performance. Through controlled ablations on MIMIC-IV, we show that the best latent-compression configurations outperforms both uncompressed cross-attention and mean pooling. Encoder choice has a clear within-modality effect, with stronger pretrained encoders consistently outperforming weaker alternatives. We further show that merely adding auxiliary modalities does not guarantee improvement on ICU mortality prediction over an EHR-only baseline. This implies that careful design of the fusion architecture and an appropriate evaluation in the clinical context are required.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_c7f3100a3003cfa4309e12cc