---
title: "Autoregressive EHR Foundation Models with Multimodal Inputs"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.22264v1"
permalink: "https://feed.astralor.com/items/itm_c7f3100a3003cfa4309e12cc"
published: "2026-07-24T12:59:20.000Z"
score: 79
channel: "tech"
tags: ["EHR Foundation Models","MIMIC-IV","Multimodal Fusion","Latent Compression","Autoregressive Model","Gated Cross-Attention"]
---

# Autoregressive EHR Foundation Models with Multimodal Inputs

## AI commentary

> This paper is highly valuable for researchers and engineers building multimodal clinical prediction models. It not only provides concrete architectural choices (latent compression outperforms no compression) but also sounds a warning: stacking modalities does not guarantee performance gains and can even hurt. I especially appreciate the authors' honesty in reporting negative results—exactly what real-world deployment needs. If you are trying to incorporate images or text into EHR models, this is a must-read.

## At a glance

This paper presents a framework that enables autoregressive EHR foundation models to incorporate auxiliary modalities such as ECG waveforms, chest X-ray images, and clinical notes through modality-specific latent compression and gated cross-attention with temporal alignment. Controlled ablations on MIMIC-IV show that the best latent-compression configuration outperforms both uncompressed cross-attention and mean pooling, and that encoder choice has a clear within-modality effect. However, simply adding auxiliary modalities does not guarantee improvement on ICU mortality prediction over an EHR-only baseline, highlighting the need for careful fusion architecture design and clinical context evaluation.

## Key points

- Proposes a multimodal framework using latent compression and gated cross-attention to extend autoregressive EHR models.
- Ablations on ECG, X-ray, and clinical notes show the best latent-compression configuration outperforms traditional methods.
- Encoder quality significantly affects within-modality performance, but multimodal fusion does not guarantee improved downstream tasks.
- Emphasizes that multimodal EHR model design requires careful evaluation in clinical contexts.

## Full text

Autoregressive foundation models trained on tokenized electronic health records (EHRs) can support zero-shot clinical prediction, yet most operate on structured event codes alone, and do not incorporate multiple modalities in a principled way. We present a framework for conditioning such models on auxiliary clinical modalities, including ECG waveforms, chest X-ray images, and clinical notes, using modality-specific latent compression and gated cross-attention with temporal alignment. We investigate two key design choices: (1) how to compress long per-modality sequences (e.g., ECG time series) before they enter the multi-modal cross-attention. This feature may be essential to reduce compute overheads and may be beneficial for generalization; (2) how the choice of pretrained encoder for each modality impacts downstream performance. Through controlled ablations on MIMIC-IV, we show that the best latent-compression configurations outperforms both uncompressed cross-attention and mean pooling. Encoder choice has a clear within-modality effect, with stronger pretrained encoders consistently outperforming weaker alternatives. We further show that merely adding auxiliary modalities does not guarantee improvement on ICU mortality prediction over an EHR-only baseline. This implies that careful design of the fusion architecture and an appropriate evaluation in the clinical context are required.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_c7f3100a3003cfa4309e12cc