---
title: "ClinFusion: A Vision-Centric Multimodal LLM System for Holistic Medical Understanding"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.24743v1"
permalink: "https://feed.astralor.com/items/itm_5c64f310358917c68f740e9a"
published: "2026-07-27T17:59:49.000Z"
score: 79
channel: "ai"
tags: ["ClinFusion","MedIF-Bench","Medical Image Understanding","MLLM","Cascade Spatial-Aware Locality Fusion"]
---

# ClinFusion: A Vision-Centric Multimodal LLM System for Holistic Medical Understanding

## AI commentary

> This paper finds a sweet spot in medical multimodal LLMs with its vision-centric design. The inclusion of blinded radiologist evaluation adds credibility, making it especially valuable for practitioners focused on clinical AI deployment and evaluation paradigms. Researchers should pay attention to the cascaded encoder design and the ROI-based evaluation metric.

## At a glance

ClinFusion is a vision-centric multimodal large language model designed to tackle heterogeneity in medical image understanding and evaluation alignment. It proposes a cascaded spatial-aware locality fusion encoder to unify 2D and 3D medical images, and introduces MedIF-Bench for instruction-following and a region-of-interest-grounded metric for report generation evaluation. Across 24 benchmarks covering VQA, report generation, and instruction-following, ClinFusion outperforms open-source medical MLLMs (e.g., Hulu-Med, Lingshu) on 20, and surpasses proprietary models like GPT-5.2 and Gemini-3-Flash on 13 out of 16. A blinded evaluation by board-certified radiologists confirms ClinFusion produces the highest-quality reports, and its RoI-grounded metric achieves the strongest correlation with expert judgment. It can be further augmented with agentic tool use for retrieval-augmented and tool-assisted clinical workflows.

## Key points

- ClinFusion uses a cascaded spatial-aware locality fusion encoder to unify 2D/3D medical image understanding.
- It introduces MedIF-Bench for instruction-following and a region-of-interest-grounded metric for report evaluation.
- Outperforms open-source medical MLLMs on 20/24 benchmarks and proprietary models (GPT-5.2, Gemini-3-Flash) on 13/16 benchmarks.
- Blinded radiologist evaluation confirms ClinFusion's reports are highest-ranked; new metric best correlates with expert judgment.

## Full text

Multimodal large language models (MLLMs) hold immense potential to revolutionize clinical practice, yet deploying them in the medical domain is fundamentally a vision-centric challenge: models must absorb knowledge from heterogeneous 2D and 3D medical images, and evaluation protocols must align with radiologists' clinical practice and provide an accurate, fine-grained and factualness-driven assessment. In this paper, we introduce ClinFusion, a vision-centric MLLM designed for holistic medical understanding that systematically addresses these limitations. We propose a compositional and cascaded vision encoder architecture featuring a Cascade Spatial-Aware Locality Fusion operator that unifies diverse 2D and native 3D medical image understanding within a fused encoder. We further introduce a vision-grounded evaluation framework, including MedIF-Bench for instruction-following assessment and a region-of-interest-grounded method for clinically aligned and factualness-driven report generation evaluation. We show that ClinFusion sets a new state-of-the-art across a comprehensive suite of 2D and 3D multimodal medical benchmarks---spanning visual question answering, report generation, and instruction following---as well as textual medical tasks, outperforming leading open-source medical MLLMs (\textit{e.g.}, Hulu-Med, Lingshu) on 20 out of 24 benchmarks and demonstrating multimodal capabilities better than powerful proprietary models such as GPT-5.2 and Gemini-3-Flash on 13 out of 16 benchmarks, and can be further augmented with agentic tool use for retrieval-augmented and tool-assisted clinical workflows. A blinded evaluation by board-certified radiologists confirms that ClinFusion produces the highest-ranked reports, and validates our RoI-grounded metric as achieving the strongest correlation with expert judgment among all automatic evaluation metrics examined.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_5c64f310358917c68f740e9a