---
title: "Diagram-MMU: A Multi-Modal Benchmark for Scientific Diagram Parsing and Understanding"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.12262v1"
permalink: "https://feed.astralor.com/items/itm_1d3dd48b7cf9c029b557b8d2"
published: "2026-08-12T17:04:13.000Z"
score: 72
channel: "tech"
tags: ["Diagram-MMU","LaTeX/TikZ","OpenAI Prism","MLLMs","Agent Evaluation","Scientific Diagram Understanding"]
---

# Diagram-MMU: A Multi-Modal Benchmark for Scientific Diagram Parsing and Understanding

## AI commentary

> The most interesting part of this benchmark is that it shifts evaluation from 'asking about diagrams' to 'writing code from diagrams' — the bottleneck of real scientific writing tools. For researchers working on multimodal agents and scientific document automation, Diagram-MMU offers a clear difficulty hierarchy and failure mode analysis. I'm excited to see how model performance evolves on diagram-to-code tasks, since code generation reflects deeper understanding than multiple-choice QA.

## At a glance

Diagram-MMU is a new multi-modal benchmark designed to assess MLLMs' ability to parse and understand scientific diagrams. It features 3.7K curated diagrams and 18.3K human-validated questions across six domains, covering three tasks: diagram-to-code parsing, diagram-to-code editing, and diagram question answering, each also evaluated under agentic settings. Evaluation of 12 MLLMs reveals that diagram-to-code tasks are more challenging than diagram question answering: models can reason well over diagrams but struggle to parse and edit them, highlighting the need to enhance diagram-to-code generation. Under agentic settings, most models improve parsing and editing but degrade on question answering, while Claude-4.6 Opus consistently improves across all three tasks. This benchmark provides a robust testbed for improving scientific diagram understanding in tools like OpenAI Prism and has implications for scientific writing and collaboration.

## Key points

- Diagram-MMU includes 3.7K diagrams and 18.3K questions across six scientific domains.
- Three core tasks: diagram-to-code parsing, diagram-to-code editing, and diagram QA, all with agentic settings.
- Evaluation of 12 MLLMs shows diagram-to-code tasks are harder than QA.
- Claude-4.6 Opus is the only model that improves on all three tasks under agentic settings.

## Full text

Multimodal Large Language Models (MLLMs) have been growing the capability for scientific writing and collaboration. For example, OpenAI Prism is a free workspace for scientific writing and collaboration. One important feature in Prism is turning scientific diagrams directly into LaTeX TikZ code. In this paper, we build a benchmark, Diagram-MMU, a multi-modal benchmark designed to assess MLLMs' ability for scientific diagram parsing and understanding. Diagram-MMU features 3.7k curated diagrams and 18.3k human-validated questions across six domains. It evaluates MLLMs on three tasks common in vibe writing workspaces: diagram-to-code parsing, diagram-to-code editing, and diagram question answering, alongside agentic settings per task. The evaluation of 12 MLLMs reveals that diagram-to-code tasks are more challenging than diagram question answering: models can reason well over diagrams but struggle to parse and edit them, underscoring the need for methods to enhance MLLMs' capability in diagram-to-code generation. Under agentic settings, most models improve parsing and editing performance but degrade on question answering, while Claude-4.6 Opus consistently improves across all three tasks. Project Page: https://vi-ocean.github.io/projects/diagram-mmu.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_1d3dd48b7cf9c029b557b8d2