---
title: "SciFigQual-Bench: A Benchmark for Scientific Figure Quality Assessment with Full-Manuscript Context"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.27084v1"
permalink: "https://feed.astralor.com/items/itm_b159fea421e9a582be15de0d"
published: "2026-07-29T16:07:44.000Z"
score: 82
channel: "ai"
tags: ["GPT-5.6 Sol","SciFigQual-Bench","SFQ-Agent","Benchmark Dataset","Scientific Figure Quality Assessment","Cross-Modal Learning"]
---

# SciFigQual-Bench: A Benchmark for Scientific Figure Quality Assessment with Full-Manuscript Context

## AI commentary

> This paper precisely targets the gap in scientific figure quality assessment: existing methods either focus on natural photos or only perform surface-level comparisons. SciFigQual-Bench not only provides a large-scale expert-annotated dataset but also binds each figure to its full manuscript context, making evaluation more aligned with real peer-review scenarios. The staged cross-modal design of SFQ-Agent is also insightful—collecting evidence from each modality before fusion, which is more interpretable than direct end-to-end reasoning. This benchmark is worth attention for academic publishing platforms, paper review systems, and teams working on scientific visualization.

## At a glance

Existing image quality assessment methods are mainly designed for natural photos or AI-generated content, and cannot be directly applied to scientific figures. This paper introduces SciFigQual-Bench, a full-text contextual benchmark containing 6,308 scientific images from top computer science conferences (2020-2025), annotated by experts across five dimensions: clarity, layout, caption fit, context relevance, and misleading risk. A staged cross-modal evaluation framework SFQ-Agent is proposed to achieve auditable and refined scoring by collecting and fusing modal evidence. Experiments show that SFQ-Agent (F3) with GPT-5.6-Sol achieves the lowest overall mean absolute error (0.418) and highest consistency rate (93.4%) on the eval1200 subset, consistently outperforming direct evaluation and auxiliary VLM schemes.

## Key points

- Constructed a full-text benchmark with 6,308 scientific images, independently scored by multiple domain experts across five dimensions to form gold-standard annotations.
- Each image is bound to its caption, citing sentence, and manuscript context, overcoming previous limitations of visual-surface-only comparisons.
- Proposed SFQ-Agent, a staged cross-modal evaluation framework that achieves auditable refined scoring through evidence collection and fusion.
- SFQ-Agent (F3) with GPT-5.6-Sol achieves the lowest MAE of 0.418 and highest consistency rate of 93.4%, outperforming direct evaluation and auxiliary VLM schemes.

## Full text

Scientific images are the core elements of presenting experimental conclusions, elaborating system architecture, and supporting comparative arguments in scientific papers. However, existing image quality assessment (IQA) methods are predominantly designed for natural photographs or AI-generated content, which cannot be directly applied to scientific papers. The few existing studies on scholarly charts remain confined to visual-surface comparisons, failing to verify caption alignment, citation relevance, or visual misleadingness. To address this, we propose SciFigQual-Bench, a full-text contextual benchmark that evaluates scientific images across five dimensions (clarity, layout, caption fit, context relevance, and misleading risk). The data covers top computer-science conferences from 2020 to 2025; 6,308 images were independently scored by multiple domain experts in five dimensions and aggregated into gold-standard annotations. Unlike previous scientific figure benchmarks, our dataset binds each image to its caption, citing sentence, and manuscript context. To enable automated evaluation on this benchmark, we designed a staged cross-modal evaluation framework SFQ-Agent to achieve auditable and refined scoring through the collection and fusion of modal evidence. Multiple mainstream large models were evaluated on the test subset eval1200, and SFQ-Agent (F3) equipped with GPT-5.6-Sol achieved the lowest overall average absolute error (0.418) and the highest consistency rate (93.4%), consistently outperforming both direct evaluation and auxiliary (Sidecar) visual language model evaluation schemes.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_b159fea421e9a582be15de0d