---
title: "Failure-Informed Image Self-Augmentation: A New Framework for Multimodal LLM Self-Improvement"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.03733v1"
permalink: "https://feed.astralor.com/items/itm_130eb098ca64c916521ebf7d"
published: "2026-08-04T14:28:42.000Z"
score: 76
channel: "tech"
tags: ["arXiv","FISA","MLLM","Self-Augmentation","Visual Question Answering"]
---

# Failure-Informed Image Self-Augmentation: A New Framework for Multimodal LLM Self-Improvement

## AI commentary

> The highlight of this paper is turning failure cases into a source of image augmentation—an elegant idea: wherever the model is weak, generate samples to target that weakness. It is especially valuable for teams working on multimodal data engineering and self-evolving models. Worth trying FISA's filtering strategy in your own tasks.

## At a glance

Multimodal large language models rely on costly high-quality multimodal data, and existing self-augmentation methods are largely text-centric, while image augmentation typically uses generic or handcrafted transformations weakly aligned with the model's actual incapability. This paper proposes FISA (Failure-informed Image Self-Augmentation), a framework that constructs augmented images from the model's own failure cases. The method generates visually challenging yet answer-preserving image complications, verifies utility through self-examination, and applies dual fidelity filtering to avoid semantic distortion. Experiments on visual question answering benchmarks show consistent improvements in both in-distribution and out-of-distribution settings. Further experiments validate compatibility with textual self-augmentation, superior data efficiency over generic image augmentation baselines, and practical effectiveness of the filtering strategy.

## Key points

- FISA generates visually challenging yet answer-preserving images from the model's own failure cases for external-supervision-free data expansion.
- Self-examination and dual fidelity filtering avoid semantic distortion, improving data quality.
- On VQA benchmarks, FISA consistently improves performance in both in-distribution and out-of-distribution settings, and is compatible with existing textual self-augmentation methods.
- Synthesized samples show superior data efficiency over generic image augmentation baselines, and the filtering strategy proves practically effective.

## Full text

Multimodal large language models (MLLMs) have achieved remarkable performance across vision-language tasks, but their progress depends heavily on large-scale, high-quality multimodal data that are costly to annotate. Self-augmentation offers a promising alternative by enabling models to expand their own training data without external supervision. However, existing MLLM self-augmentation methods are largely text-centric, while image augmentation remains underexplored and typically relies on generic or handcrafted transformations that are weakly aligned with the model's actual incapability. We propose Failure-informed Image Self-Augmentation (\textbf{FISA}), a framework for MLLM self-improvement that constructs augmented images from the model's own failure cases. Our method generates visually challenging yet answer-preserving image complications, verifies their utility through self-examination, and applies dual fidelity filtering to avoid semantic distortion. Experiments on visual question answering benchmarks show that the proposed method consistently improves performance across both in-distribution and out-of-distribution settings. Further experiments validate the compatibility of FISA with existing textual self-augmentation approaches, the superior data efficiency of the synthesized samples over generic image augmentation baselines, and the practical effectiveness of the proposed filtering strategy.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_130eb098ca64c916521ebf7d