---
title: "失败驱动的图像自增强：多模态大模型自我改进新框架"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.03733v1"
permalink: "https://feed.astralor.com/items/itm_130eb098ca64c916521ebf7d"
published: "2026-08-04T14:28:42.000Z"
score: 76
channel: "tech"
tags: ["arXiv","FISA","多模态大语言模型","自增强","视觉问答"]
---

# 失败驱动的图像自增强：多模态大模型自我改进新框架

## AI 点评

> 这篇文章的亮点在于把失败案例变成图像增强的源头，思路很妙——模型哪里不会，就针对哪里造样本。对做多模态数据工程和自进化模型的团队尤其有参考价值，值得结合自己的任务试试 FISA 的过滤策略。

## 速览

多模态大语言模型依赖昂贵的高质量多模态数据，现有自增强方法以文本为中心，图像增强通常使用通用或手工变换，与模型实际缺陷弱对齐。本文提出 FISA（失败信息驱动的图像自增强）框架，从模型自身失败案例构建视觉上有挑战但保持答案不变的图像，通过自检验证和双保真过滤避免语义失真。在视觉问答基准上，FISA 在分布内和分布外设置中均稳定提升性能，与文本自增强兼容，数据效率优于通用图像增强，过滤策略实用有效。

## 关键点

- FISA 从模型自身失败案例生成具有视觉挑战性但答案不变的图像，实现无需外部监督的数据扩展。
- 通过自检和双保真过滤机制，确保合成样本不产生语义失真，提升数据质量。
- 在 VQA 基准上，FISA 在分布内和分布外场景均稳定提升性能，并与现有文本自增强方法兼容。
- 合成样本的数据效率优于通用图像增强基线，过滤策略在实践中被证明有效。

## 全文

Multimodal large language models (MLLMs) have achieved remarkable performance across vision-language tasks, but their progress depends heavily on large-scale, high-quality multimodal data that are costly to annotate. Self-augmentation offers a promising alternative by enabling models to expand their own training data without external supervision. However, existing MLLM self-augmentation methods are largely text-centric, while image augmentation remains underexplored and typically relies on generic or handcrafted transformations that are weakly aligned with the model's actual incapability. We propose Failure-informed Image Self-Augmentation (\textbf{FISA}), a framework for MLLM self-improvement that constructs augmented images from the model's own failure cases. Our method generates visually challenging yet answer-preserving image complications, verifies their utility through self-examination, and applies dual fidelity filtering to avoid semantic distortion. Experiments on visual question answering benchmarks show that the proposed method consistently improves performance across both in-distribution and out-of-distribution settings. Further experiments validate the compatibility of FISA with existing textual self-augmentation approaches, the superior data efficiency of the synthesized samples over generic image augmentation baselines, and the practical effectiveness of the proposed filtering strategy.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_130eb098ca64c916521ebf7d