---
title: "VISA: Agentic Self-Evolving Data Synthesis for Multimodal Instruction Following"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.26013v1"
permalink: "https://feed.astralor.com/items/itm_bde570cf5e7919d0118b635e"
published: "2026-08-26T16:55:58.000Z"
score: 72
channel: "ai"
tags: ["arXiv","VISA","Multimodal Instruction Following","Reinforcement Learning","Data Synthesis","Agentic Framework"]
---

# VISA: Agentic Self-Evolving Data Synthesis for Multimodal Instruction Following

## AI commentary

> VISA's value lies in turning data synthesis from a one-shot tool into an adaptive system that lets a model repeatedly refine its own weak spots, which is inspiring for building smarter multimodal assistants. Those working on data pipelines or RL training should read it closely, especially the design of using verifier signals as rewards.

## At a glance

Multimodal instruction-following models require training data that is accurate, diverse, verifiable, and challenging, but existing synthesis pipelines usually follow a one-pass generate-and-filter paradigm, discarding feedback from failures. VISA proposes an agentic self-evolving loop: each round analyzes an image, filters incompatible constraints, samples diversity- and difficulty-aware constraint sets, generates candidate instructions, and verifies them with executable tools and structured LLM judges. Failed samples trigger diagnostic-guided recovery, accepted samples are probed against the target model for difficulty, and verifier signals plus failure profiles are written back to memory to adaptively expand the constraint space and focus on weaknesses. Experiments on MM-IFEval show VISA consistently improves multimodal instruction following over strong baselines while preserving general multimodal capability across seven public benchmarks.

## Key points

- VISA formulates multimodal instruction synthesis as a self-evolving loop, dynamically leveraging failed samples, verifier outcomes, and target-model errors.
- Candidate instructions are verified with executable tools and structured LLM judges; failed samples trigger diagnostic-guided recovery.
- Verifier signals and target-model failure profiles are written back to memory to adaptively expand constraint space and focus on unresolved weaknesses.
- The verifier contracts provide reward signals for reinforcement learning without a separately trained reward model.

## Full text

Multimodal instruction-following models require training data that is accurate, diverse, verifiable, and challenging. Existing synthesis pipelines typically follow a one-pass generate-and-filter paradigm, discarding feedback from failed samples, verifier outcomes, and target-model errors. We present VISA (Visual Instruction Synthesis Agent), an agentic framework that reformulates multimodal instruction synthesis as a self-evolving loop. At each round, VISA analyzes an image to filter incompatible constraints and discover new verifiable ones, samples diversity- and difficulty-aware constraint sets from persistent memory, generates candidate instructions, and verifies the resulting samples with executable tools and structured large language model judges. Failed samples trigger diagnostic-guided recovery, while accepted samples are probed against the target model to estimate difficulty. The resulting verifier signals and target-model failure profiles are written back to memory, allowing subsequent rounds to adaptively expand the constraint space, reduce template repetition, and focus on unresolved model weaknesses. The same verifier contracts further provide reward signals for reinforcement learning without a separately trained reward model. Experiments on MM-IFEval show that VISA consistently improves multimodal instruction following over strong baselines, while preserving general multimodal capability across seven public benchmarks.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_bde570cf5e7919d0118b635e