---
title: "VISA：智能体自进化数据合成框架，提升多模态指令跟随能力"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.26013v1"
permalink: "https://feed.astralor.com/items/itm_bde570cf5e7919d0118b635e"
published: "2026-08-26T16:55:58.000Z"
score: 72
channel: "ai"
tags: ["arXiv","VISA","多模态指令跟随","强化学习","数据合成","智能体框架"]
---

# VISA：智能体自进化数据合成框架，提升多模态指令跟随能力

## AI 点评

> VISA的价值在于把数据合成从“一次性工具”变成“自适应系统”，让模型能盯着自己的短板反复锤炼，这对训练更聪明的多模态助手很有启发。做数据流水线或RL训练的人值得细读这篇，尤其是验证信号当奖励的设计。

## 速览

多模态指令跟随模型需要准确、多样、可验证且具有挑战性的训练数据，但现有合成流程多为一次性生成-过滤，忽视了失败样本和验证反馈。VISA提出了一种智能体化自进化循环，每轮对图像进行分析、筛选约束、采样约束集、生成指令并用工具和结构化大语言模型验证。失败样本触发诊断恢复，通过样本经目标模型探测估计难度，验证信号与失败模式回写记忆，指导后续约束空间扩展和难度聚焦。实验表明VISA在MM-IFEval上持续优于强基线，并在七个公开基准上保持通用多模态能力。

## 关键点

- VISA将多模态指令合成建模为自进化循环，动态利用失败样本、验证结果和模型错误反馈。
- 通过可执行工具和结构化大语言模型评估器验证候选指令，失败样本触发诊断引导的恢复机制。
- 验证信号和目标模型失败档案回写记忆，为后续迭代提供约束空间扩展和难度聚焦依据。
- 验证器协议可直接为强化学习提供奖励信号，无需额外训练奖励模型。

## 全文

Multimodal instruction-following models require training data that is accurate, diverse, verifiable, and challenging. Existing synthesis pipelines typically follow a one-pass generate-and-filter paradigm, discarding feedback from failed samples, verifier outcomes, and target-model errors. We present VISA (Visual Instruction Synthesis Agent), an agentic framework that reformulates multimodal instruction synthesis as a self-evolving loop. At each round, VISA analyzes an image to filter incompatible constraints and discover new verifiable ones, samples diversity- and difficulty-aware constraint sets from persistent memory, generates candidate instructions, and verifies the resulting samples with executable tools and structured large language model judges. Failed samples trigger diagnostic-guided recovery, while accepted samples are probed against the target model to estimate difficulty. The resulting verifier signals and target-model failure profiles are written back to memory, allowing subsequent rounds to adaptively expand the constraint space, reduce template repetition, and focus on unresolved model weaknesses. The same verifier contracts further provide reward signals for reinforcement learning without a separately trained reward model. Experiments on MM-IFEval show that VISA consistently improves multimodal instruction following over strong baselines, while preserving general multimodal capability across seven public benchmarks.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_bde570cf5e7919d0118b635e