---
title: "Perturb, Repair, Verify: Self-Play Vision-Language Verifiers for Compositional Understanding"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2611.05158v1"
permalink: "https://feed.astralor.com/items/itm_7f4666313b7d4a1b7bcb04c0"
published: "2026-11-03T16:31:34.000Z"
score: 59
channel: "tech"
tags: ["多模态","Perturb","Repair","Verify","Self-Play Vision-Language Verifiers","Compositional Understanding Vision-language","Vision-language","Such"]
---

# Perturb, Repair, Verify: Self-Play Vision-Language Verifiers for Compositional Understanding

## 速览

Vision-language verifiers judge whether an image faithfully matches a textual description, yet existing training pipelines typically rely on static supervision from stronger external judges or curated hard-negative data…

## 关键点

- Vision-language verifiers judge whether an image faithfully matches a textual description, yet existing training pipelines typically rely on static supervision from stronger external judges or curated hard-negative data.
- Such supervision requires external generation or annotation, is teacher-bounded, and is difficult to adapt as model failures evolve.
- We introduce PROVE (Perturb-and-Repair Optimization for Vision-Language Evaluation), a self-play framework that trains verifiers through a closed perturb-verify-repair-verify loop.

## 全文

Vision-language verifiers judge whether an image faithfully matches a textual description, yet existing training pipelines typically rely on static supervision from stronger external judges or curated hard-negative data. Such supervision requires external generation or annotation, is teacher-bounded, and is difficult to adapt as model failures evolve. We introduce PROVE (Perturb-and-Repair Optimization for Vision-Language Evaluation), a self-play framework that trains verifiers through a closed perturb-verify-repair-verify loop. Starting from a known-correct image-caption pair, a Mutator introduces a typed perturbation over objects, attributes, counts, or relations. The verifier then judges the perturbed caption and provides textual feedback, which a Repairer uses to recover a faithful caption for re-verification. These two verification rounds produce structured rewards for informative mismatches and visually grounded repairs. Since perturbation, verification, and repair are role-conditioned behaviors of the same evolving multimodal backbone, updating the shared parameters also improves the verifier itself. Experiments show that PROVE improves fine-grained compositional verification across model scales and families and strengthens visual-detail robustness on the reference backbone, with limited, benchmark-dependent changes on broader multimodal tasks.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_7f4666313b7d4a1b7bcb04c0