---
title: "Words Before Pixels: Selective Modality Routing for Vision-Language Model Unlearning"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2611.00608v1"
permalink: "https://feed.astralor.com/items/itm_88ffdf1cd609a962d93c3d6b"
published: "2026-10-31T04:09:27.000Z"
score: 58
channel: "ai"
tags: ["推理效率","多模态","Words Before Pixels","Selective Modality Routing","Vision-Language Model Unlearning","Vision-language","VLM","Existing"]
---

# Words Before Pixels: Selective Modality Routing for Vision-Language Model Unlearning

## At a glance

Vision-language model (VLM) unlearning is often treated as an objective-design problem: given multimodal forget data, the goal is to optimize a loss that removes undesirable behavior while preserving retained capabiliti…

## Key points

- Vision-language model (VLM) unlearning is often treated as an objective-design problem: given multimodal forget data, the goal is to optimize a loss that removes undesirable behavior while preserving retained capabiliti…
- We argue that this view overlooks an equally important data-centric question: how should each forget example be represented before optimization?
- Existing methods typically keep image-conditioned failures in their original image-text form, implicitly assuming that pixels are the right route for unlearning whenever pixels appear in the input.

## Full text

Vision-language model (VLM) unlearning is often treated as an objective-design problem: given multimodal forget data, the goal is to optimize a loss that removes undesirable behavior while preserving retained capabilities. We argue that this view overlooks an equally important data-centric question: how should each forget example be represented before optimization? Existing methods typically keep image-conditioned failures in their original image-text form, implicitly assuming that pixels are the right route for unlearning whenever pixels appear in the input. We challenge this assumption by constructing modality-decomposed versions of VLM benchmarks, enabling controlled comparisons among image-only, text-only, and multimodal representations of the same examples. Our modality-mixing experiments show that unsafe instruction-following behavior is more effectively suppressed through textual representations than through images alone, while purely text-only unlearning can still weaken visual grounding when the target behavior depends on visual evidence. Motivated by this trade-off, we introduce ModRoute, a gradient-guided per-sample modality routing strategy that selectively converts high visual-pressure examples to text-only form while keeping the remaining examples multimodal. Notably, on VLGuard with LLaVA-1.5-7B, ModRoute has the strongest composite unlearning-utility score of 0.847 at $γ=0.5$, a 2.5% improvement over the best random-switching score and $\geq$46% better than pure multimodal or text unlearning. Overall, our results show that effective VLM unlearning depends not only on the objective, but also on the composition and per-sample modality routing of the unlearning data.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_88ffdf1cd609a962d93c3d6b