---
title: "The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.18237v1"
permalink: "https://feed.astralor.com/items/itm_1f9c39a456e16b8109e11e27"
published: "2026-07-20T17:59:51.000Z"
score: 79
channel: "ai"
tags: ["TPIPS","Text-guided Retrieval","Visual Similarity","Vision-Language Model"]
---

# The Many Senses of Visual Similarity: A Text-Prompted Image Perceptual Metric

## AI commentary

> I believe this paper addresses a key pain point in visual perceptual metrics—the lack of semantic conditioning. TPIPS allows users to specify comparison dimensions via natural language, offering much more flexibility than fixed metrics. For researchers in image retrieval and generative model evaluation, this could become a practical new tool. The open-sourced resources also lower the barrier to adoption, making it worth paying attention to.

## At a glance

Human visual similarity judgments are highly context-dependent—two images may be similar in shape but distinct in color, yet existing metrics collapse these nuances into a single scalar without conditioning on specific aspects. This work introduces a large-scale dataset of human similarity judgments over image triplets, each annotated across multiple free-form semantic aspects. Benchmarking frontier Vision-Language Models (VLMs) reveals a substantial gap compared to human consensus. By fine-tuning a VLM on this data, the authors propose the Text-Prompted Image Perceptual Similarity (TPIPS) metric, which captures multiple senses of visual similarity based on a text prompt. TPIPS aligns better with human perception and generalizes reliably beyond the training distribution. It unlocks new capabilities in text-guided retrieval, compositional search, and fine-grained evaluation of generative models.

## Key points

- Constructed a large-scale dataset of human similarity judgments over image triplets, each annotated with multiple free-form semantic aspects.
- Benchmarked frontier Vision-Language Models (VLMs) and identified a significant gap compared to human consensus.
- Fine-tuned a VLM to produce the TPIPS metric, which captures multiple senses of visual similarity conditioned on a text prompt.
- TPIPS excels in text-guided retrieval, compositional search, and fine-grained evaluation of generative models.

## Full text

Human visual similarity judgments are context-dependent. For example, two images may be similar in shape but distinct in color. Existing perceptual similarity metrics, however, collapse these nuances into a single scalar value, offering no mechanism to condition on specific aspects. To bridge this gap, we introduce a large-scale dataset of human similarity judgments over image triplets, where each triplet is annotated across multiple, free-form semantic aspects of similarity. Benchmarking a broad range of frontier vision-language models (VLMs) reveals a considerable performance gap compared to human annotators' consensus. Leveraging our data, we fine-tune a VLM to produce our Text-Prompted Image Perceptual Similarity (TPIPS) metric, capturing multiple senses of visual similarity depending on the specified text prompt. We demonstrate that TPIPS aligns more closely with human perception and generalizes reliably beyond the training distribution. Finally, we show that TPIPS unlocks new capabilities in text-guided retrieval, compositional search, and the fine-grained evaluation of generative models. Our code, data, and trained models are at https://peterwang512.github.io/TPIPS

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_1f9c39a456e16b8109e11e27