---
title: "SceneBind: Binding What and Where Across Vision, Audio and Language"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.15265v1"
permalink: "https://feed.astralor.com/items/itm_ee266a401e4c43b32a69b72a"
published: "2026-07-16T17:55:15.000Z"
score: 77
channel: "ai"
tags: ["3D Spatial Understanding","SceneBind","Multimodal Representation","Cross-modal Retrieval","Zero-Shot Transfer"]
---

# SceneBind: Binding What and Where Across Vision, Audio and Language

## AI commentary

> SceneBind takes a significant step in multimodal representation by unifying 'what' and 'where', enabling machines to not only recognize objects but also understand spatial relationships. This paper is particularly valuable for developers working on multimodal retrieval, embodied intelligence, and scene understanding—its lightweight design also facilitates practical deployment. I look forward to seeing its application on real robotic platforms.

## At a glance

SceneBind introduces an omni-modal representation that captures both global semantics and 3D spatial structure of real-world scenes. Each scene is represented as a semantic-spatial entity combining a global semantic embedding with object-centric semantic-spatial slots, explicitly modeling object attributes, spatial attributes, and uncertainty. A semantic-spatial matching scheme enables cross-modal scene retrieval and object grounding. The method trains on a newly curated binaural audio-visual dataset, adds lightweight spatial tokens to pretrained encoders, achieves SOTA on scene and spatial retrieval, and demonstrates strong zero-shot transfer to tasks like audio-visual localization.

## Key points

- Proposes SceneBind, an omni-modal representation fusing semantics and 3D spatial info.
- Each scene is represented by a global semantic embedding and object-centric semantic-spatial slots.
- Curates a new binaural audio-visual dataset for training and evaluation.
- Achieves SOTA on scene/spatial retrieval and zero-shot transfer to audio-visual localization.

## Full text

We present SceneBind, an omni-modal representation of realistic scenes with joint semantic and 3D spatial understanding across vision, audio and language. Existing omni-modal encoders excel at instance-level semantics (i.e., what is present), but often lack explicit spatial structure (i.e., where it is). SceneBind addresses this gap by representing each scene as a semantic-spatial entity, combining a global semantic embedding with object-centric semantic-spatial slots. This representation explicitly captures object-level semantics, spatial attributes, and uncertainty. We further propose SceneBind Matching, a semantic-spatial matching scheme that integrates global scene similarity with object alignment, supporting cross-modal scene retrieval and object grounding. To train and evaluate SceneBind, we curate a novel real-world binaural audio-visual dataset with structured semantic and spatial annotations, and propose a training protocol for aligning semantic and spatial signals across modalities. SceneBind is compatible with large-scale pretrained semantic encoders, adds lightweight spatial modeling with only a few additional tokens. It achieves state-of-the-art scene and spatial retrieval while enabling strong zero-shot transfer to downstream tasks such as audio-visual localization.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_ee266a401e4c43b32a69b72a