---
title: "SceneBind：融合视觉、音频与语言的语义-空间全模态场景表示"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.15265v1"
permalink: "https://feed.astralor.com/items/itm_ee266a401e4c43b32a69b72a"
published: "2026-07-16T17:55:15.000Z"
score: 77
channel: "ai"
tags: ["3D空间理解","SceneBind","多模态表示","跨模态检索","零样本迁移"]
---

# SceneBind：融合视觉、音频与语言的语义-空间全模态场景表示

## AI 点评

> SceneBind 在多模态表示方向迈出了重要一步，将“是什么”与“在哪里”统一建模，让机器不仅能识别物体还能理解空间关系。这篇论文对从事多模态检索、具身智能和场景理解的开发者尤其值得关注，其轻量级设计也便于实际部署。我不禁期待后续能在真实机器人平台上看到它的应用效果。

## 速览

SceneBind 提出一种全模态场景表示方法，同时捕获场景的全局语义和3D空间结构。通过将每个场景表示为语义-空间实体（全局语义嵌入 + 物体级语义-空间槽），显式建模物体属性、位置及不确定性。配合语义-空间匹配方案，支持跨模态场景检索与物体定位。作者构建了新的双耳音视频数据集用于训练与评估，仅需少量额外 token 即可实现强空间建模，在场景与空间检索上达到 SOTA，并零样本迁移到音视频定位等下游任务。

## 关键点

- 提出SceneBind全模态表示，融合语义与3D空间信息。
- 每个场景由全局语义嵌入和物体级语义-空间槽组成。
- 新构建双耳音视频数据集，支持训练与评估。
- 在场景/空间检索上达到SOTA，零样本迁移至音视频定位。

## 全文

We present SceneBind, an omni-modal representation of realistic scenes with joint semantic and 3D spatial understanding across vision, audio and language. Existing omni-modal encoders excel at instance-level semantics (i.e., what is present), but often lack explicit spatial structure (i.e., where it is). SceneBind addresses this gap by representing each scene as a semantic-spatial entity, combining a global semantic embedding with object-centric semantic-spatial slots. This representation explicitly captures object-level semantics, spatial attributes, and uncertainty. We further propose SceneBind Matching, a semantic-spatial matching scheme that integrates global scene similarity with object alignment, supporting cross-modal scene retrieval and object grounding. To train and evaluate SceneBind, we curate a novel real-world binaural audio-visual dataset with structured semantic and spatial annotations, and propose a training protocol for aligning semantic and spatial signals across modalities. SceneBind is compatible with large-scale pretrained semantic encoders, adds lightweight spatial modeling with only a few additional tokens. It achieves state-of-the-art scene and spatial retrieval while enabling strong zero-shot transfer to downstream tasks such as audio-visual localization.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_ee266a401e4c43b32a69b72a