---
title: "Online Neural Space Time Memory: Real-Time Memory Mechanism for Dynamic Novel View Synthesis"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.15271v1"
permalink: "https://feed.astralor.com/items/itm_2064eae62ccac994bdf35ff3"
published: "2026-07-16T17:58:18.000Z"
score: 77
channel: "tech"
tags: ["Test-Time Training","Dynamic Novel View Synthesis","Real-time Rendering","Neural Space Time Memory"]
---

# Online Neural Space Time Memory: Real-Time Memory Mechanism for Dynamic Novel View Synthesis

## AI commentary

> This paper hits the pulse of real-time novel view synthesis. The idea of decoupling update frequencies is both clever and practical, especially for latency-sensitive applications. If you are working on AR/VR or dynamic scene reconstruction, this might be one of the most noteworthy memory mechanisms this year.

## At a glance

For online novel view synthesis from multi-view streaming videos, this paper proposes a framework that decouples memory update frequency from application, using periodic updates and per-frame memory application with cross-view attention for deformation, plus auxiliary Memory Loss and caching to stabilize long contexts, achieving real-time state-of-the-art performance on dynamic human scenes.

## Key points

- Proposes decoupling memory update and application frequencies to reduce computation and meet real-time constraints.
- Cross-view attention handles deformations between prior memory state and current frame.
- Introduces auxiliary Memory Loss and Memory Caching to prevent catastrophic drift over long contexts.
- Achieves real-time state-of-the-art performance on dynamic human motion and minute-scale online memorization.

## Full text

Online novel view synthesis from multi-view streaming videos faces a fundamental trade-off: maintaining a persistent, long-horizon memory to reconstruct temporarily occluded regions while operating under strict real-time constraints. While Test-Time Training (TTT) offers a powerful memory mechanism, standard models mandate gradient-based memory updates at every frame to adapt to the changing motion in dynamic scenes. The computational cost of heavy memory updates precludes real-time application and can lead to instability over long contexts. Given that memory updates are more demanding than memory application and video content is largely redundant, we propose to decouple the frequencies of these two processes. Our approach performs periodic memory updates while applying the memory on a per-frame basis, using cross-view attention to manage deformations between the prior memory state and the current frame. To lock in the historical context, we introduce two critical mechanisms: an auxiliary Memory Loss that forces persistent internalization of the scene, and a Memory Caching strategy that regularizes active weights against catastrophic drift. Our method demonstrates real-time, state-of-the-art performance on scenes with dynamic human motion as well as minute-scale online memorization.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_2064eae62ccac994bdf35ff3