---
title: "VoiceMem: Streaming Dual-Brain Memory Architecture Paves the Way for Real-Time Speech Interaction"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.26005v1"
permalink: "https://feed.astralor.com/items/itm_cb239a52103a3521857a3364"
published: "2026-08-26T16:50:05.000Z"
score: 78
channel: "ai"
tags: ["arXiv","VoiceMem","Duplex SLM","Real-Time Speech Interaction","Affective Computing","Memory Architecture"]
---

# VoiceMem: Streaming Dual-Brain Memory Architecture Paves the Way for Real-Time Speech Interaction

## AI commentary

> What stands out most is how the authors split memory into 'informational' and 'emotional' channels while still achieving real-time latency. For engineers building voice assistants or agent memory layers, this dual-brain architecture and pluggable design offer useful references. The 134ms figure proves memory retrieval is no longer a bottleneck for speech interaction, freeing product teams to design more ambitious experiences.

## At a glance

A new arXiv paper introduces VoiceMem, a streaming memory architecture designed for duplex speech language models (SLMs). It features a parallel 'left brain' for informational memory and a 'right brain' for emotional memory, with streaming memory I/O mechanisms, along with a complete pipeline for memory-aware SLM training, long-horizon evaluation, and decoupled deployment. Experiments show top-5 retrieval accuracy nearly 30 points higher than the classical Mem0 at top-200; the right brain achieves SOTA on three persona benchmarks, improving the aggregate score by 4.29 points; retrieval latency is only 134ms, well within VAD limits, adding no conversational delay. VoiceMem provides a practical memory foundation for real-time, personalized, and emotionally aware speech interaction.

## Key points

- VoiceMem adopts a dual-brain architecture: the left brain handles informational memory, the right brain handles emotional memory, enabling parallel streaming processing.
- Retrieval performance is significant: top-5 retrieval outperforms Mem0's top-200 by nearly 30 points, with only 134ms latency.
- Emotional and personal modeling: the right brain combines short- and long-horizon affective attribution with dual-node persona modeling, achieving SOTA on three benchmarks.
- Supports interchangeable memory backends for decoupled deployment, providing a complete open pipeline from training to evaluation.

## Full text

Conversational systems, such as duplex speech language models (SLMs), still lack a streaming, accurate, and empathetic memory system as their soul. We introduce VoiceMem, a simple memory architecture with a parallel informational left brain, an emotional right brain, and streaming memory I/O mechanisms. We further build a complete pipeline for memory-aware SLM training, long-horizon evaluation, and decoupled deployment with interchangeable memory backends. Experiments and real-world deployment show three advantages: i) Accuracy: under top-5 retrieval, the left brain outperforms classical systems such as Mem0 at top-200 by nearly 30 points; ii) Emotional & Personal: the right brain, with short- and long-horizon affective attribution and dual-node persona modeling, achieves state-of-the-art performance across three persona benchmarks and improves the aggregate score by 4.29 points over the previous best system; and iii) Real-Time & Cheap: VoiceMem completes retrieval in 134 ms, well within standard VAD latency, adding no extra conversational delay while maintaining high accuracy and low cost. These results show that VoiceMem provides a practical memory foundation for real-time, personalized, and emotionally aware speech interaction.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_cb239a52103a3521857a3364