---
title: "Hierarchical Acoustic-Semantic Modeling: Modality Separation and Semantic Coherence for Full-Duplex SLMs"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.06540v1"
permalink: "https://feed.astralor.com/items/itm_57c72e603bb45ba12c89394c"
published: "2026-07-07T17:43:36.000Z"
score: 76
channel: "tech"
tags: ["Lychee-FD","Full-Duplex SLM","Gradient Conflict","Modality Separation","Semantic Alignment"]
---

# Hierarchical Acoustic-Semantic Modeling: Modality Separation and Semantic Coherence for Full-Duplex SLMs

## AI commentary

> This paper finally reveals the nature of modality interference in full-duplex speech models—previously we only knew it didn't work well, now we have a clear attribution to gradient conflicts. For engineers working on speech interaction and end-to-end dialogue systems, the hierarchical parameter separation approach is especially valuable as it improves fluency without adding latency, making it highly practical for deployment.

## At a glance

This paper reveals the root cause of modality interference in full-duplex Spoken Language Models (SLMs) through fine-grained analysis: gradient conflicts arising from shared deep parameter spaces between acoustic and semantic modeling. It proposes Lychee-FD, a framework with hierarchical parameter separation to decouple conflicting modalities and a semantic alignment channel for cross-modality coherence. Experiments show +7.4% on Spoken QA and +28.5% on FullDuplexBench 1.5 without compromising inference efficiency. This is the first work to uncover and solve the modality interference root cause for full-duplex SLMs.

## Key points

- Identifies gradient conflicts between acoustic and semantic modeling as the root cause of modality interference in full-duplex SLMs.
- Proposes Lychee-FD framework with hierarchical parameter separation and semantic alignment channel.
- Achieves significant gains on Spoken QA (+7.4%) and full-duplex interaction fluency (+28.5%).
- First work to achieve high-performance native full-duplex SLM without compromising inference efficiency.

## Full text

Developing seamless, high-performance, native intelligent full-duplex Spoken Language Models (SLMs) remains a critical challenge and long-standing goal for the speech and NLP community. Despite notable progress, recent endeavors are fundamentally constrained by severe modality interference, which causes substantial knowledge degradation and compromises semantic integrity -- ultimately making full-duplex SLMs feel unnatural and unintelligent. In this paper, through an exhaustive fine-grained analysis of model optimization dynamics, we uncover the root cause of such performance degradation, revealing that modality interference arises from inherent gradient conflicts between acoustic and semantic modeling when the two modalities are forced to share a deep parameter space. Guided by this key insight, we introduce Lychee-FD, a native end-to-end full-duplex framework designed to mitigate modality interference. Importantly, we propose a hierarchical parameter separation strategy that decouples conflicting modalities in deep layers while preserving cross-modality coherence via a dedicated semantic alignment channel. Extensive experiments on multiple full-duplex benchmarks demonstrate that our method significantly advances the state of the art, yielding substantial improvements in both speech intelligence (+7.4% on Spoken QA) and full-duplex interaction fluidity (+28.5% on FullDuplexBench 1.5) without compromising inference efficiency. To the best of our knowledge, this work is the first to achieve two key advances: 1) uncovering and elucidating the root cause of modality interference in full-duplex SLMs, and 2) designing an elegant hierarchical model together with a practical solution for seamless, high-performance, native intelligent full-duplex SLMs.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_57c72e603bb45ba12c89394c