---
title: "分层声学-语义建模：实现全双工语音语言模型的模态分离与语义连贯"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.06540v1"
permalink: "https://feed.astralor.com/items/itm_57c72e603bb45ba12c89394c"
published: "2026-07-07T17:43:36.000Z"
score: 76
channel: "tech"
tags: ["Lychee-FD","全双工语音语言模型","梯度冲突","模态分离","语义对齐"]
---

# 分层声学-语义建模：实现全双工语音语言模型的模态分离与语义连贯

## AI 点评

> 这篇论文终于捅破了全双工语音模型模态干扰的窗户纸，以前大家只是笼统知道效果不好，现在有了明确的梯度冲突归因。对做语音交互和端到端对话系统的工程师来说，分层参数分离的思路很值得借鉴，尤其是能在不增加延迟的前提下提升流畅度，落地价值很高。

## 速览

该论文通过细粒度分析揭示全双工语音语言模型中模态干扰的根本原因——声学与语义建模在深层参数共享时产生梯度冲突。提出Lychee-FD框架，采用分层参数分离策略解耦冲突模态，并通过语义对齐通道保持跨模态连贯性。实验表明，在Spoken QA上提升7.4%，在全双工交互流畅度上提升28.5%，且不影响推理效率。这是首次揭示全双工SLM模态干扰根源并给出优雅解决方案的工作。

## 关键点

- 发现全双工SLM中模态干扰的根源在于声学与语义建模的梯度冲突。
- 提出Lychee-FD框架，采用分层参数分离+语义对齐通道。
- 在Spoken QA和全双工交互基准上取得显著提升。
- 首次实现高性能原生全双工SLM且不牺牲推理效率。

## 全文

Developing seamless, high-performance, native intelligent full-duplex Spoken Language Models (SLMs) remains a critical challenge and long-standing goal for the speech and NLP community. Despite notable progress, recent endeavors are fundamentally constrained by severe modality interference, which causes substantial knowledge degradation and compromises semantic integrity -- ultimately making full-duplex SLMs feel unnatural and unintelligent. In this paper, through an exhaustive fine-grained analysis of model optimization dynamics, we uncover the root cause of such performance degradation, revealing that modality interference arises from inherent gradient conflicts between acoustic and semantic modeling when the two modalities are forced to share a deep parameter space. Guided by this key insight, we introduce Lychee-FD, a native end-to-end full-duplex framework designed to mitigate modality interference. Importantly, we propose a hierarchical parameter separation strategy that decouples conflicting modalities in deep layers while preserving cross-modality coherence via a dedicated semantic alignment channel. Extensive experiments on multiple full-duplex benchmarks demonstrate that our method significantly advances the state of the art, yielding substantial improvements in both speech intelligence (+7.4% on Spoken QA) and full-duplex interaction fluidity (+28.5% on FullDuplexBench 1.5) without compromising inference efficiency. To the best of our knowledge, this work is the first to achieve two key advances: 1) uncovering and elucidating the root cause of modality interference in full-duplex SLMs, and 2) designing an elegant hierarchical model together with a practical solution for seamless, high-performance, native intelligent full-duplex SLMs.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_57c72e603bb45ba12c89394c