Cosine-Similarity Routing with Semantic Anchors for Interpretable Mixture-of-Experts Language Models

评分 5.5 · 来源：cs.CL updates on arXiv.org · 发布于 2026-04-17

评分依据：Interpretable MoE routing via cosine similarity to semantic anchors, every routing decision becomes traceable

arXiv:2509.14255v2 Announce Type: replace Abstract: Mixture-of-Experts (MoE) models improve efficiency through sparse activation, but their learned gating functions provide limited insight into routing decisions. This work introduces the Semantic Resonance Architecture (SRA), which routes tokens to experts via cosine similarity between token representations and learnable semantic anchors, making every routing decision directly traceable to anchor-token similarity scores. We evaluate SRA on WikiText-103 across 17 configurations.