---
title: "Server-Side Sparse Assignment for Federated Mixture-of-Experts"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2611.07093v1"
permalink: "https://feed.astralor.com/items/itm_f9caa86ae1bebded3851806e"
published: "2026-11-04T19:57:11.000Z"
score: 58
channel: "tech"
tags: ["LLM","Server-Side Sparse Assignment","Federated Mixture-of-Experts Large","LLMs","Large","However"]
---

# Server-Side Sparse Assignment for Federated Mixture-of-Experts

## At a glance

Large language models (LLMs) are increasingly recognized for their remarkable capabilities; federated learning (FL) offers a promising framework for collaboratively fine-tuning LLMs on edge devices while achieving data …

## Key points

- Large language models (LLMs) are increasingly recognized for their remarkable capabilities; federated learning (FL) offers a promising framework for collaboratively fine-tuning LLMs on edge devices while achieving data …
- However, edge devices are inherently heterogeneous: clients may exhibit diverse data distributions and resource constraints, including limited memory and computation budgets.
- Recent federated Mixture-of-Experts (MoE) approaches address these challenges by using sparse expert activation or partial expert loading.

## Full text

Large language models (LLMs) are increasingly recognized for their remarkable capabilities; federated learning (FL) offers a promising framework for collaboratively fine-tuning LLMs on edge devices while achieving data privacy. However, edge devices are inherently heterogeneous: clients may exhibit diverse data distributions and resource constraints, including limited memory and computation budgets. Recent federated Mixture-of-Experts (MoE) approaches address these challenges by using sparse expert activation or partial expert loading. However, existing partial-expert methods rely on heuristic full-pool observations or prioritize system-wide load balance, without guarantees for recovering client-specific relevant experts from partial observations. We propose $S^3$-MoE, a server-side sparse expert assignment framework that estimates client-specific expert relevance from partial observations and refines assignments through an exploit--explore mechanism. We also establish relevance-estimation error and exact Top-$K$ recovery guarantees. Experiments using a CNN-based MoE on CIFAR-100 show that $S^3$-MoE recovers task-dependent client--expert preferences and reduces conflicts caused by correlated expert selection across clients. Compared with Full-20 FedAvg, which gives each client access to all 20 experts for local Top-5 routing, $S^3$-MoE achieves 1.31 percentage points higher mean final accuracy while loading only $5$ experts per client.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_f9caa86ae1bebded3851806e