---
title: "First-Token Attraction in Mamba Dynamics"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2611.08303v1"
permalink: "https://feed.astralor.com/items/itm_08ba5e43aa7a97948296a5d2"
published: "2026-11-05T17:06:17.000Z"
score: 59
channel: "tech"
tags: ["First-Token Attraction","Mamba Dynamics","Mamba","Unlike"]
---

# First-Token Attraction in Mamba Dynamics

## 速览

We study token dynamics in deep Mamba models, with a particular focus on the attracting role of the first token. Unlike attention mechanisms, where token interactions are governed by positive scalar coefficients, Mamba …

## 关键点

- We study token dynamics in deep Mamba models, with a particular focus on the attracting role of the first token.
- Unlike attention mechanisms, where token interactions are governed by positive scalar coefficients, Mamba dynamics involve full matrix-valued interaction coefficients with no positivity or definiteness guarantees.
- This structural difference makes Mamba dynamics substantially harder to analyze and prevents the direct application of existing techniques for attention-based models.

## 全文

We study token dynamics in deep Mamba models, with a particular focus on the attracting role of the first token. Unlike attention mechanisms, where token interactions are governed by positive scalar coefficients, Mamba dynamics involve full matrix-valued interaction coefficients with no positivity or definiteness guarantees. This structural difference makes Mamba dynamics substantially harder to analyze and prevents the direct application of existing techniques for attention-based models. To overcome this challenge, we introduce a new analytical framework based on a high-dimensional invariant box and a sharp exit-time argument, which together recover an effective form of sign-definiteness. Using this framework, we characterize the possible limit points of token trajectories and prove explicit exponential convergence rates. Our analysis reveals that the first token has privileged dynamics: its limiting direction attracts subsequent tokens, inducing token clustering and attention-sink phenomena in deep Mamba models. We further show that this attraction mechanism extends to broader sequence dynamics, including state-space models, causal attention, and classical time-series models. Experiments on Falcon-Mamba-7B validate our predictions, showing that concentration on the first token increases with depth and dominates in later layers.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_08ba5e43aa7a97948296a5d2