---
title: "When can Optimally Trained Transformers Achieve Near-Lossless Token Prediction for non-Markovian Data?"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2610.41038v1"
permalink: "https://feed.astralor.com/items/itm_3fe4619978b0fa5d2c8c8249"
published: "2026-10-29T22:29:59.000Z"
score: 55
channel: "ai"
tags: ["When","Optimally Trained Transformers","Achieve Near-Lossless Token","Prediction","Markovian Data","There","Transformer","Markovian"]
---

# When can Optimally Trained Transformers Achieve Near-Lossless Token Prediction for non-Markovian Data?

## 速览

There exists little mathematical theory explaining why Transformer architectures are empirically successful even for non-Markovian data. To this end, this paper studies next-token prediction via stochastic optimal contr…

## 关键点

- There exists little mathematical theory explaining why Transformer architectures are empirically successful even for non-Markovian data.
- To this end, this paper studies next-token prediction via stochastic optimal control theoretically trained Transformers for such sources, where we consider either stationary and ergodic non-Markovian sources or those wh…
- In either case, we first obtain error bounds via finite models with finite window approximations.

## 全文

There exists little mathematical theory explaining why Transformer architectures are empirically successful even for non-Markovian data. To this end, this paper studies next-token prediction via stochastic optimal control theoretically trained Transformers for such sources, where we consider either stationary and ergodic non-Markovian sources or those which are hidden Markov models of some finite order (possibly non-stationary). In either case, we first obtain error bounds via finite models with finite window approximations. For stationary ergodic sources, the resulting error is bounded by an entropy gap; for $N$th order Markovian sources, it decays at an exponential rate $α^{\lfloor M/N\rfloor}$, where $M$ is the window length and $α$ is determined by filter stability (and the Dobrushin coefficients of the transition and measurement kernels). We then construct a Transformer of depth $M+1$, one head per layer, realizing the truncated filter, and we optimally train it with stochastic control developed in earlier work, which rigorously returns a near optimal Transformer. In particular, we show that for a non-Markovian source, either under stationarity and ergodicity for general models or under filter stability for hidden Markov models, an optimally trained Transformer has arbitrarily small performance error with increasing context length, data, and approximation refinement. Numerical studies accompany the theory.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_3fe4619978b0fa5d2c8c8249