---
title: "Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.11875v1"
permalink: "https://feed.astralor.com/items/itm_20bdda93ef468db186757c28"
published: "2026-07-13T17:56:11.000Z"
score: 79
channel: "ai"
tags: ["arXiv","Transformer","Learning Dynamics","Inductive Reasoning","Attention Mechanism"]
---

# Invariant Learning Dynamics of Transformers in Inductive Reasoning Tasks

## AI commentary

> What fascinates me most about this paper is how it reduces complex Transformer training dynamics to a low-dimensional manifold, making the previously black-box circuit formation process predictable and interpretable. For theoretical researchers focusing on the internal mechanisms of language models, this is a must-read deep work; for engineers, although a certain mathematical background is required, the method for automatically detecting circuits is also highly insightful.

## At a glance

This paper presents a theoretical framework explaining how inductive reasoning abilities emerge in Transformer language models. Unlike previous work tied to specific tasks, this study unifies several synthetic tasks (e.g., in-context n-grams and multi-hop reasoning) and proves that training dynamics of attention models can be confined to a low-dimensional invariant manifold. On this manifold, learning dynamics are captured by a handful of interpretable coordinates, revealing how data statistics govern competition between in-context and in-weights learning, how random initializations determine winning circuits, and enabling automatic detection of learned circuits in trained models.

## Key points

- Unifies multiple inductive reasoning tasks into a generalized class and proves training dynamics lie on a low-dimensional invariant manifold.
- Reveals how data statistics govern the competition between in-context learning and in-weights learning.
- Shows how random initializations determine which circuit wins among multiple possible solutions.
- Manifold coordinates enable automatic detection of learned circuits in trained models, improving interpretability.

## Full text

We present a theoretical framework to explain the emergence of inductive reasoning abilities in Transformer language models. While previous works on Transformer learning dynamics have so far been mostly tied to specific tasks, we study a generalized class of inductive tasks that unifies several synthetic tasks known in the literature, including in-context n-grams and multi-hop reasoning. In this class, we theoretically prove that the training dynamics of attention models can be confined to a highly interpretable, low-dimensional invariant manifold. On this manifold, the learning dynamics are captured by a handful of interpretable coordinates rather than millions of parameters, making both theoretical and empirical analysis more tractable. Using this framework, we characterize how data statistics govern the competition between in-context and in-weights learning, we study how random initializations determine the `winning' circuit when multiple solutions are possible, and we demonstrate that the coordinate frame associated with the manifold can be used to automatically detect which circuits have been learned in trained models. By casting circuit formation as a low-dimensional dynamical phenomenon, we take a step toward a predictive theory of how Transformers learn.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_20bdda93ef468db186757c28