---
title: "Transformer在归纳推理任务中的不变学习动力学"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.11875v1"
permalink: "https://feed.astralor.com/items/itm_20bdda93ef468db186757c28"
published: "2026-07-13T17:56:11.000Z"
score: 79
channel: "ai"
tags: ["arXiv","Transformer","学习动力学","归纳推理","注意力机制"]
---

# Transformer在归纳推理任务中的不变学习动力学

## AI 点评

> 这篇论文最吸引我的是它将复杂的Transformer训练动力学约化为低维流形，让原本黑箱的电路形成过程变得可预测、可解释。对于关注语言模型内部机制的理论研究者来说，这是一篇必读的深度作品；对于工程师而言，虽然需要一定数学基础，但其中自动检测电路的方法也很有启发价值。

## 速览

本文提出了一个理论框架，解释Transformer语言模型如何涌现归纳推理能力。不同于以往局限于特定任务的工作，本研究统一了多种合成任务（如上下文n-gram和多跳推理），并证明注意力模型的训练动力学可限制在低维不变流形上。该流形仅用少数可解释坐标即可描述学习动态，揭示了数据统计如何影响上下文学习与权重学习的竞争、随机初始化如何决定获胜电路，并可用于自动检测训练模型中学到的电路。

## 关键点

- 统一了多种归纳推理任务，提出广义任务类，并证明训练动力学可在低维不变流形上建模。
- 揭示了数据统计如何调控上下文学习与权重学习之间的竞争机制。
- 展示了随机初始化如何决定多个可能解中哪个电路最终胜出。
- 流形坐标可用于自动检测训练模型中学到的电路，提升解释性。

## 全文

We present a theoretical framework to explain the emergence of inductive reasoning abilities in Transformer language models. While previous works on Transformer learning dynamics have so far been mostly tied to specific tasks, we study a generalized class of inductive tasks that unifies several synthetic tasks known in the literature, including in-context n-grams and multi-hop reasoning. In this class, we theoretically prove that the training dynamics of attention models can be confined to a highly interpretable, low-dimensional invariant manifold. On this manifold, the learning dynamics are captured by a handful of interpretable coordinates rather than millions of parameters, making both theoretical and empirical analysis more tractable. Using this framework, we characterize how data statistics govern the competition between in-context and in-weights learning, we study how random initializations determine the `winning' circuit when multiple solutions are possible, and we demonstrate that the coordinate frame associated with the manifold can be used to automatically detect which circuits have been learned in trained models. By casting circuit formation as a low-dimensional dynamical phenomenon, we take a step toward a predictive theory of how Transformers learn.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_20bdda93ef468db186757c28