---
title: "Training-Free Universal Approximation by Prompting Random Transformers"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.09558v1"
permalink: "https://feed.astralor.com/items/itm_22d2263a0c3d3ed3080d7324"
published: "2026-08-10T12:57:22.000Z"
score: 74
channel: "tech"
tags: ["arXiv","Transformer","Prompt Learning","Kernel Regression","Universal Approximation Theory"]
---

# Training-Free Universal Approximation by Prompting Random Transformers

## AI commentary

> What makes this paper worth reading is that it turns prompting from black magic into linear algebra: random weights are not a problem if the prompt is constructed right. For prompt engineers and interpretability researchers, it offers a new lens on how much work inference-time computation can actually do.

## At a glance

This paper addresses a fundamental question: how expressive can a Transformer become via prompting alone? The authors show, in an approximation-theoretic sense, that pretraining is optional. A single-layer softmax attention network with random, untrained weights can approximate any Hölder continuous function on a compact manifold when steered by an appropriate soft prompt. Leveraging the connection between softmax attention and kernel methods, they construct explicit soft prompts as solutions to linear systems that match attention logits to Gaussian kernel exponents, causing the frozen Transformer to emulate the classical Nadaraya-Watson kernel estimator. The construction only requires a mild rank condition that holds almost surely under Gaussian initialization. The prompted network inherits the theoretical guarantees of kernel regression, yielding universal approximation theorems with minimax-optimal rates depending on the intrinsic dimension. They also quantify the cost of prompting in terms of prompt norm, prompt length, and hidden dimension. Numerical experiments corroborate the constructions and predicted rates.

## Key points

- Shows that a single-layer softmax attention network with random, untrained weights can universally approximate Hölder functions when steered by soft prompts.
- Constructs prompts as solutions to linear systems via the attention-kernel connection, making the frozen network equivalent to Nadaraya-Watson kernel regression.
- Quantifies the cost of prompting, revealing a tradeoff among prompt token norm, prompt length, and hidden dimension.

## Full text

How expressive is prompting a transformer? Answering this question is important for separating the roles of prompting, architecture, and pretraining in transformer models, and for determining whether task-specific behavior must be stored in model weights or can instead be induced at inference time through the prompt. We show, in an approximation-theoretic sense, that pretraining is optional: a single-layer softmax attention network with random, untrained weights can approximate any Hölder function on a compact manifold when steered by an appropriate soft prompt. Guided by the connection between softmax attention and kernel methods, we construct explicit soft prompts (a prompt per target function, independent of the query) as solutions to linear systems matching attention logits to Gaussian kernel exponents, under which the frozen transformer emulates the classical Nadaraya-Watson kernel estimator. The construction requires only a mild rank condition on the weights, which we show holds almost surely under Gaussian initialization. The prompted network inherits the theoretical guarantees of kernel regression, leading to universal approximation theorems with minimax-optimal rates that depend on the intrinsic dimension. We further quantify the cost of prompting, exposing a tradeoff between the norm of the constructed soft prompt tokens, prompt length, and hidden dimension. Numerical experiments corroborate the constructions and predicted rates.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_22d2263a0c3d3ed3080d7324