---
title: "提示随机 Transformer 可实现免训练的通用逼近"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.09558v1"
permalink: "https://feed.astralor.com/items/itm_22d2263a0c3d3ed3080d7324"
published: "2026-08-10T12:57:22.000Z"
score: 74
channel: "tech"
tags: ["arXiv","Transformer","提示学习","核回归","通用逼近理论"]
---

# 提示随机 Transformer 可实现免训练的通用逼近

## AI 点评

> 这篇论文最值得读的地方是把“提示”从玄学变成了线性代数：随机权重并不可怕，只要提示构造得当，模型照样能干活。对做提示工程或者研究模型可解释性的人来说，它提供了一个新视角来理解推理期计算到底能承担多少任务。

## 速览

本文从逼近论角度回答了一个关键问题：提示（prompting）到底能赋予 Transformer 多强的表达能力？研究证明，预训练并非必需——一个单层 softmax 注意力网络，即使权重随机且完全未训练，只要辅以合适的软提示，也能在紧流形上逼近任意 Hölder 连续函数。作者利用 softmax 注意力与核方法之间的联系，将软提示构造为线性系统的解，使冻结的 Transformer 等价于 Nadaraya-Watson 核回归器。该构造在温和的秩条件下几乎必然成立，并继承了核回归的理论保证，达到与内在维度相关的 minimax 最优收敛率。数值实验验证了构造和预测速率。

## 关键点

- 证明随机权重、未训练的单层 softmax 注意力网络在软提示驱动下可普遍逼近 Hölder 函数。
- 利用注意力与核方法的映射，将提示构造成线性系统的解，使冻结网络等价于 Nadaraya-Watson 核回归。
- 给出提示代价的权衡关系：提示 token 范数、提示长度与隐藏维度之间存在折衷。

## 全文

How expressive is prompting a transformer? Answering this question is important for separating the roles of prompting, architecture, and pretraining in transformer models, and for determining whether task-specific behavior must be stored in model weights or can instead be induced at inference time through the prompt. We show, in an approximation-theoretic sense, that pretraining is optional: a single-layer softmax attention network with random, untrained weights can approximate any Hölder function on a compact manifold when steered by an appropriate soft prompt. Guided by the connection between softmax attention and kernel methods, we construct explicit soft prompts (a prompt per target function, independent of the query) as solutions to linear systems matching attention logits to Gaussian kernel exponents, under which the frozen transformer emulates the classical Nadaraya-Watson kernel estimator. The construction requires only a mild rank condition on the weights, which we show holds almost surely under Gaussian initialization. The prompted network inherits the theoretical guarantees of kernel regression, leading to universal approximation theorems with minimax-optimal rates that depend on the intrinsic dimension. We further quantify the cost of prompting, exposing a tradeoff between the norm of the constructed soft prompt tokens, prompt length, and hidden dimension. Numerical experiments corroborate the constructions and predicted rates.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_22d2263a0c3d3ed3080d7324