---
title: "Understanding and Mitigating Role Spoofing in LLMs"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2610.40396v1"
permalink: "https://feed.astralor.com/items/itm_a146c907fb01e12346c24785"
published: "2026-10-29T16:33:08.000Z"
score: 55
channel: "ai"
tags: ["LLM","AI Agent","RAG","Understanding","Mitigating Role Spoofing","LLMs Despite","LLMs","Despite"]
---

# Understanding and Mitigating Role Spoofing in LLMs

## 速览

Despite growing capabilities, LLMs remain susceptible to prompt injections: malicious inputs that can manipulate or hijack models. In particular, it has recently been observed that LLMs can exhibit role confusion (Ye et…

## 关键点

- Despite growing capabilities, LLMs remain susceptible to prompt injections: malicious inputs that can manipulate or hijack models.
- In particular, it has recently been observed that LLMs can exhibit role confusion (Ye et al., 2026), where models attribute the source of text based on its style instead of its labeled role.
- Malicious text in tool outputs (for instance, a malicious webpage accessed by an agent) can leverage role confusion to impersonate the user, an attack we call role spoofing}.

## 全文

Despite growing capabilities, LLMs remain susceptible to prompt injections: malicious inputs that can manipulate or hijack models. In particular, it has recently been observed that LLMs can exhibit role confusion (Ye et al., 2026), where models attribute the source of text based on its style instead of its labeled role. Malicious text in tool outputs (for instance, a malicious webpage accessed by an agent) can leverage role confusion to impersonate the user, an attack we call role spoofing}. In this work, we seek to understand and mitigate role spoofing. We study how models track "who said what" in a conversation, showing that they use cues from both content (e.g., semantics and writing style) and structure (e.g., control tokens and formatting). We show that LLMs are highly susceptible to role spoofing attacks that combine both fake structure and content. Additionally, in a toy setting, we show that adversarial training which teaches the model to ignore content-based injections does mitigate content-based attacks but leaves the model susceptible to structure-based attacks. This can explain why frontier models, with increased resistance to many injection attacks, can still be susceptible to role spoofing. Finally, we propose a new method for detecting role spoofing using linear probes trained to predict roles in a conversation, and show that it is effective across different settings.

---

引用前请核对原文;本文由星际流动整理:https://feed.astralor.com/items/itm_a146c907fb01e12346c24785