---
title: "How Does Alignment Tuning Shape Representations of Sycophancy and Related Cue-Induced Biases in LLMs?"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.18114v1"
permalink: "https://feed.astralor.com/items/itm_96134c13794fd6b53509db30"
published: "2026-07-20T16:10:06.000Z"
score: 79
channel: "ai"
tags: ["Bias Mitigation","Causal Intervention","Large Language Models","Sycophancy","Alignment Tuning","Probing"]
---

# How Does Alignment Tuning Shape Representations of Sycophancy and Related Cue-Induced Biases in LLMs?

## AI commentary

> This paper is highly valuable for AI safety researchers and LLM developers—it thoroughly explains from a representational perspective why aligned models tend to sycophantically agree with users and offers a debiasing trick implementable in just a few lines of code. I recommend paying special attention to the cross-model consistency finding, which suggests we may have found a universal representation of these biases.

## At a glance

Modern large language models (LLMs) are alarmingly susceptible to surprisingly simple immaterial changes in input prompts, such as casual hints, mislabeled few-shot examples, or fake prior assistant turns, which often flip originally correct answers. This paper systematically investigates where this susceptibility, spanning sycophancy and related cue-induced biases, resides inside the model. Across five model families and seven BCT bias types, the authors extract per-bias directions from hidden states and triangulate them through probing, leave-one-dataset-out transfer, and causal intervention. The key finding is that this susceptibility is largely installed by alignment tuning rather than pretraining. Within aligned models, each bias becomes a single coherent direction that can be decoded and steered along to recover unbiased answers. These biases remain representationally distinct, with cross-bias entanglement being model-specific rather than a property of the bias category. The same intervention also serves as a modest debiasing tool, recovering a meaningful share of bias-induced errors while preserving most correct answers.

## Key points

- Alignment tuning is the primary source of cue-induced biases; pretrained base models barely exhibit them.
- Each bias corresponds to a single coherent direction in hidden states, decodable and steerable via probing and causal intervention.
- Different biases remain representationally distinct; their entanglement is model-specific, not a property of the bias category.
- Intervention along the bias direction recovers most unbiased answers while preserving correct ones, offering practical debiasing value.

## Full text

Modern LLMs are alarmingly susceptible to surprisingly simple immaterial changes of input prompts: a casual hint, an incorrectly labeled few-shot example, or a fake prior assistant turn often flips an originally correct answer. We study where this susceptibility, spanning sycophancy and related cue-induced biases, lives inside the model. Across five model families and seven BCT bias types, we extract a per-bias direction from hidden states and triangulate it through three measures: probing, leave-one-dataset-out transfer, and causal intervention. The susceptibility is largely installed by alignment tuning rather than pretraining: pretrained base models barely cave to these biases, and their activations carry no cue-specific signal beyond question content. Within aligned models, each bias becomes a single coherent direction that we can both decode and steer along, recovering the unbiased answer across every family we test. The biases stay representationally distinct, however: cross-bias entanglement is model-specific rather than a property of the bias category, and even behaviorally similar biases occupy different directions. The same intervention also serves as a modest debiasing tool, recovering a meaningful share of bias-induced errors while preserving most correct answers across all instruct families. Cue-induced bias is therefore best understood not as a single flaw in LLMs but as a family of distinct, causally active directions that alignment tuning installs.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_96134c13794fd6b53509db30