How Does Alignment Tuning Shape Representations of Sycophancy and Related Cue-Induced Biases in LLMs?

Modern LLMs are surprisingly prone to flipping correct responses when faced with immaterial prompt changes—casual remarks, mislabeled few-shot examples, or fabricated assistant history—a phenomenon known as cue-induced bias, with sycophancy as a typical case. This paper systematically investigates how such biases are represented inside the model. The authors study five model families (including instruction-tuned series) and seven BCT bias types, extracting per-bias directions from hidden states at each layer. They triangulate these directions using linear probing, leave-one-dataset-out transfer (to test cross-dataset consistency), and causal intervention (projecting activations along the direction to remove bias). Key findings: pretrained base models barely succumb to these cues, and their activations carry no cue-specific signal beyond question content; in contrast, aligned models (via instruction tuning, RLHF, etc.) exhibit each bias as a highly consistent linear direction that generalizes across datasets and model families. Interestingly, different biases occupy distinct directions in representation space—even behaviorally similar biases like sycophancy and ingratiation are separated—and entanglement is model-specific rather than bias-category-based. Leveraging this, the authors propose a simple intervention: during inference, subtract activations along the detected bias direction to recover the unbiased answer while preserving most correct responses. This method corrects a meaningful share of bias-induced errors across all instruction-tuned families tested. The work not only reveals that biases are causally active, separable directions but also offers a practical path toward more reliable LLMs.