---
title: "Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge Bias"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.11871v1"
permalink: "https://feed.astralor.com/items/itm_1bc5cce454463fdaa0f42583"
published: "2026-07-13T17:55:19.000Z"
score: 79
channel: "tech"
tags: ["LLM-as-Judge","Bias","Causal Intervention","Mechanistic Interpretability","Activation Geometry"]
---

# Inside the Unfair Judge: A Mechanistic Interpretability Account of LLM-as-Judge Bias

## AI commentary

> What truly impressed me about this paper is that it reframes bias from 'noise' to 'geometric structure', thereby simultaneously gaining understanding and control. For engineers seeking to improve LLM evaluation reliability, the linear projection prediction method is almost directly applicable; for mechanistic interpretability researchers, it demonstrates how an activation subspace can be simultaneously descriptive, causal, and predictive. Worth reading.

## At a glance

This paper proposes that scoring biases in LLM-as-judge are not only present at the input-output level but can also be explained at the representation level in the hidden state. Across 7 judges, 7 bias types, and 9 benchmarks, they find that biased inputs are displaced along a low-dimensional subspace in activation space; causal steering along this subspace bidirectionally modulates scores; and a simple linear projection onto the subspace predicts judge failures on unseen benchmarks, substantially outperforming text-based methods.

## Key points

- Biased inputs are displaced along a type-specific low-dimensional subspace in activation space that sharpens with depth.
- Causal steering along the bias subspace bidirectionally modulates scores, with effects significantly larger than random directions.
- Linear projection onto bias-direction features predicts judge failures on three entirely unseen benchmarks, outperforming text-based alternatives.
- Reading bias as activation geometry unifies geometric structure, causal control, and operational prediction within a single framework.

## Full text

Existing studies of LLM-as-judge scoring bias work predominantly at the input-output level: they perturb inputs, measure score deltas, and propose prompt-level mitigations. We argue that the same biases admit a representation-level account in the judge's hidden state, complementary to the input-output view and operationally useful in ways it does not afford. We report three findings, across seven judges, seven bias types, and nine benchmarks. Geometry: baseline judging inputs occupy a tight activation manifold while biased inputs are displaced along a low-dimensional, type-specific subspace that sharpens with depth and is recovered consistently by three families of estimators. Causal control: steering hidden states along this subspace drives scoring in both directions, forward shifts reproducing biased scoring on clean inputs and reverse shifts restoring baseline scoring on biased ones, while matched-norm random directions produce shifts an order of magnitude smaller. Operational: a simple linear projection onto the same bias-direction features anticipates judge failures on three entirely unseen benchmarks, substantially outperforming text-based alternatives. Reading bias as activation geometry, rather than as input-output noise, unifies geometric structure, causal control, and operational prediction within a single framework. The project page is available at https://xzx34.github.io/unfair-judge/

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_1bc5cce454463fdaa0f42583