---
title: "KAISEN: Reproducible Subgroup Fairness Auditing for Clinical Risk Models"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2607.28608v1"
permalink: "https://feed.astralor.com/items/itm_6059c4ea9e78541a23705ac1"
published: "2026-07-30T17:57:18.000Z"
score: 73
channel: "ai"
tags: ["CUSUM","Equalized Odds","KAISEN","Clinical Risk Models","Fairness Auditing"]
---

# KAISEN: Reproducible Subgroup Fairness Auditing for Clinical Risk Models

## AI commentary

> What makes this paper worth reading is that it puts the audit tools themselves under the microscope—finding that seemingly reasonable audit components can fail silently at critical moments. For any team responsibly evaluating clinical AI, this belongs on the mandatory reading list. Its emphasis on reporting variance rather than averages is a much-needed wake-up call for current fairness reporting practice.

## At a glance

KAISEN is a reproducible subgroup fairness auditing pipeline for clinical risk models, covering subgroup stratification, disparity measurement, mechanism diagnostics, post-hoc mitigation, and drift monitoring. Using a synthetic benchmark of 16 disease tasks, 15 social-determinant axes from Healthy People 2030, and three prespecified intersections, it finds: significance counts track raw equalized-odds difference only after standardization by minimum detectable effect; per-group threshold optimization reduces EOD in all 48 held-out runs, while Platt scaling behaves like a coin flip; the mechanism diagnostic fails silently under proxy misspecification; and CUSUM drift detection thresholds do not transfer across cohorts. The audit should report variance, not averages.

## Key points

- KAISEN is a five-phase audit pipeline stress-tested to failure on a synthetic benchmark;
- Per-group threshold optimization reduces EOD in 48/48 held-out runs, while Platt scaling improves only 19/48;
- The mechanism diagnostic classifies 144/144 controlled cases but recovers zero model-driven cases under proxy misspecification, with no failure signal;
- CUSUM drift detection at a reference threshold yields 27 false alarms and 7/8 missed shifts across cohorts, showing thresholds do not transfer.

## Full text

Clinical risk models routinely achieve strong aggregate performance while producing materially different error rates across patient subgroups. Audit pipelines have been proposed to catch this, but their components are rarely stress-tested, so it is unclear which parts of an audit can be trusted and under what conditions. We present KAISEN, a five-phase audit pipeline covering subgroup stratification, disparity measurement, mechanism diagnostics, post-hoc mitigation, and drift monitoring, evaluated to the point of failure on a synthetic benchmark of 16 disease tasks, 15 social-determinant axes from Healthy People 2030, and three prespecified intersections. Four findings follow. (i) Significance tracks each axis's gap against its own minimum detectable effect: rank correlation between significance count and raw equalized-odds difference (EOD) across the 15 axes is rho = 0.56, rising to rho = 0.78 once EOD is standardized by that floor. (ii) Per-group threshold optimization reduces EOD in 48 of 48 held-out runs (paired delta = -0.285, 95% CI [-0.313, -0.252]), while group-wise Platt scaling -- the better calibrator -- behaves as a coin flip on EOD (19 of 48 runs improved, 95% CI [0.26, 0.55]) with mean effect near zero, so what an audit should report is the variance, not the average. (iii) The mechanism diagnostic classifies 144 of 144 controlled cases correctly but recovers none of 48 model-driven cases under proxy misspecification, with no signal that it failed. (iv) CUSUM failures and false alarms track cohort realization far more than disease: at the reference threshold, all 27 false alarms and 7 of 8 missed shifts come from different seeds (chi-squared p = 0.002), so a threshold tuned on one cohort fails to transfer. All results are synthetic with known ground truth and do not establish clinical validity. Code, artifacts, and scripts reproducing every number are released.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_6059c4ea9e78541a23705ac1