---
title: "Right Diagnoses, Decorative Reasoning: A Perturbation Audit of Medical Chain-of-Thought"
source: "arXiv AI/CL/LG（论文）"
source_tier: "t2"
url: "https://arxiv.org/abs/2608.24790v1"
permalink: "https://feed.astralor.com/items/itm_64b771ff9237edf1bcebd8dc"
published: "2026-08-25T16:37:39.000Z"
score: 78
channel: "ai"
tags: ["Medical AI","Interpretability","Large Language Models","Chain-of-Thought"]
---

# Right Diagnoses, Decorative Reasoning: A Perturbation Audit of Medical Chain-of-Thought

## AI commentary

> What makes this paper worth reading is how it uses simple yet rigorous experiments to puncture an industry illusion: model-generated rationales may be beautiful documentation, not real reasoning. For medical AI product managers, clinical decision support developers, and regulatory auditors, CDR is a metric worth adopting for routine fidelity checks.

## At a glance

Clinicians often read chain-of-thought (CoT) rationales as evidence of medical reasoning, but this paper tests whether that assumption holds. Using a 30-operator perturbation battery that edits both the chain and the question with clinically motivated operators—severity reversal, negation flip, demographic swap, and evidence ablation—the authors audit 14 large language models across four medical QA benchmarks. Three independent tests converge: the Chain-Decoupling Rate (CDR), measuring cases where the chain does not register the edit and the answer does not flip, is 72.9% on clinically meaningful destructive edits; chain corruption leaves accuracy unchanged; and removing CoT prompting does not reduce accuracy. Two board-certified clinicians re-annotated 197 perturbed questions, finding 98.5% of gold answers defensible. The pattern persists across medical and reasoning fine-tuning and scale; closed-source models show answer-side signals consistent with the same decoupling. The framework and CDR offer a reusable yardstick for auditing whether medical CoT is faithful or merely documentation.

## Key points

- Introduces a 30-operator perturbation battery editing both chain and question, paired with a chain-update times answer-flip joint analysis.
- Across 14 LLMs and 4 medical QA benchmarks, average CDR is 72.9%; chain corruption leaves accuracy unchanged and removing CoT does not reduce performance.
- Two board-certified clinicians re-annotated 197 perturbed questions; 98.5% of gold answers remain defensible.
- Suggests medical CoT may be decorative rather than faithful; CDR offers a reusable yardstick for faithfulness auditing.

## Full text

Clinicians read chain-of-thought (CoT) rationales as evidence of medical reasoning, but whether the visible chain plays that role is rarely tested. General-domain CoT-faithfulness probes ignore clinical cost, and medical LLM evaluations treat the chain as a black box. We close this gap with a medical perturbation audit: a 30-operator battery edits both the chain and the question with clinically motivated operators (severity reversal, negation flip, demographic swap, evidence ablation), paired with a chain-update times answer-flip joint analysis that classifies each model by its failure mode. Applied to 14 LLMs on four medical QA benchmarks, three independent tests converge: the Chain-Decoupling Rate (CDR; chain does not register the edit and the answer does not flip) is 72.9% panel-wide on clinically meaningful destructive edits, chain corruption leaves accuracy unchanged, and removing CoT prompting does not reduce accuracy. Two board-certified clinicians re-annotate N=197 perturbed questions; 98.5% leave the gold defensible. The pattern holds across medical and reasoning fine-tuning and scale; on the closed-source tier, where the chain text is unavailable, the answer-side signals are consistent with the same decoupling. Our framework and CDR provide a reusable yardstick for auditing whether medical CoT is faithful or merely documentation.

---

Verify against the original before citing; curated by Astral Feed: https://feed.astralor.com/items/itm_64b771ff9237edf1bcebd8dc