Clinical AI Explanations Can Mislead Beginners

A scan, a confidence score, and two very different doctors

Picture two people looking at the same skin lesion photo. One is a first year resident. The other is a board certified dermatologist. Both get help from an AI model: a diagnosis, a confidence score, and in some cases a plain language explanation of why the model thinks what it thinks.

The resident's accuracy goes up. So does her confidence, even in the cases where the AI is wrong. The dermatologist's accuracy barely moves. In some setups, the dermatologist actually does best with the least explanation: just a bare prediction, no reasoning attached.

That is the finding from a study out of MIT, Columbia, and Stanford, published in Nature Medicine on August 4, 2026. Researchers Orson Xu, Marzyeh Ghassemi, and Roxana Daneshjou tested several kinds of explainable AI on skin disease diagnosis: confidence scores alone, similar image comparisons, heat maps over the image, and full LLM generated explanations in plain English.

Explanation is not neutral

The core result: the same explanation helped an expert and misled a beginner. Non-experts leaned on the AI's reasoning and trusted it even when it was wrong, becoming more confident in their incorrect answers when an LLM explained itself. Clinicians did not get fooled the same way. They caught the AI's mistakes more often, and their best performance came with no explanation at all, just the raw prediction.

Ghassemi put it plainly in MIT's writeup: "Good AI systems can improve performance in some health settings, but this has to be balanced carefully with algorithmic deference that can lead to more error." The team's stated worry is automation bias: users deferring to the machine instead of using it to fill gaps in their own judgment.

Why this is a ward problem, not a lab problem

Swap "skin lesion photo" for "chest X-ray on a busy overnight shift" and the finding gets sharper edges. Hospitals are staffed by a mix of experience levels. The intern reading a scan at 2 a.m. and the attending reviewing it the next morning are not the same user, even though most AI tools ship one explanation style for both.

A tool that writes fluent, confident sounding reasoning is good for adoption metrics and bad for a trainee who cannot yet tell a plausible explanation from a correct one. The paper's fix is not "remove explanations." The researchers point to fairness constrained, tuned models that improved accuracy and reduced diagnostic disparities tied to skin tone, alongside interfaces that adapt to who is actually using them.

The design lesson

This is the argument for building clinical AI as something a doctor can check, not something a doctor defers to. An explanation that cannot be independently verified against the chart, the labs, the actual patient in the bed, is exactly the kind of explanation that flattered the non-experts in this study into being more wrong, more confidently.

At Medows we think about this every time we design how the workspace shows its reasoning to a resident on rounds. Fluent is easy. Checkable is the harder, more useful thing to build for a doctor's shift.

Sources

• MIT News: The benefits of medical AI assistance vary based on user expertise
• Nature Medicine paper (DOI 10.1038/s41591-026-04553-w)