30 October 2026 · Medows
Microsoft's AI Beat Doctors. On Simulated Cases.
Microsoft's diagnostic AI beat physicians in a Science study, but on simulated cases. What the numbers show, and what they can't.
Last week Microsoft published a study in Science with a headline number built to travel: its AI diagnostic system beat physicians. Read the fine print before you repeat it on rounds.
What was reported
According to ICT&health's coverage of the Science paper (published October 22), Microsoft's AI Diagnostic Orchestrator correctly diagnosed about 83% of complex cases drawn from the New England Journal of Medicine. Physicians reached 36% with access to textbooks and online resources, and 22% without.
On emergency department cases, the AI reached about 80% accuracy. Physicians reached 47%. The ED set is 488 cases of patients who were later admitted, shared publicly through PhysioNet.
How the test worked
The system does not get a full chart. It starts with a limited case description, then decides which questions to ask and which tests to order. Several simulated clinical roles propose diagnoses, challenge them, and judge whether another test is worth it. The benchmark also tracks the cost of the tests ordered.
That design is clever. It is closer to real reasoning than a multiple-choice exam.
What the article itself says about the limits
The same coverage is plain about this. The comparisons "involved simulated diagnostic scenarios rather than direct patient care." When information is missing from a record, the framework can generate synthetic responses. Simulated scenarios cannot fully capture communication problems, incomplete histories, or unexpected clinical developments.
The technology, the article notes, has yet to show effectiveness in routine clinical practice. A prospective study with Beth Israel Deaconess Medical Center is planned, with patients interacting with the system directly. It has not happened yet.
And one line is worth underlining: diagnostic accuracy alone is insufficient to establish clinical value. Safety, unnecessary testing, system integration, and clinical responsibility all matter too.
The scene the benchmark skips
Picture the real version of that ED case. The history comes from a relative, in pieces, partly wrong. The old records are in another system. A nurse flags something at the bedside that never makes it into a note. The doctor is also carrying twenty-seven other patients.
A benchmark can reward the right final diagnosis. It cannot easily measure whether the tool fits that chaos, or who answers for the result.
Why this matters for the ward
We read this as a useful signal, not a verdict. Strong results on structured cases tell us the reasoning engines are improving fast. They do not tell a doctor how to trust an output at 6 a.m. on a crowded round.
That gap is the real product problem. A doctor needs to see where an answer came from, check it against the patient in front of them, and keep the decision. Accuracy numbers on simulated cases do not supply that. Workflow, context, and verifiability do.
So when the next "AI beats doctors" headline arrives, ask three questions. Were the patients real? Who supplied the missing data? What happens to the doctor's accountability? Until a prospective study answers them, treat the number as promising and unproven.
This post is commentary on published reporting, not clinical guidance.
Sources
Medows is a clinical AI workspace for the doctor on rounds. Learn more or write to alapan@medows.ai / alapanx@gmail.com.