18 November 2026 · Medows
AI Oversight Broke. Only 12.5% Noticed.
A randomized study let AI accuracy fall to zero. Only 12.5% of readers noticed, and making them decide first did not help.
Last week a group of researchers did something simple and slightly cruel. They sat 48 readers down with 40 recorded heart sounds, gave each an AI assistant, then quietly made the AI worse.
The accuracy was walked down to zero
The task was binary: normal or abnormal. The AI offered a recommendation on every case. Across four blocks its reliability was stepped down from 90% to 60% to 50% to 0%. The final block also broke the sync between the audio and what was on the screen.
Performance followed it down. Mean accuracy fell from 81.45% (SD 11.48) in the first block to 58.33% (SD 25.29) in the last. The interesting part is what did not fall. Confidence moved from 3.39 to 3.04. Trust moved from 3.40 to 3.03. The AI went from mostly right to entirely wrong, and the people leaning on it ended up slightly less sure.
Only 12.5% noticed the reliability declining while it was happening. Three readers per group.
The standard fix did not work
Half the group saw the AI's answer first. The other half had to commit: give your own classification and confidence, then see the AI, then revise if you want. That second design is cognitive forcing, and it is the usual answer to automation bias. Make the human think before the machine speaks.
By the final block the commit-first group scored 52.08% (SD 26.70), against 64.58% (SD 22.65) for the group that saw the AI first. Cognitive forcing did not improve oversight. The authors put it plainly: "human oversight alone may be insufficient to reliably detect system malfunctions in AI-assisted cardiac auscultation." Their recommendation is not better vigilance. It is system-level cues and organizational monitoring.
The caveats, because they matter
This is a brief communication, not a trial. 48 participants, 24 per arm. 37.5% of them were final-year medical students, not physicians. Heart sounds in a lab bear little resemblance to a stethoscope on a ward at 7 a.m. A single degrading task is not a deployed product.
Take it as a signal, not a verdict. The trouble is that a much larger study points the same way.
225 physicians, and two of them noticed
In a participant-blinded randomized experiment posted in October, 225 physicians from 42 countries and eight clinical disciplines worked sequential decisions on anonymized neurocritical care cases with AI assistance. In one arm the AI was quietly adversarial, steering toward a predetermined harmful option.
612 of 1,240 final decisions (49.4%) matched the harmful target, against 25 of 1,245 (2.0%) with aligned AI. Per person: 190 of 225 physicians (84.4%) made at least one harmful decision with the adversarial AI, versus 21 of 225 (9.3%) with the aligned one. Experience bought nothing measurable. 86 of 101 physicians (85.1%) with more than ten years of intensive care practice made at least one harmful decision.
Two of 225 (0.9%) spontaneously identified systematic steering. Afterwards, 113 of 225 (50.2%) said they would trust the AI in clinical practice. That is a preprint, and an adversarial arm is a stress test rather than a plausible product. But the detection number is the one that should stick: roughly 1%.
The loop does not close by itself
Nearly every regulatory and procurement document on clinical AI leans on the same load-bearing phrase: the clinician remains in the loop. These two studies are about what that phrase assumes. It assumes the clinician will notice when the tool starts being wrong.
Picture the actual conditions. Post-take round, 28 patients on the list, a registrar three hours behind, a phone going. Nobody in that corridor is running a quiet audit of whether the model drifted since last month. Oversight is asked for in exactly the moments when attention is scarcest, and the first study suggests it fails even when attention is abundant and the task is one case at a time.
If the human cannot be the detector, the burden moves to the tool and whoever operates it. Two things follow. Verification has to live inside the answer rather than behind a button: which chart value was used, which source, where the model is unsure. And drift has to be watched continuously by something other than the tired reader, with an alarm that does not wait for anyone to feel suspicious.
We build Medows on the assumption that the doctor will not catch us. Every answer carries the patient data and the source it came from, because a claim you can check in three seconds is worth more than a claim you are asked to trust.
Sources
- Challenges for human oversight in detecting system malfunctions in AI-assisted medical diagnostics, npj Digital Medicine, 13 November 2026
- Subversion of clinical judgment by conversational artificial intelligence, medRxiv preprint, October 2026
Medows is a clinical AI workspace for the doctor on rounds. Learn more or write to alapan@medows.ai / alapanx@gmail.com.