In News
-Sathish Raman
Medical AI Needs Safety-Critical Evaluation Rigor
A radiologist opens a chest scan and finds a second opinion already sitting in the margin, produced by a model that scored well on a benchmark nobody in the reading room has seen. The scene is no longer unusual. The U.S. Food and Drug Administration has authorized 1,451 AI-enabled medical devices, and 1,104 of them, roughly 76%, are radiology tools. Imaging is where clinical AI arrived first, because pixels are abundant and labels are cheaper there than almost anywhere else in medicine. What has not kept pace is the discipline for proving that a model works on the patients actually walking through the door. The models improved faster than the methods used to judge them.
Aneesh Gupta, a machine learning engineer working on model evaluation, computer vision, and production ML infrastructure, has watched a version of this problem play out before. As an undergraduate researcher in a university machine learning lab, he worked on semantic segmentation for medical and satellite imagery, where a model’s score on a curated dataset and its behavior on a new image can diverge sharply. He later spent more than 3 years at a leading autonomous vehicle company building the evaluation systems used to measure and gate perception models before they reached the road, and he now builds large-scale machine learning infrastructure in production at a major technology company. He has made the argument in print before, in a published article on why AI evaluation remains the critical gap in safe autonomous driving deployment. His view is that medical imaging is inheriting a measurement problem other safety-critical fields already paid for, and that the fixes transfer.
Clearance Is Not the Same Thing as Proof
Regulatory authorization tells you a device met a standard. It does not tell you the device was tested on patients. Of 521 AI and machine learning devices the FDA authorized through 2022, 43% had no published clinical validation data, and only 22 were validated through randomized controlled trials. Most reached the market through a pathway that compares a new product against an existing one rather than asking what it does in a clinic. The result is a market where accuracy gets quoted from the paper that shipped with the model, and the paper describes a dataset the hospital does not have.
Gupta’s position is that a number on a model card is a starting point, not evidence. Evaluation, done properly, is its own engineering artifact: a defined set of metrics, a dataset that reflects the population the model will actually see, and a scoring process anyone on the team can rerun and get the same answer from. In his work on perception systems, that meant deciding in advance what good performance meant in numbers, before anyone argued about whether a new model felt better than the old one. Applied to a lesion detector, the same discipline asks which scanners, which protocols, which patients, and which failure modes the score was computed over. A missing answer is a finding, not a footnote.
“A benchmark number is a claim, and a claim is only as good as the method behind it,” says Aneesh Gupta. “If I can’t tell you what data that 95% was computed on and who was in it, I haven’t told you anything useful about a patient.”
What Safety-Critical Engineering Learned the Hard Way
Fields that hurt people when they fail tend to build measurement cultures early. Aviation developed incident reporting and repeatable certification testing. Autonomous driving, after years of demo reels, moved toward defined benchmarks and crash-rate methodology that outside parties can check. The pattern is consistent. The engineering community stops arguing about whether a system feels safe and starts publishing how it was measured. Medical imaging AI is early in that arc. Model architectures circulate fast. Evaluation methodology does not.
Gupta built that kind of measurement layer for a perception stack, and the transferable parts are structural. Metric design comes first, because a metric that rewards the wrong behavior will quietly steer a year of model development in the wrong direction. Then comes a scoring pipeline stable enough that a change in the result can be attributed to the model rather than to the evaluation setup. Last comes a gate, a bar the system has to clear before it touches anything real. The analogues in imaging are direct. A missed nodule is not the same error as a false alarm on a benign one, and an evaluation that averages the two into a single accuracy figure has thrown away the information a clinician needs to act.
“Every field that got serious about safety got boring about measurement first,” Gupta explains. “That is the trade you make. You give up the exciting demo and you get a number you can defend in front of a regulator.”
Averages Hide the Patients Who Do Not Look Average
Reporting practice in medical AI is thinner than the field’s confidence suggests. Of the 168 machine learning devices the FDA authorized in 2024, 29.2% reported both sensitivity and specificity, and only 15.5% included demographic data on the populations used in testing. A model can post a strong headline figure and still fail on a scanner it has never seen, a body habitus underrepresented in training, or a presentation that shows up once in 10,000 studies. Those are the cases where a clinician most needs the tool to be right. They are also the cases an aggregate score is built to smooth away.
Gupta handled this in perception by refusing to let the average carry the argument. The evaluation was designed to hunt for the rare and dangerous case rather than report the mean, which meant slicing performance by condition, by object type, by the circumstances that made a failure expensive. In imaging, the equivalent slices are sitting right there and mostly go unused: scanner vendor, acquisition protocol, field strength, contrast timing, patient demographics, disease prevalence in the site’s own population. He wants model evaluation reported the way a trial reports subgroups, with the uncomfortable strata printed rather than folded into one number.
“The average case is the one you were never going to get wrong,” Gupta observes. “Safety lives in the tail, and if your evaluation isn’t built to go looking for it, you’ll find it in production instead.”
The Model Still Has to Run Inside a Hospital
Evaluation does not end at the paper. A deployed imaging model runs on whatever hardware the health system owns, inside a workflow with latency budgets, under a scanner fleet that gets replaced one machine at a time. Protocols change. Patient populations shift. A model that was accurate at validation drifts, quietly, and nothing in the clearance process guarantees that anyone is watching. Post-market monitoring in medical AI is closer to an aspiration than a practice.
Gupta’s current work sits on that side of the problem. He builds machine learning infrastructure in production, including inference on custom accelerator hardware, where the job is making models run correctly and efficiently across different compute platforms rather than in a research notebook. That work has made him blunt about the distance between a model that performs and a model that ships. Numerical behavior shifts across hardware. Input conditions change at runtime. Correctness has to be verified on the path the model actually executes, not the one it was benchmarked on. Clinical AI has the same gap with higher stakes, and his answer is continuous evaluation, treated as infrastructure the way monitoring is treated in any serious production system.
“A model isn’t a static object once it’s deployed, and neither is the hospital around it,” Gupta notes. “The measurement has to keep running, not fire once at the moment somebody needed an approval.”
What Clinical Trust Will Be Built On
Regulators are moving in this direction. Recent FDA guidance asks manufacturers to pre-specify how a model may change after clearance and what performance drop would trigger a new submission, which is an admission that a one-time evaluation was never enough. Professional societies are pushing for reporting standards that state plainly what a model was tested on and who it was tested with. The direction of travel is toward evidence a third party can check, and away from accuracy figures that arrive attached to a sales deck.
That shift is where Gupta wants the engineering effort to go. The tooling to build imaging models is widely available now. The tooling to prove one works on a given population, in a given hospital, on a given scanner, is not, and that asymmetry decides which systems earn their way into care. He treats evaluation as a product decision rather than paperwork filed afterward. The fields that got there before medicine did not arrive by consensus. They arrived after failures forced the question, and imaging still has a chance to skip that part.
“The models can already read the scan. What we haven’t built is the argument that they read it correctly for this patient, in this hospital, on this machine,” Gupta reflects. “That’s an engineering problem, and it’s the one I’d rather spend the next decade on than another point of benchmark accuracy.”
