Essay 01 / Clinical AI
The hardest part of clinical AI isn’t the model
Why an accurate algorithm is only the beginning of a useful clinical product.
Key takeaways
- A correct prediction isn’t the same as a useful clinical intervention.
- Who receives this prediction, when, and what can they do with it?
- The goal isn’t fewer alerts. It’s alerts that earn attention.
Imagine an AI model correctly flags a patient at risk of deterioration. The alert goes out. But the clinician who gets it doesn’t know what changed, whether someone is already responding, or what to do next. Was the AI successful?
As a model, maybe. As a product, it’s much less clear. A correct prediction isn’t the same as a useful clinical intervention.
Model performance matters. Sensitivity, specificity, AUC and the choice of operating threshold all shape how many events you catch and how many false alarms you create.[1] But they don’t tell you whether the product will work in a hospital. For that, I look at three things: whether it fits clinical workflow, whether clinicians can trust it, and whether it creates real value. These are product questions, and they need to shape the work from day one.
1. The workflow is part of the product
A prediction is only useful if the right person gets it at the right time and knows what to do with it. Should it go to the bedside nurse, the physician, or a rapid response team? Does that change with the patient’s risk level? Those decisions determine whether an accurate prediction leads to timely care or becomes one more notification.
Patients also don’t deteriorate in one uniform way. Changes in vitals, labs and organ function raise different concerns, which affects what the interface should show. A bare risk score, the clinical changes behind it, or a pointer to an established care pathway? The answer depends on what the model was validated to do and how care is actually delivered.
Timing matters too. An alert that arrives after the clinician already sees the problem adds nothing, and one without enough context gets ignored. So when I evaluate a clinical AI concept, I start with one question: who receives this prediction, when, and what can they do with it? If we can’t answer that, the product isn’t defined yet.
2. Trust requires more than accuracy
A clinician seeing a risk alert wants to know why, what changed, and where to look. In a busy unit, those answers need to come in seconds.
Show too little and they won’t trust it. Show too much and you add cognitive load at the worst moment. I prefer leading with the risk trend and the most important clinical changes. A trend shows whether risk is climbing and how fast, which a single score can’t. Supporting detail can sit one click deeper.
I’m also careful not to confuse what drives the model with what’s driving the patient. A feature that contributes to a prediction isn’t necessarily the cause of deterioration, and the product shouldn’t imply that it is.
This carries into alert fatigue. Beyond counting alerts and false positives, I’d track how many alerts lead to meaningful clinical action. An alert can be statistically correct and still useless if it’s late or tells clinicians what they already know. “Meaningful action” needs a careful definition, since sometimes the right response is to keep watching, and action alone doesn’t prove better outcomes. The goal isn’t fewer alerts. It’s alerts that earn attention.
3. Deployment is not adoption, and adoption is not impact
Getting AI live in a hospital is a big milestone, but it doesn’t mean the product is delivering value.
When adoption falls short, I don’t assume the cause. It could be trust, a confusing interface, bad timing, or a problem clinicians don’t actually care about. My first step is to talk to them. Analytics show what’s happening; conversations explain why. In healthcare especially, the workflow described in a meeting often looks very different from a real shift.
Value also looks different to different people. A clinician wants to catch deterioration earlier. A CMO cares about outcomes, a CIO about integration and ROI. Hospitals have different priorities too, whether that’s time to treatment, bed capacity or readmissions. So I start with what matters to that hospital and work backward to what the product can credibly influence and how to measure it, keeping potential benefits separate from proven results.
That’s why I see customer success as part of the product feedback loop. The teams closest to hospitals see what’s working and what isn’t, and that should feed directly into product decisions.
The real product is bigger than the algorithm
The pieces are connected. Thresholds drive alert burden, alert design drives trust, trust drives adoption, and adoption is what makes clinical impact possible. The most important product decisions usually sit between disciplines, not inside one.
A strong model is essential. But the hardest part of clinical AI isn’t predicting what might happen. It’s designing everything around that prediction so it actually improves care.
[1] Sensitivity is the share of true events a model catches; specificity is the share of non-events it correctly leaves alone. Moving the threshold trades one for the other, and AUC summarizes that tradeoff across all thresholds. Positive predictive value (PPV) is the share of alerts that are real. With 5% prevalence, 90% sensitivity and 90% specificity, a hypothetical model would generate about 45 true and 95 false alerts per 1,000 patients, a PPV of roughly 32%. Strong statistics don’t automatically mean a workable clinical workflow. These figures are illustrative only.