Article #4 established that data governance is a clinical safety issue. Article #5 showed how an aggregate performance number can hide a subgroup gap nobody tested for.
This article closes out Phase 2 with the reality that ties both together: even a model that was fairly and thoroughly validated at launch does not stay static once it's deployed.
The question was never whether your model will drift. It's whether anyone will notice when it does.
(Governing the Algorithm, Article #6)
Drift Isn't a Failure of the Model. It's a Property of Deployment.
A model is trained and validated against a snapshot of the world: a specific patient population, a specific set of imaging equipment, a specific era of clinical practice. The real world doesn't hold still after that snapshot is taken.
Equipment gets upgraded. Patient populations shift. Clinical protocols evolve. Disease prevalence changes. None of this is a malfunction. It's simply what happens to any model deployed into a live clinical environment over time. Drift is not a sign something went wrong with the model. It's an expected consequence of time passing.
Where This Becomes a Governance Failure
Drift itself isn't negligence. The absence of a system to detect it is.
"The model performed well in our validation study."
That statement can be true on day one and quietly stop being true eighteen months later, with nobody aware the gap has opened, because nothing was in place to notice.
This is the distinction that matters:
✔ Expected drift, monitored: performance is tracked against live data, and a defined threshold triggers review before the gap becomes clinically meaningful
✔ Expected drift, unmonitored: the same gap opens, silently, until it's discovered by a clinician noticing something feels off, or worse, after a preventable miss
The model in both scenarios might degrade at exactly the same rate. Only one of these companies is governing it.
What Real Drift Monitoring Requires
- A defined performance baseline
established at validation, specific enough to be measured against, not just a general accuracy claim
- Ongoing measurement against live production data
, not just periodic manual spot-checks
- A defined threshold that triggers action
: a specific, pre-agreed point at which "performance has changed" becomes "someone needs to review this now"
- A documented response plan
(retraining, retesting, restricting use, or notifying deployment sites) so a detected drift event has a defined next step rather than an ad hoc scramble
Very few companies have all four of these built before their first deployment. Most build the first one (a baseline) and treat the rest as a future problem.
Why "We'll Deal With It If It Comes Up" Doesn't Hold
Drift monitoring is exactly the kind of governance work that's easy to defer, because its absence is invisible right up until it isn't. A model that hasn't been actively monitored for eighteen months doesn't announce that it's drifted: it just quietly performs worse, in ways that are hard to distinguish from normal clinical variability until someone specifically looks for the pattern.
By the time drift is discovered informally, through a clinician's suspicion or a cluster of unexpected outcomes, the company is no longer managing a known, monitored risk. It's explaining an unmonitored one, after the fact.
Final Thought
Every deployed model will drift, given enough time in a changing clinical environment. That part is unavoidable.
What is avoidable is not knowing.
Drift is a fact of deployment.
Undetected drift is a choice not to look.
Building the monitoring infrastructure to catch drift before it becomes clinically significant isn't an advanced governance feature. It's the baseline expectation for any AI product that stays in use longer than its original validation study.
Next in the Governing the Algorithm Series:
Human-in-the-Loop Is a Design Requirement, Not a Marketing Line
#HealthcareAI #ModelDrift #AIGovernance #SaMD #RiskManagement #Compliance #DigitalHealth #HealthcareInnovation #ArtificialIntelligence #MedTech #QscriptionTechnologies