Training & Test Set Independence
GMLP Principle 4. Real validation requires real independence: across patients, sites, and acquisition methods, not just across patient IDs.
Overview
Naive train/test splits leak signal through site effects, acquisition-method correlations, and patient ID reuse across timepoints. The result is validation that overstates performance, and a deployed product that disappoints.
We design splits that respect every dependency dimension and document the rationale so regulators and customers can verify the validation is real.
Our Process
-
1
Dependency inventory
Patient, site, acquisition device, acquisition method, time.
-
2
Split design
Stratified across dependency dimensions.
-
3
Leakage audit
Verify no leak across the dependency dimensions.
-
4
Sensitivity testing
Performance under alternate splits.
-
5
Documentation
Audit-ready rationale for the chosen split.
Frequently Asked Questions
Site-stratified splits common?
Yes, and increasingly expected by regulators.
Time-aware splits for longitudinal data?
Yes, leakage through time is a common failure mode.
Can we recover from a leaky validation post hoc?
Sometimes, with rigorous re-validation; sometimes the model needs retraining.
How do we know we caught all dependencies?
Disciplined dependency inventory and external review.
Validate on data the model has not really seen.
Send us your current train/test setup. We will return an independence audit within three weeks.
Start a Conversation