General Quantification of Covariate and Concept Shifts
Measuring how machine learning models fail when data changes unexpectedly
When machine learning models trained on one dataset face new, different data in the real world, they often fail — but predicting exactly how much worse they'll perform has proven theoretically elusive. This paper fixes the broken mathematical definitions used to measure these failures and introduces a new method that actually works across different types of problems, allowing researchers to estimate performance drops before deployment.
Machine learning systems deployed in hospitals, cars, and financial systems encounter shifted data constantly — loan applicants look different than training examples, disease patterns evolve, weather patterns change. This work provides a practical tool to measure and predict accuracy loss in advance, helping engineers decide whether a model is safe to deploy or needs retraining before real-world consequences occur.