Prediction-Powered Smoothing and Validation for Disaggregated AI Evaluation
Getting accurate AI scores when you can't test everything
When AI systems perform differently across different types of tasks or conversations, testing them thoroughly is expensive. This paper introduces a method called prediction-powered smoothing that borrows information from well-tested domains to get better estimates in domains with few test results, while also providing a new way to validate which estimation approach works best. The technique improved accuracy and maintained reliable confidence intervals across real benchmarks and deployed AI systems.
AI developers need to know how well their systems perform on specific task types and user groups before deployment, but labeling enough examples in every category is costly. This method lets evaluators get accurate performance estimates while testing fewer examples—reducing the time and expense of AI evaluation without sacrificing confidence in the results. The validation tool also helps teams choose the right estimation approach automatically, rather than guessing or holding out expensive test data.