Can We Trust Evaluations of AI Weather Forecasts for Extremes? How Evaluation Protocols Shape Model Rankings
How we measure AI weather forecasts changes which model wins
When researchers compare AI weather models to traditional physics-based models for predicting extreme events, the winner depends almost entirely on how they measure success. AI models tend to underpredict how often extremes occur, while physics-based models overpredict them—and different scoring methods penalize these opposite mistakes in different ways, making it impossible to draw firm conclusions from conflicting studies.
AI weather models are already being used operationally because they're faster and cheaper than traditional models, but we still don't reliably know if they're better at forecasting dangerous extreme weather. Without agreeing on standard evaluation methods, operational forecasters and emergency planners can't confidently compare models or know which one to trust when lives depend on the forecast.