Skill Scores Forecast Models Including AIGS and Graph Cast

Discussion: Four forecast models are considered to offer a broad view of forecast skill during the past 60 days for the northern hemisphere, North America, and Europe. The four processed considered are two physics-based models: ECM ENS and GFS plus two AI models: AI Graph Cast ECM ENS and AIFS. ECM ENS is selected due to the consistent highest skill score while GFS is reviewed due to high visibility. The AI Graph Cast ECM ENS is selected due to consistently highest AI model skill scores while AIFS became operational at ECMWF yesterday. The forecast parameter that is evaluated is the 500 MB height forecasts, essentially predicting the weather pattern.

Skill scores for the northern hemisphere 500 MB forecasts during the past 60 days are consistently the best by ECM ENS in both the 6-10-day and 11-15-day periods (Fig. 1). The AI Graph Cast ECM ENS is nearly as skillful. Although ranking 3^rd, AIFS is competitive in the 6-10-day period while forecast skill eases back in the 11-15-day period. As usual, the volatile nature of GFS causes the lowest skill score of the 4 forecast processes reviewed. The results presented are repeated when focused on North America and Europe (Fig. 2-3). Overall, skill scores are slightly better for North America versus Europe.

At hour 360, forecasts of 500 MB heights representing the prevailing weather pattern are usually quite low. Skill scores average near or slightly above 0.40 on a 0 to 1 scale by the ECM ENS for the northern hemisphere, North America, and Europe (Fig. 4). AI Graph Cast ECM ENS was quite changeable showing the best skill of all models at hour-360 during the past 60 days for North America (0.52) while at the same time skill score for Europe was about 50% lower (0.26). AIFS averaged 3^rd best and near 0.33 for each sector. GFS lagged way back…just above 0.20 skill for each sector.

Summary: A review of 500 MB height forecast skill scores, provided by CWG/Storm Vista Weather Models, to determine the best model at predicting the weather pattern in the 6-10-day period, 11-15-day period, and at hour-360 during the past 60 days for the northern hemisphere, North America, and Europe renders consistent results. ECM ENS is the best model followed closely by AI Graph Cast ECM ENS. AIFS became operational at ECMEWF yesterday. The model has been adjusted for its operational arrival. During the past 60 days, AIFS ranked 3^rd among the models evaluated. The GFS is a highly visible/available model and therefore the 4^th forecast process chosen for evaluation. The volatile GFS consistently scored last in skill score.

Fig. 1: The 60-day skill scores (zero to 1) of 500 MB forecasts (the weather pattern) across the northern hemisphere for the 6-10-day/11-15-day medium range.

Fig. 2: The 60-day skill scores (zero to 1) of 500 MB forecasts (the weather pattern) across North America for the 6-10-day/11-15-day medium range.

Fig. 3: The 60-day skill scores (zero to 1) of 500 MB forecasts (the weather pattern) across Europe for the 6-10-day/11-15-day medium range.

Fig. 4: The 60-day skill scores (zero to 1) of 500 MB forecasts at hour-360 for the northern hemisphere, North America, and Europe.