Systematic Global Evaluation of Seasonal Climate Forecast Skill for Monthly Precipitation of JMA/MRI-CPS2 Compared with a Statistical Forecast System Using Climate Indices

Abstract

This study aimed to systematically and globally evaluate the monthly precipitation forecasts of Japan Meteorological Agency/Meteorological Research Institute-Coupled Prediction System ver. 2 (JMA/MRI-CPS2), a dynamical seasonal climate forecast (Dyn-SCF) system operated by the Japan Meteorological Agency, by comparing its forecasts with those of a statistical SCF (St-SCF) system using climate indices. We developed a new global St-SCF system using 17 climate indices and compared the monthly precipitation of this system with those of JMA/MRI-CPS2. Consequently, the skill of JMA/MRI-CPS2 was determined to be globally higher than that of the St-SCF for zero-month lead forecasts. Contrarily, for forecasts made with a lead time of 1 month or longer, the deterministic skill of JMA/MRI-CPS2 was comparable to that of the St-SCF, and the probabilistic skill of JMA/MRI-CPS2 remained slightly higher. In addition to evaluating the skill of JMA/MRI-CPS2, we identified several regions and seasons, for which JMA/MRI-CPS2 exhibited a low forecast skill, compared with the St-SCF. This indicated that JMA/MRI-CPS2 cannot sufficiently reproduce certain dynamics. In conclusion, comparing Dyn-SCFs with St-SCFs can elucidate the potential regions and seasons to improve the forecast skill of Dyn-SCFs.

1. Introduction

Seasonal climate forecasts (SCFs), which are capable of predicting weather with lead times of 1 month to 1 year, provide useful information for decision-making and early warning systems in various fields, such as agriculture and water resource management (Doblas-Reyes et al. 2006; Jones et al. 2000; Klemm and McPherson 2017; Meinke and Stone 2005; Pozzi et al. 2013). However, their utility relies on forecast skill. Therefore, SCF skill evaluation is crucial in the construction of SCF systems (Kim et al. 2012).

Generally, evaluating the skill of SCFs involves analysis of their degree of similarity with observed data. As a more advanced approach, using climatology or simple statistical methods in the assessment of added values compared with the SCF system has been proposed (Luo et al. 2012; Pappenberger et al. 2015; Turco et al. 2017). For dynamical SCF (Dyn-SCF) systems, particularly those with large computational loads, the benefits of these added values need to outweigh their high cost compared with the forecast skill of less-expensive and simpler methods.

In the case of forecasting a few specific climate variables, statistical SCF (St-SCF) systems are an alternative to the Dyn-SCF systems (Doblas-Reyes et al. 2013). The forecast skills of the two systems have been compared in various manners and regions (Folland et al. 1991; Anderson et al. 1999; Barnston et al. 1999; van Oldenborgh et al. 2005; Quan et al. 2006; Wu et al. 2009; Pappenberger et al. 2015; Turco et al. 2017; Lenssen et al. 2020). Systematic global comparisons can be employed to identify the regions and seasons in which Dyn-SCF systems have advantages and disadvantages in forecasting.

Among the various statistical methods used in St-SCF systems, numerous studies have used climate indices such as Niño 3.4, the Southern Oscillation Index, and the Madden–Julian Oscillation (Quayle 1929; Nicholls et al. 1982; McBride and Nicholls 1983; Gordon 1986; Chu 1989; Stone et al. 1996; Chiew et al. 1998; Kirono et al. 2010; Schepen et al. 2012; Eden et al. 2015; Singh and Qin 2020). The predictability in using climate indices relies on the slow dynamics of the ocean and atmosphere as well as their associated climate states. This is similar for Dyn-SCF systems, whose predictability also depends on the presence of slow variations in soil moisture, snow cover, sea ice, and ocean surface temperature (Doblas-Reyes et al. 2013). Therefore, the forecast skill of St-SCFs that utilize climate indices is a suitable benchmark for Dyn-SCFs. By comparing Dyn-SCFs and St-SCFs, slow dynamics, which are insufficiently reproduced in Dyn-SCF systems, can be elucidated, contributing to the improvement of the skill of Dyn-SCFs.

The global Dyn-SCF system known as Japan Meteorological Agency/Meteorological Research Institute-Coupled Prediction System ver. 2 (JMA/MRICPS2) (Takaya et al. 2018) developed by the Japan Meteorological Agency (JMA) and Meteorological Research Institute (MRI) is used for operational seasonal forecasting in Japan. Takaya et al. (2018) reported that JMA/MRI-CPS2 exhibited improved forecast skill performance on interannual variability in the ocean and atmosphere, including El Niño events, compared with its predecessor model, JMA/MRI-CPS1 (Takaya et al. 2017). The Tokyo Climate Center of the World Meteorological Organization publishes monthly forecast skills of JMA/MRI-CPS2. Their evaluation includes reports on where and when precipitation forecast skill is high or low. Currently, comparisons with St-SCFs have not been performed for JMA/MRI-CPS2.

Precipitation forecasting is essential for effective water management and disaster reduction. The precipitation forecast skill of Dyn-SCF systems is shown to be lower than its temperature forecast skill, and areas with highly accurate precipitation forecasts are limited in the tropics (Doblas-Reyes et al. 2013). To date, the added value of the skill of Dyn-SCF systems for precipitation forecasts compared with St-SCF systems has not been determined.

In this study, we aimed to evaluate the monthly precipitation forecast skill of JMA/MRI-CPS2 compared with that of an St-SCF system by using climate indices. Furthermore, we discussed its likelihood of improving the forecast skill of Dyn-SCFs. The outline of this paper is as follows. Section 2 describes the data and methods, explaining the precipitation observation data (Section 2.1); the two models, JMA/MRI-CPS2 (Section 2.2) and an St-SCF using climate indices (Section 2.3); and the evaluation method of forecast skill (Section 2.4). Section 3 presents the results of forecast skill from three viewpoints: global (Section 3.1), spatial (Section 3.2), and regional (Section 3.3). Section 4 outlines and discusses the main findings. Finally, the conclusion is presented in Section 5.

2. Data and methods

The forecast skill of JMA/MRI-CPS2 was evaluated by comparing it with observed precipitation. An St-SCF system that utilizes 17 climate indices was then developed to generate monthly precipitation forecasts. The forecast skill of this St-SCF system was evaluated and compared with that of JMA/MRI-CPS2. The data and models, as well as the method used for evaluating forecast skill, are presented in Table 1.

2.1 Observation data on precipitation

Monthly precipitation data from the Global Precipitation Climatology Project (GPCP) (Adler et al. 2003, 2018) v2.3 provided by the Physical Sciences Laboratory of the National Oceanic & Atmospheric Administration/Office of Air and Radiation/Earth System Research Laboratories (NOAA/OAR/ESRL) were used as observations. Data from 1981 to 2020 was first divided into two half periods: 1981–2000 and 2001–2020. The data in the first period was used for the bias-correction of JMA/MRI-CPS2 (Section 2.2) and model development of the St-SCF system using climate indices (Section 2.3), and the second half period data was used for the evaluation of forecast skill of the two models (Section 2.4). As a preprocessing step, GPCP v2.3 was re-gridded to follow JMR/MRI-CPS2 using the bilinear method, as the center of their grids did not match even though the spatial resolution of both JMA/MRI-CPS2 and GPCP v2.3 was 2.5° × 2.5°.

2.2 JMA/MRI-CPS2

The main component of JMA/MRI-CPS2 is a coupled atmosphere–ocean model (JMA/MRI-CGCM2), with an atmospheric component based on the low-resolution version of the JMA Global Spectral Model (GMS1011C, Japan Meteorological Agency 2013). Its spatial resolution is TL159 (approximately 110 km) with 60 vertical layers. The ocean component of JMA/MRI-CGCM2 is based on the MRI Community Ocean Model ver. 3 (Tsujino et al. 2010), which includes a sea ice model; its spatial resolution is 1° east–west and 0.3–0.5° north–south and contains 52 vertical layers. The Japanese 55-year Reanalysis (Kobayashi et al. 2015) was employed to initialize the atmospheric data, and the Global Ocean Data Assimilation System [MOVE/MRI.COM-G2 (Toyoda et al. 2013)] was used for ocean data.

The JMA/MRI-CPS2 hindcast data were obtained from the Japan Meteorological Business Support Center. The hindcast period was 1979–2020, and the time resolution was daily. The daily values were averaged to produce monthly values. The hindcast data had a spatial resolution of 2.5° × 2.5° and included five ensembles with different initial conditions, explained as follows. Two forecasts started near the middle and end of each month; this study used the forecast closer to the end of the month with the following dates: January 31, February 25, March 27, April 26, May 31, June 30, July 30, August 29, September 28, October 28, November 27, and December 27. The forecast values in the first 6 months of the hindcast data, which covered 240 days, were used in the study. Figure 1 presents an example of 5-month lead forecasts of JMA/MRI-CPS2. For example, the precipitation forecasts for July used monthly precipitation forecasts that began in January.

Fig. 1.

Five-month lead forecasts by JMA/MRI-CPS2 and the St-SCF using climate indices.

The hindcast data for 2001–2020 was used for the evaluation of the forecast skill of JMA/MRI-CPS2. Before the evaluation, the data was bias-corrected as follows:   

  

where denotes bias-corrected forecast values for grid i, ensemble k (= 1−5), lead month LM (= 0−5) for forecast year Y (= 2001−2020), and month M. is the observed precipitation averaged over the years for bias-correction, Y_BC (= 1981−2000), for grid i and month M, and are forecasted precipitation averaged over the years for bias-correction (Y_BC = 1981−2000) and ensembles (k = 1−5) for grid i and lead month LM.

2.3 Statistical seasonal climate forecast system using climate indices

The St-SCF system using climate indices was constructed by first producing 17 precipitation forecasts from 1981 to 2000 with statistical models for the 17 climate indices. Second, the statistical model for the climate index with the highest Mean Squared Skill Score (MSSS) was selected. Third, 100 ensembles of the statistical model for the climate index were produced using the resampling method. Fourth, 100 ensembles of precipitation forecasts from 2001 to 2020 were produced by using the 100 ensembles of the statistical model.

The statistical models used in this study treat the climate index as the explanatory variable and precipitation as the objective variable. The model is expressed as follows:   

where PRE _{i, j, LM} (Y, M) denotes the forecast values of precipitation for grid i, climatic index j, lead month LM (= 0−5), forecast year Y, and month M. The expression IDX _j [M − (LM + 1)] is the value of climatic index j in M − (LM + 1), and f _{i, j, M, LM} is a function for the precipitation in M for grid i from climatic index j in M − (LM + 1). Figure 1 presents an example of a 5-month lead forecast by the St-SCF using climate indices. In the statistical model for precipitation forecasts in July, the precipitation for that month was treated as the objective variable and the climate indices in January as the explanatory variables.

In the first step, the leave-one-out method was employed to produce precipitation forecasts from 1981 to 2000. After removing the data of one forecast year from 1981 to 2000, the function f_{i, j, M, LM} was determined by using the remaining data through the smoothing spline method (Wood 2017). In this study, the “gam” function in the “mgcv” package of the R software v4.05 was used for the smoothing spline method. An example of this function, f_{i, j, M, LM}, is presented in Fig. 2. Next, the forecast values were obtained using the function determined from f _{i, j, M, LM} and the data removed in the first step. By repeating the aforementioned procedures for all years from 1981 to 2000, forecast values were obtained. In addition to the smoothing spline method, linear models were also used for the function, f _{i, j, M, LM}. The description of a part of the forecast skill of the linear models has been provided in the Appendix section.

Fig. 2.

Spline interpolation curve (red line) of MEI in July for estimating August precipitation at 110° longitude and −2.5° latitude. The plots denote the observational precipitation and MEI values.

In the third step, 100 ensembles of the statistical model for the climate index with the highest MSSS were constructed using the resampling method (Efron 1979; Masutomi et al. 2012, 2015). First, f _{i, j, M, LM} was determined by using the statistical model for the climate index with the highest MSSS and precipitation data from 1981 to 2000 using the smoothing spline method. Note that the leave-one-out method was not used in this step. Then, new precipitation data from 1981 to 2000 were produced by resampling residues between observed precipitation and precipitation calculated by the determined f _{i, j, M, LM} and by adding the resampled residues to the observed precipitation. By repeating the resampling procedures 100 times, 100 ensembles of new precipitation data were obtained. In the final step, 100 ensembles of f _{i, j, M, LM} were constructed by the 100 ensembles of new precipitation data using the smoothing spline method.

Table 2 summarizes the 17 climate indices used in this study by category. These indices were selected from climate indices provided by the NOAA Physical Sciences Laboratory, and the values were updated within approximately 1 week after the end of each month.

2.4 Evaluation of forecast skill

The deterministic and probabilistic forecast skills of JMA/MRI-CPS2 and St-SCF using climate indices were evaluated. The MSSS was used for the deterministic forecast skill, whereas the area under receiver operating characteristic curve (AUC) was used for the probabilistic one. The MSSS value is expressed as follows:   

where MSE (M) denotes the mean squared error for month M; VAR (M), the variance for month M; F (Y, M) and O (Y, M), forecast and observation values in year Y (= 2001−2020), respectively; N (= 20), the number of years for the evaluation; and Ō (M), the mean precipitation over 20 years from 2001 to 2020 for month M. A positive MSSS value indicates that the forecast has higher skill than climatological forecasts. For the evaluation of deterministic forecast skill, five ensemble mean values of JMA/MRI-CPS2 and 100 of the St-SCF using climate indices were used.

The AUC was calculated above and below the mean observational precipitation during 2001–2020. The forecast probability for each category was calculated using the ensembles of each model. The mean value of AUCs for each category was used for the evaluation. The detailed calculation of the AUC is described in Mason (2018). An AUC is smaller than 0.5 indicates that the forecast skill is lower than random forecasts.

The evaluation was conducted for global data at a spatial resolution of 2.5° × 2.5° for JMA/MRI-CPS2 and the St-SCF system using climate indices. In addition to the global evaluation by grids, the following five global metrics were calculated: (i) MSSS-rp, the ratio of areas with positive MSSS; (ii) MSSS-hi, the ratio of areas with positive and higher MSSS between JMA/MRI-CPS2 and the St-SCF using climate indices; (iii) AUC-av, the global average of AUC; (iv) AUC-r0.5, the ratio of areas with AUC > 0.5; and (v) AUC-hi, the ratio of areas with AUC > 0.5 and the higher AUC between JMA/MRI-CPS2 and St-SCF using climate indices. The calculations of these metrics are presented in Fig. 3. AUC-av is a simple metric for representing the global average AUC. While the global average of MSSS (MSSS-av) can be calculated, it was not used in this study as MSSS has no lower limit. Furthermore, large MSSS negative values in any grid tend to influence the global mean. MSSS-rp and AUC-r0.5 are used to represent the ratio of areas where models have higher forecast skill than climatology or random forecasts, respectively. MSSS-hi and AUC-hi are the metrics for representing the ratio of areas with higher forecast skill and are appropriate for understanding the improved model. Note that all forecast skill values are shown with three significant digits, although the significant digits of the original data are unknown.

Fig. 3.

Example calculations of (i) MSSS-rp, the ratio of areas with positive MSSS; (ii) MSSS-hi, the ratio of areas with positive and higher MSSS between JMA/MRI-CPS2 and the St-SCF using climate indices; (iii) AUC-av, the global average AUC; (iv) AUC-r0.5, the ratio of areas with AUC > 0.5; and (v) AUC-hi, the ratio of areas with AUC > 0.5 and higher AUC between JMA/MRI-CPS2 and the St-SCF using climate indices. The boxes represent grids, and the numbers in the boxes indicate the grid area and MSSS/AUC.

3. Results

3.1 Comparison of global forecast skill

Figure 4 presents the global values of MSSS-rp and AUC-av for JMA/MRI-CPS2 for each lead month. Although JMA/MRI-CPS2 has a high forecast skill in zero-month lead forecasts, the forecast skill rapidly decreases in the 1-month lead forecasts and gradually declines thereafter. The highest forecast skill of zero-month lead forecasts was observed in February, with an MSSS-rp of 0.325 and AUC-av of 0.643, whereas the worst forecast skill was observed in two different months: May with an MSSS-rp of 0.213 and September with an AUC-av of 0.590. For 1-month lead forecasts, February had an MSSS-rp of 0.157, less than half the value of zero-month lead forecasts. Comparing the MSSS-rp and AUC-av of the ocean and land, the forecast skill for ocean is evidently higher than that for land.

Fig. 4.

MSSS-rp (top) and AUC-av (bottom) by JMA/MRI-CPS2 [left, global average (GLB); center, average over land (LND); right, average over ocean (OCN)].

Figure 5 presents the global values of MSSS-rp and AUC-av for the St-SCF using climate indices for each lead month. The forecast skill decreases as the lead month increases, but the decrease is significantly smaller than that in JMA/MRI-CPS2. The highest forecast skill of zero-month lead forecasts for the global forecast was observed in 2 months, December with an MSSS-rp of 0.172 and January with an AUC-av of 0.537, whereas the worst forecast skill was observed in April with an MSSS-rp of 0.136 and June with an AUC-av of 0.513. For 1- and 5-month lead forecasts, the MSSS-rps of December were 0.157 and 0.145, and the AUC-avs of January were 0.532 and 0.521, respectively. Comparison of the ocean and land areas shows that the forecast skill is higher for ocean forecasts with lead times of 0 to 5 months.

Fig. 5.

MSSS-rp (top) and AUC-av (bottom) by the St-SCF [left, global average (GLB); center, average over land (LND); right, average over ocean (OCN)].

Figure 6 presents a comparison of the annual mean deterministic (MSSS-rp, MSSS-hi) and probabilistic (AUC-av, AUC-r0.5, and AUC-hi) forecast skills between JMA/MRI-CPS2 and the St-SCF using climate indices. Evidently, both the deterministic and probabilistic forecast skills of JMA/MRI-CPS2 were much higher than those of the St-SCF for zero-month lead time. The difference drastically became smaller for 1-month leads. For lead forecasts of 1 month or longer, the deterministic forecast skills (MSSS-rp, MSSS-hi) between JMA/MRI-CPS2 and the St-SCF using climate indices were not different. The probabilistic forecast skills (AUC-av, AUC-r0.5, and AUC-hi) of JMA/MRI-CPS2 were still higher than those of the St-SCF using climate indices, although the difference was small and gradually decreased for longer lead forecasts. Therefore, the forecast skill of JMA/MRI-CPS2 is generally higher for zero-month lead forecasts. However, if the forecasts are longer than 1 month, the deterministic skill of JMA/MRI-CPS2 is comparable to that of the St-SCF using climate indices, whereas the probabilistic skill of JMA/MRI-CPS2 remains slightly higher.

Fig. 6.

Comparison of MSSS-rp (top left), MSSS-hi (top right), AUC-av (bottom left), AUC-r0.5 (bottom center), and AUC-hi (bottom right) between JMA/MRI-CPS2 and the St-SCF using climate indices.

3.2 Spatial comparison of global forecast skill

Figure 7 presents the spatial distribution of MSSS for JMA/MRI-CPS2 in March, June, September, and December. In zero-month lead forecasts, areas with a positive MSSS are distributed worldwide, even in the middle latitudes, such as east Australia in September and Kazakhstan in December. However, the areas with positive MSSS were limited to low latitudes in lead forecasts of 1 month or longer.

Fig. 7.

Spatial distribution of MSSS for JMA/MRI-CPS2. The left, center-left, center-right, and right columns denote March, June, September, and December, respectively. The top, middle, and bottom denote the zero- to 2-month lead forecasts.

Figure 8 presents the spatial distribution of MSSS for the St-SCF using climate indices for the same months as presented in Fig. 7. The figure shows that the areas with high MSSS are generally limited to low latitudes even in zero-month lead forecasts. Figure 9 presents the climate indices selected for each grid with positive MSSS, demonstrating that the selected indices depend on the regions and forecast month. Table 3 presents the area ratio of selected climate indices for zero-month lead forecasts. The climate index with the largest selected area was MEI. The indices whose ratio of the selected area is > 0.01 were MEI, NINO1.2, NINO3, NINO3.4, NINO4, and SOI, which are ENSO-related, indicating that the St-SCF using climate indices largely relied on ENSO, presenting a physical background of the model.

Fig. 8.

Spatial distribution of MSSS for the St-SCF using climate indices. The left, center-left, center-right, and right columns denote March, June, September, and December, respectively. The top, middle, and bottom denote the zero- to 2-month lead forecasts.

Fig. 9.

Climate indices selected for grids with positive MSSS. The left, center-left, center-right, and right columns denote March, June, September, and December, respectively. The top, middle, and bottom denote the zero- to 2-month lead forecasts.

Figure 10 presents the annual mean of MSSS-hi (left) and AUC-hi (right) by latitudes for JMA/MRI-CPS2 (red line) and the St-SCF using climate indices (black line). The deterministic and probabilistic forecast skills of JMA/MRI-CPS2 were generally higher than those of the St-SCF using climate indices at all latitudes for zero-month lead forecasts. For lead forecasts of 1 month or longer, the skills of JMA/MRI-CPS2 were still higher than those of the St-SCF using climate indices at low latitudes, but as the lead month increased, the differences between the two models decreased, and the latitudes at which JMA/MRI-CPS2 has a higher forecast skill tended to narrow. Comparing the probabilistic and deterministic forecasts, the differences in probabilistic forecasts between JMA/MRI-CPS2 and the St-SCF using climate indices were larger at low latitudes than those in the deterministic forecasts. This is because the AUC-hi of JMA/MRI-CPS2 at low latitudes was higher than that at high-latitudes, whereas the AUC-hi of the St-SCF using climate indices did not change by latitudes. This is the reason why the global probabilistic forecast skills of JMA/MRI-CPS2 were slightly higher than those of the St-SCF using climate indices for lead forecasts of 1 month or longer, whereas there were no differences in the global deterministic forecasts between the two models (Fig. 6).

Fig. 10.

Comparison of latitude for MSSS-hi (left) and AUC-hi (right) for zero-month (top) to 5-month (bottom) lead forecasts. The red and black lines are the values by JMA/MRI-CPS2 and the St-SCF using climate indices, respectively.

3.3 Regional comparison of forecast skill for south Philippines in April and southwest Australia in December

Figures 7 and 8 present that the St-SCFs using climate indices have a positive MSSS in south Philippines during April and southwest Australia during December from zero- to 2-month lead forecasts; JMA/MRI-CPS2 had a negative MSSS. Figure 11 presents a comparison of the precipitation at a grid in south Philippines (a; 120°E, 10°N) for April and in southwest Australia (b; 117.5°E and 30°S) for December from 2001 to 2020 between observations and forecasts for zero- to 2-month lead times by JMA/MRI-CPS2 and the St-SCF using climate indices. The MSSS values are also presented in Fig. 11. The MSSSs of the St-SCF using climate indices were positive and higher than those by JMA/MRI-CPS2. In particular, they could accurately reproduce the annual variation of observations, even for forecasts 2 months in advance, whereas JMA/MRI-CPS2 was unable to forecast them. Figure 12 presents the relationship between climate indices and precipitation based on observations and forecasts of JMA/MRI-CPS2 for south Philippines and southwest Australia. From this figure, the inadequacies of the forecasts of JMA/MRI-CPS2 are evident. For example, in the zero-month lead forecasts of JMA/MRI-CPS2 for south Philippines, the linear relationship between climate indices and the forecasted precipitation was weak, with the forecasted precipitation showing a larger variation than the observation precipitation, especially for indices in the range of −1 to 0. Although the forecasts of JMA/MRI-CPS2 with 1- and 2-month lead times showed a clear linear relationship with the climate indices, they tended to overestimate precipitation in south Philippines. In addition, the forecasts for Australia could not reproduce higher precipitation for large negative indices, i.e., below −1; at an index of approximately 0, the zero-month lead forecasts of JMA/MRI-CPS2 were associated with a large error. These results indicated that certain dynamics were not well reproduced by JMA/MRI-CPS2, implying that further analysis and incorporation of these dynamics into this forecast system will improve its forecast skill.

Fig. 11.

Comparison of precipitation at south Philippines (a; 120°E, 10°N) and southwest Australia (b; 117.5°E, 30°S) from 2001 to 2020 between observations (GPCP: black circle) and forecasts (red dots) by JMA/MRI-CPS2 and the St-SCF with NINO3.4 or NINO4. The MSSS values are also shown.

Fig. 12.

Relationship between climate indices and precipitation in south Philippines (a; 120°E, 10°N) and southwest Australia (b; 117.5°E, 30°S). The dots indicate the observational precipitation and climate index values. The red triangles indicate the forecasted precipitation and climate indices.

4. Discussion

4.1 Forecast skill of JMA/MRI-CPS2 in comparison with St-SCF

The forecast skill of JMA/MRI-CPS2 was evaluated by Takaya et al. (2018) and published by the Tokyo Climate Center. This evaluation showed that the forecast skill of precipitation was higher at low latitudes and for zero-month lead forecasts; MSSS, the deterministic forecast skill, is highest in February and lowest in April to June, whereas AUC, the probabilistic skill, is highest in February and lowest in September. The same forecast skills were confirmed in this study (Figs. 4, 7). Furthermore, by comparing with the St-SCF using climate indices as benchmark, we identified the regions and lead periods in which JMA/MRI-CPS2 was advantageous. For example, the zero-month lead forecast skill of JMA/MRI-CPS2 was globally higher than that of the St-SCF (Figs. 6, 10). Generally, Dyn-SCF systems are known to have particularly high forecast skill in the tropics (Doblas-Reyes et al. 2013). Our study is the first to demonstrate the added value of Dyn-SCF systems on zero-month lead forecasts over St-SCF systems. Additionally, for forecasts longer than 1 month, the deterministic skill of JMA/MRI-CPS2 was comparable to that of the St-SCF using climate indices, and the probabilistic skill of JMA/MRI-CPS2 was slightly higher (Fig. 6). At mid- and high-latitudes, no large differences were observed in deterministic and probabilistic forecast skills between the two models (Fig. 10). These results clearly indicate that improving the skill of JMA/MRI-CPS2 for longer-term forecasts over 1 month is a challenge that must be addressed. The improvement in the skill of Dyn-SCFs in comparison with St-SCFs is discussed in the next section.

4.2 Improvement of forecast skill by comparing with St-SCF using climate indices

Various methods have been proposed to improve the forecast skill of Dyn-SCFs, including the initialization of soil moisture (Prodhomme et al. 2016b) and higher resolution (Prodhomme et al. 2016a). For JMA/MRI-CPS2, Takaya et al. (2021) demonstrated that the forecast skill significantly increases with the number of ensembles. However, the realization of these improvements required a great deal of effort. In addition, if potential regions and seasons for improvement were known in advance, the model improvement could have been more efficient. In this study, by comparing the forecast skill of JMA/MRI-CPS2 with that of the St-SCF system, we found that in several regions and seasons, JMA/MRI-CPS2 exhibited a low forecast skill, whereas the St-SCF using climate indices showed a high forecast skill. This clearly indicated the presence of certain dynamics that are not well reproduced by JMA/MRI-CPS2, implying that the skill of the Dyn-SCF system could still be improved via the incorporation of these dynamics. Therefore, the comparison between them clearly highlights potential regions and seasons for improvement of forecast skill. Thus, we proposed an approach for identifying such regions and seasons by comparing the forecast skill of Dyn-SCFs with that of St-SCFs.

5. Conclusions

By comparing JMA/MRI-CPS2 with St-SCF using climate indices as a benchmark, we identified the regions and lead periods in which JMA/MRI-CPS2 performed better. The main findings are as follows:

1. (i): The skill of JMA/MRI-CPS2 for global zero-month lead forecasts was higher than that of the St-SCF.
2. (ii): For lead forecasts of 1 month or longer, the deterministic skill of JMA/MRI-CPS2 was comparable to that of the St-SCF, and its probabilistic skill was slightly higher.

These findings not only present the significant added value of JMA/MRI-CPS2 but also its challenges for model improvement. Furthermore, the comparison of JMA/MRI-CPS2 and the St-SCF using climate indices showed the potential regions and seasons for which JMA/MRI-CPS2 does not adequately reproduce climate dynamics, implying that the skill of Dyn-SCFs can still be improved by incorporating these dynamics into the Dyn-SCF system. Thus, we concluded that:

1. (iii): Comparison between Dyn-SCFs and St-SCFs enables the determination of potential regions and seasons for the improvement of the forecast skill of Dyn-SCFs.

This approach is expected to be widely applied to improve the forecast skill of Dyn-SCFs.

Data Availability Statement

The JMA/MRI-CPS2 hindcast data can be purchased from the Japan Meteorological Business Support Center (http://www.jmbsc.or.jp/en/index-e.html). GPCP v2.3 is available at https://psl.noaa.gov/data/gridded/data.gpcp.html. The URLs where the 17 climate indices were obtained are listed in Table 2. All of the source codes used for the analyses in the present paper are stored at https://doi.org/10.5281/zenodo.5090304, whereas the source code of JMA/MRI-CPS2 is not open for the public.

Acknowledgments

This study was partly supported by the Coordination Fund for Promoting AeroSpace Utilization (JPJ000959) from the Ministry of Education, Culture, Sports, Science and Technology (MEXT), by KAKENHI (JP19H03069) from the Japan Society for the Promotion of Science, and by the Climate Change Adaptation Research Program at National Institute for Environmental Studies.

Appendix

Figure A1 presents MSSS-rp and MSSS-hi for JMA/MRI-CPS2 and the St-SCF using climate indices based on the spline method (St-SCF-spl) and St-SCF using climate indices based on the linear method (St-SCF-lin). No difference was observed in the forecast skill between St-SCF-spl and St-SCF-lin.

Fig. A1.

Comparison of MSSS-rp (left) and MSSS-hi (right) between JMA/MRI-CPS2 and the St-SCF using climate indices based on the spline method (St-SCF-spl) and linear method (St-SCF-lin).

References

•  Adler,  R. F.,  G. J.  Huffman,  A.  Chang,  R.  Ferraro,  P.-P.  Xie,  J.  Janowiak,  B.  Rudolf,  U.  Schneider,  S.  Curtis,  D.  Bolvin,  A.  Gruber,  J.  Susskind,  P.  Arkin, and  E.  Nelkin, 2003: The version-2 Global Precipitation Climatology Project (GPCP) monthly precipitation analysis (1979–present). J. Hydrometeor., 4, 1147–1167.
•  Adler,  R. F.,  M. R. P.  Sapiano,  G. J.  Huffman,  J.-J.  Wang,  G.  Gu,  D.  Bolvin,  L.  Chiu,  U.  Schneider,  A.  Becker,  E.  Nelkin,  P.  Xie,  R.  Ferraro, and  D.-B.  Shin, 2018: The Global Precipitation Climatology Project (GPCP) monthly analysis (new version 2.3) and a review of 2017 global precipitation. Atmosphere, 9, 138, doi:10.3390/atmos9040138.
•  Anderson,  J.,  H.  van den Dool,  A.  Barnston,  W.  Chen,  W.  Stern, and  J.  Ploshay, 1999: Present-day capabilities of numerical and statistical models for atmospheric extratropical seasonal simulation and prediction. Bull. Amer. Meteor. Soc., 80, 1349–1362.
•  Barnston,  A. G.,  M. H.  Glantz, and  Y.  He, 1999: Predictive skill of statistical and dynamical climate models in SST forecasts during the 1997–98 El Niño episode and the 1998 La Niña onset. Bull. Amer. Meteor. Soc., 80, 217–244.
•  Chiew,  F. H. S.,  T. C.  Piechota,  J. A.  Dracup, and  T. A.  McMahon, 1998: El Niño/Southern Oscillation and Australian rainfall, streamflow and drought: Links and potential for forecasting. J. Hydrol., 204, 138–149.
•  Chu,  P.-S., 1989: Hawaiian drought and the Southern Oscillation. Int. J. Climatol., 9, 619–631.
•  Doblas-Reyes,  F. J.,  R.  Hagedorn, and  T. N.  Palmer, 2006: Developments in dynamical seasonal forecasting relevant to agricultural management. Climate Res., 33, 19–26.
•  Doblas-Reyes,  F. J.,  J.  García-Serrano,  F.  Lienert,  A. P.  Biescas, and  L. R. L.  Rodrigues, 2013: Seasonal climate predictability and forecasting: Status and prospects. WIREs Climate Change, 4, 245–268.
•  Eden,  J. M.,  G. J.  van Oldenborgh,  E.  Hawkins, and  E. B.  Suckling, 2015: A global empirical system for probabilistic seasonal climate prediction. Geosci. Model Dev., 8, 3947–3973.
•  Efron,  B., 1979: Bootstrap methods: Another look at the jackknife. Breakthroughs in Statistics.  Kotz,  S., and  N. L.  Johnson (eds.), Springer Series in Statistics, Springer, New York, 1–26.
•  Folland,  C.,  J.  Owen,  M. N.  Ward, and  A.  Colman, 1991: Prediction of seasonal rainfall in the Sahel region using empirical and dynamical methods. J. Forecasting, 10, 21–56.
•  Gordon,  N. D., 1986: The Southern Oscillation and New Zealand weather. Mon. Wea. Rev., 114, 371–387.
• Japan Meteorological Agency, 2013: Outline of the operational numerical weather prediction at the Japan Meteorological Agency. 201 pp. [Available at: https://www.jma.go.jp/jma/jma-eng/jma-center/nwp/outline2013-nwp/pdf/outline2013_all.pdf.]
•  Jones,  J. W.,  J. W.  Hansen,  F. S.  Royce, and  C. D.  Messina, 2000: Potential benefits of climate forecasting to agriculture. Agric. Ecosyst. Environ., 82, 169–184.
•  Kim,  H.-M.,  P. J.  Webster, and  J. A.  Curry, 2012: Seasonal prediction skill of ECMWF System 4 and NCEP CFSv2 retrospective forecast for the Northern Hemisphere winter. Climate Dyn., 39, 2957–2973.
•  Kirono,  D. G. C.,  F. H. S.  Chiew, and  D. M.  Kent, 2010: Identification of best predictors for forecasting seasonal rainfall and runoff in Australia. Hydrol. Processes, 24, 1237–1247.
•  Klemm,  T., and  R. A.  McPherson, 2017: The development of seasonal climate forecasting for agricultural producers. Agric. For. Meteorol., 232, 384–399.
•  Kobayashi,  S.,  Y.  Ota,  Y.  Harada,  A.  Ebita,  M.  Moriya,  H.  Onoda,  K.  Onogi,  H.  Kamahori,  C.  Kobayashi,  H.  Endo,  K.  Miyaoka, and  K.  Takahashi, 2015: The JRA-55 reanalysis: General specifications and basic characteristics. J. Meteor. Soc. Japan, 93, 5–48.
•  Lenssen,  N. J. L.,  L.  Goddard, and  S.  Mason, 2020: Seasonal forecast skill of ENSO teleconnection maps. Wea. Forecasting, 35, 2387–2406.
•  Luo,  Y. Q.,  J. T.  Randerson,  G.  Abramowitz,  C.  Bacour,  E.  Blyth,  N.  Carvalhais,  P.  Ciais,  D.  Dalmonech,  J. B.  Fisher,  R.  Fisher,  P.  Friedlingstein,  K.  Hibbard,  F.  Hoffman,  D.  Huntzinger,  C. D.  Jones,  C.  Koven,  D.  Lawrence,  D. J.  Li,  M.  Mahecha,  S. L.  Niu,  R.  Norby,  S. L.  Piao,  X.  Qi,  P.  Peylin,  I. C.  Prentice,  W.  Riley,  M.  Reichstein,  C.  Schwalm,  Y. P.  Wang,  J. Y.  Xia,  S.  Zaehle, and  X. H.  Zhou, 2012: A framework for benchmarking land models. Biogeosciences, 9, 3857–3874.
•  Mason,  S. J., 2018: Guidance on verification of operational seasonal climate forecasts. World Meteorological Organization, Commission for Climatology XIV Technical Report, 74 pp. [Available at https://iri.columbia.edu/wp-content/uploads/2016/12/CCl-verification-recommendations-final.pdf.]
•  Masutomi,  Y.,  T.  Iizumi,  K.  Takahashi, and  M.  Yokozawa, 2012: Estimation of the damage area due to tropical cyclones using fragility curves for paddy rice in Japan. Environ. Res. Lett., 7, 014020, doi:10.1088/1748-9326/7/1/014020.
•  Masutomi,  Y.,  M.  Arakawa,  T.  Minoda,  T.  Yonekura, and  T.  Shimada, 2015: Critical air temperature and sensitivity of the incidence of chalky rice kernels for the rice cultivar “Sai-no-kagayaki”. Agric. For. Meteorol., 203, 11–16.
•  McBride,  J. L., and  N.  Nicholls, 1983: Seasonal relationships between Australian rainfall and the Southern Oscillation. Mon. Wea. Rev., 111, 1998–2004.
•  Meinke,  H., and  R. C.  Stone, 2005: Seasonal and inter-annual climate forecasting: The new tool for increasing preparedness to climate variability and change in agricultural planning and operations. Climatic Change, 70, 221–253.
•  Nicholls,  N.,  J. L.  McBride, and  R. J.  Ormerod, 1981: On predicting the onset of the Australian wet season at Darwin. Mon. Wea. Rev., 110, 14–17.
•  Pappenberger,  F.,  M. H.  Ramos,  H. L.  Cloke,  F.  Wetterhall,  L.  Alfieri,  K.  Bogner,  A.  Mueller, and  P.  Salamon, 2015: How do I know if my forecasts are better? Using benchmarks in hydrological ensemble prediction. J. Hydrol., 522, 697–713.
•  Pozzi,  W.,  J.  Sheffield,  R.  Stefanski,  D.  Cripe,  R.  Pulwarty,  J. V.  Vogt,  R. R.  Heim, Jr.,  M. J.  Brewer,  M.  Svoboda,  R.  Westerhoff,  A. I. J. M.  van Dijk,  B.  Lloyd-Hughes,  F.  Pappenberger,  M.  Werner,  E.  Dutra,  F.  Wetterhall,  W.  Wagner,  S.  Schubert,  K.  Mo,  M.  Nicholson,  L.  Bettio,  L.  Nunez,  R.  van Beek,  M.  Bierkens,  L. G. G.  de Goncalves,  J. G. Z.  de Mattos, and  R.  Lawford, 2013: Toward global drought early warning capability: Expanding international cooperation for the development of a framework for monitoring and forecasting. Bull. Amer. Meteor. Soc., 94, 776–785.
•  Prodhomme,  C.,  L.  Batté,  F.  Massonnet,  P.  Davini,  O.  Bellprat,  V.  Guemas, and  F. J.  Doblas-Reyes, 2016a: Benefits of increasing the model resolution for the seasonal forecast quality in EC-Earth. J. Climate, 29, 9141–9162.
•  Prodhomme,  C.,  F.  Doblas-Reyes,  O.  Bellprat, and  E.  Dutra, 2016b: Impact of land-surface initialization on sub-seasonal to seasonal forecasts over Europe. Climate Dyn., 47, 919–935.
•  Quan,  X.,  M.  Hoerling,  J.  Whitaker,  G.  Bates, and  T.  Xu, 2006: Diagnosing sources of U.S. seasonal forecast skill. J. Climate, 19, 3279–3293.
•  Quayle,  E. T., 1929: Long-range rainfall forecasting from tropical (Darwin) air pressures. Proc. R. Soc. Victoria, 41, 160–164.
•  Schepen,  A.,  Q. J.  Wang, and  D.  Robertson, 2012: Evidence for using lagged climate indices to forecast Australian seasonal rainfall. J. Climate, 25, 1230–1246.
•  Singh,  V., and  X.  Qin, 2020: Study of rainfall variabilities in Southeast Asia using long-term gridded rainfall and its substantiation through global climate indices. J. Hydrol., 585, 124320, doi:10.1016/j.jhydrol.2019.124320.
•  Stone,  R. C.,  G. L.  Hammer, and  T.  Marcussen, 1996: Prediction of global rainfall probabilities using phases of the Southern Oscillation Index. Nature, 384, 252–255.
•  Takaya,  Y.,  T.  Yasuda,  Y.  Fujii,  S.  Matsumoto,  T.  Soga,  H.  Mori,  M.  Hirai,  I.  Ishikawa,  H.  Sato,  A.  Shimpo,  M.  Kamachi, and  T.  Ose, 2017: Japan Meteorological Agency/Meteorological Research Institute-Coupled Prediction System version 1 (JMA/MRI-CPS1) for operational seasonal forecasting. Climate Dyn., 48, 313–333.
•  Takaya,  Y.,  S.  Hirahara,  T.  Yasuda,  S.  Matsueda,  T.  Toyoda,  Y.  Fujii,  H.  Sugimoto,  C.  Matsukawa,  I.  Ishikawa,  H.  Mori,  R.  Nagasawa,  Y.  Kubo,  N.  Adachi,  G.  Yamanaka,  T.  Kuragano,  A.  Shimpo,  S.  Maeda, and  T.  Ose, 2018: Japan Meteorological Agency/Meteorological Research Institute-Coupled Prediction System version 2 (JMA/MRI-CPS2): Atmosphere–land–ocean–sea ice coupled prediction system for operational seasonal forecasting. Climate Dyn., 50, 751–765.
•  Takaya,  Y.,  Y.  Kosaka,  M.  Watanabe, and  S.  Maeda, 2021: Skilful predictions of the Asian summer monsoon one year ahead. Nat. Commun., 12, 2094, doi:10.1038/s41467-021-22299-6.
•  Toyoda,  T.,  Y.  Fujii,  T.  Yasuda,  N.  Usui,  T.  Iwao,  T.  Kuragano, and  M.  Kamachi, 2013: Improved analysis of seasonal-interannual fields using a global ocean data assimilation system. Theor. Appl. Mech. Japan, 61, 31–48.
•  Tsujino,  H.,  T.  Motoi,  I.  Ishikawa,  M.  Hirabara,  H.  Nakano,  G.  Yamanaka,  T.  Yasuda, and  H.  Ishizaki, 2010: Reference manual for the Meteorological Research Institute Community Ocean Model (MRI. COM) version 3. Tech. Rep. Meteor. Res. Inst, No. 59, 265 pp.
•  Turco,  M.,  A.  Ceglar,  C.  Prodhomme,  A.  Soret,  A.  Toreti, and  J.  Doblas-Reyes Francisco, 2017: Summer drought predictability over Europe: Empirical versus dynamical forecasts. Environ. Res. Lett., 12, 084006, doi:10.1088/1748-9326/aa7859.
•  van Oldenborgh,  G. J.,  M. A.  Balmaseda,  L.  Ferranti,  T. N.  Stockdale, and  D. L. T.  Anderson, 2005: Did the ECMWF seasonal forecast model outperform statistical ENSO forecast models over the last 15 years?. J. Climate, 18, 3240–3249.
•  Wood,  S. N., 2017: Generalized Additive Models: An Introduction with R. 2nd edtion. Chapman and Hall/CRC, 496 pp.
•  Wu,  Z.,  B.  Wang,  J.  Li, and  F.-F.  Jin, 2009: An empirical seasonal prediction model of the east Asian summer monsoon using ENSO and NAO. J. Geophys. Res., 114, D18, doi:10.1029/2009JD011733.