Assessing the subnational-level yield forecast skills of the 2019/20 season NARO-APCC Joint Crop Forecasting Service for Southern Hemisphere countries

An unstable supply of commodity crops and associated increases in food prices are recent and growing concerns due to increasing temperatures, changing precipitation patterns and increasing frequencies of some extreme climate events. Agricultural monitoring and forecasting can support national food agencies, international organizations and commercial entities in better responding to anticipated production shocks induced by seasonal climate extremes. The global seasonal crop forecasting service jointly developed in 2018 by the National Agriculture and Food Research Organization ( NARO ) , Japan and the Asia-Pacific Economic Cooperation Climate Center ( APCC ) , South Korea is an emerging and unique example of agricultural forecasting tailored to major commodity crops ( maize, rice, wheat and soybean ) . The present study evaluates the skills of the NARO-APCC yield forecasts in five countries located in the Southern Hemisphere ( the 2019/20 season in Australia and Uruguay and the 2018/19 season in Argentina, Brazil and Paraguay ) , following the previous assessment for the 2019 season in Northern Hemisphere countries. The results reveal that the NARO-APCC forecasts can capture the major characteristics of reported state yields even six months before harvesting, with variations by crop ( the correlation coefficients calculated between the forecasted and reported state yields within a country in a season of interest were frequently over 0.8 for maize, rice and wheat and approximately 0.3 for soybean ) . In three-fifths of the 122 crop-state combinations assessed here, the NARO-APCC forecasts showed smaller forecast errors than those of the simple forecasts derived solely based on the reported yields. The findings of this study emphasize the novelty of long-range crop forecasting, such as the NARO-APCC forecasts that provide yield forecast information available even just after planting. Together, the NARO-APCC forecasts and existing regional crop forecasts contribute to making objective yield forecast information more seamlessly available throughout the season from planting to harvesting than what is currently available.


Introduction
The accumulated evidence indicates that the observed climate change is already affecting crop production in many regions of the world through increasing temperatures, changing precipitation patterns and increasing frequencies of some types of extreme climate events Mbow et al., 2019;FAO, 2021 . Increases in the frequency and intensity of high temperatures, heavy precipitation and agricultural droughts under possible future climates in addition to changes in long-term mean temperature and precipitation pose growing concerns for unstable supply of commodity crops and associated increases in food prices Porter et al., 2014;Wiebe et al., 2015;Tigchelaar et al., 2018, Hasegawa et al., 2021 Agricultural monitoring and forecasting play an important role in strengthening the preparedness of societies for anticipated seasonal climate extremes and production shocks FAO, 2016a;Becker-Reshef et al., 2019;Fritz et al., 2019;Kim et al., 2021 . Importantly, these activities also enable adaptation to climate change for the coming decades Ceglar et al., 2018;Iizumi and Kim, 2019 . Therefore, seasonal climate and crop forecasts have been utilized by national food agencies and international organizations in recent years Iizumi, 2014;FAO, 2016b; The Crop Monitors, 2021 . As many more countries have increasingly relied on food imports in recent decades and will in future decades due to the globalization of the economy FAO, 2011; OECD/FAO, 2021 , global seasonal crop forecasting is an emerging and unique effort that aims to inform not only national food agencies and international organizations but also commercial entities in particular those in import-dependent countries about seasonal climate-induced variations in yields of major commodity crops worldwide three to six months before harvesting.
A global seasonal crop forecasting service based on a multimodel ensemble MME of seasonal temperature and precipitation forecasts was developed in 2018 as a joint research project between the National Agriculture and Food Research Organization NARO , Japan and the Asia-Pacific Economic Cooperation APEC Climate Center APCC , South Korea Iizumi et al., 2018 , and it was proposed in the early 2010s Iizumi et al., 2013 . The NARO-APCC Joint Crop Forecasting Service was initially tested from June 2019 to March 2021 Iizumi, 2020 . The skills of the NARO-APCC forecasts for the 2019 season in countries located in the Northern Hemisphere, including 17 European countries, the United States US , Japan and South Korea, were evaluated in Iizumi et al. 2021a . Country selection was based on the availability of governmental yield statistics for the 2019 season at the time of analysis. Based on the assessment, the NARO-APCC forecasts can provide an outlook of yields one to five months earlier than existing operational crop forecasting services for specific regions, such as those operated by the US Department of Agriculture USDA and the European Commission's Joint Research Centre JRC , while the forecast errors of the NARO-APCC forecasts are larger than those of the regional services.
Considering that the skills of the NARO-APCC forecasts in countries located in the Southern Hemisphere have not yet been assessed, this study aimed to evaluate the skills of the NARO-APCC forecasts for the 2019/20 season in Australia and Uruguay as well as the 2018/19 season in Argentina, Brazil and Paraguay. Four major commodity crops maize, rice, wheat and soybean were studied, as in a previous assessment in Northern Hemisphere countries Iizumi et al., 2021a . These Southern Hemisphere countries were selected based on their share of exports. According to the Food and Agriculture Organization of the United Nations FAO , Argentina, Brazil, Paraguay and Uruguay together accounted for nearly 100 of the total export of the four previously noted crops in South America in 2019. The same was true for Australia in Oceania.
The present study complements the previous assessment and contributes to having a more complete picture of the NARO-APCC forecast skills than those that are currently understood. Furthermore, this study presents a state-level assessment for Argentina, Brazil and Australia. Assessments at a spatial resolution finer than the national level were only performed in the US in the previous assessment. Therefore, this study provides additional insights into the value of the NARO-APCC forecasts at the subnational level beyond what we already learned from the earlier national-level assessments.

Reported yields
The annual yield statistics for the four seasons from 2016/17   to 2019/20 or 2015/16 to 2018/19 were collected. Data for the  2019/20 season were available for Australia and Uruguay when  this study was carried out as of 18 June 2021 , while data for  the 2018/19 season were the most recent available for Argentina,  Brazil, and Paraguay at the time of the study. Subnational data  were available for Argentina, Brazil and Australia Table 1 . Only national data were collected for Paraguay and Uruguay. The area of these two countries together is relatively close to the area of a single state in the remaining three countries, justifying the use of national-level data for Paraguay and Uruguay Fig. 1 b and c . As shown in Table S1, the subnational data used for this study were as follows: Argentina, the yields of maize, rice, soybean and wheat in 15 of 24 crop-producing states, available from the Ministry of Agriculture, Livestock and Fisheries of Argentina; Brazil, the yields of the four crops in all 27 states available from the Brazilian Institute of Geography and Statistics; and Australia, the yields of wheat 6 states , maize 5 states , soybean 3 states and rice 2 states available from the Department of Agriculture, Water and the Environment, Australian Government.
We confirmed that the total national area harvested and production as well as the national average yield calculated using the subnational data collected were equal to the records in the FAO database after aligning the harvesting year between the national reports and FAO database Table S2 . The data for the 2019/20 season in Australia and Uruguay were not yet available in the FAO database and therefore were not compared. The national data for Uruguay and Paraguay were obtained from the agricultural statistical year books and FAO database, respectively see Table S1 for the sources of data .  Table S1 for the state codes. The x-axis and y-axis of each panel indicate the longitude and latitude, respectively.

. NARO-APCC forecasts
We used the yield anomaly forecast data at a grid size of 1.125 approximately 120 km at the equator provided by the NARO-APCC Joint Crop Forecasting Service from September 2015 to October 2020. As the test of the NARO-APCC service started June 2019, the data used here were mostly obtained by reforecasting using the same system as that used in the test of the service. The forecast data of the service are currently not available online but have been shared with interested users upon reasonable request to the corresponding author.
The multiple linear regression models used in the system associated key growing season temperature and precipitation anomalies with yield anomalies. The modeled yield anomalies are primarily driven by variations in climate conditions. These anomalies might include damages due to pests and diseases to some degree since the models were built based on actual yields and it is empirically known that some climate conditions e.g., warmer and wetter than normal one affect activities of pest and disease species Rosenzweig et al., 2001 . The models were established grid cell by grid cell to consider the diversity of local agronomic management using the representation of actual yield and seasonal climate conditions. The models were designed to use APCC MME monthly average 2-m air temperature and precipitation 6 month forecast data Min et al., 2014;Sohn et al., 2019 , which are issued on the 20th of each month, as inputs to derive yield anomaly forecasts. Once generated, the NARO-APCC Join Crop Forecast report is automatically sent to interested users on the 1st of the following month. More details of the models and in-depth validation are available in the literature Iizumi et al., 2018;2021a . Apart from the NARO-APCC yield forecast skills explored here, the APCC's seasonal climate forecast skills are available in the literature and compared to climate forecasts from other major weather centers, such as the National Oceanic and Atmospheric Administration's global climate forecast system version 2 NCEP/GFSv2 Min et al., 2017 for interested readers.
Postprocessing was performed on the NARO-APCC forecasts since the predicted variable of the NARO-APCC forecasts is the yield anomaly as a percentage of the normal yield but not the yield in tons per hectare. Yield forecasts are a part of production forecasts, and forecast skills of yield rather than those of yield anomalies are the primary interest for users of crop forecast information. As described in Iizumi et al. 2018 , the yield anomaly used in the NARO-APCC forecasts is defined as where the subscript t indicates the harvesting year; ΔY is the yield anomaly of the normal yield ; Y t and Y t-1 are the yields in the coming harvesting year and the previous year t ha -1 , respectively; and Y t-3:t-1 is the 3-year average yield for the period from year t-3 to t-1 that represents the time-varying normal yield in year t-1. Accordingly, the yield anomaly forecast can be converted into the yield forecast by where the superscripts f and r indicate the forecast and reported data, respectively. Y f,NARO-APCC is the NARO-APCC yield forecast in tons per hectare derived by combining the yield anomaly forecast with the reported yields for the 3 years from t -3 to t -1. If the 2019/20 yield forecast was taken as the example for explanatory purposes, then the reported data for the period from 2015/16 to 2017/18 were used for the conversion.

Simple forecasts
We also used a simple forecasting method to provide 'reference' yield forecasts to compare with the NARO-APCC forecasts. In the method adopted here, the average reported yield for the 3 years from t-3 to t-1 is deemed as the yield forecast for Although they are empirical, the average crop yields for last a few years serve as reasonable forecasts. This method is solely based on reported yields, and no additional information derived from producer surveys, field measurements, weather observations, climate forecasts, satellite remote sensing, or crop models is considered. The yield forecasts derived from this method provide a benchmark to measure the added value of the NARO-APCC forecasts gained by incorporating seasonal climate forecast information. Using simple method-based forecasts for benchmarking is a popular practice when assessing forecasts based on more sophisticated methods Lecerf et al., 2019;Iizumi et al., 2021a . Climatological forecasts based solely on climatological statistics for a region American Meteorological Society, 2012 play a similar role in weather and climate forecast evaluations.

Skill scores
We used the Pearson correlation coefficient, the root-mean-squared error RMSE , the absolute error AE , and the absolute percentage error APE to measure the skill of the yield forecasts. These skill scores were selected to be consistent with those in the earlier assessment Iizumi et al., 2021a . When we assessed the skills of the 2019/20 season yield forecasts, the correlation coefficient R; dimensionless is calculated as

Crop calendars
The crop calendars for the individual crops and countries were used supplementally. These were collected from two sources of information to account for the possible discrepancies between the sources. One was the FAO/Global Information and Early Warning System on Food and Agriculture FAO/GIEWS, 2021 , and the other was the Agricultural Market Information System AMIS, 2021 . The crop calendars used within the NARO-APCC system were based on Sacks et al. 2010 , which in part used the information from the FAO/GIEWS in its compilation.

Argentina
The crop calendars in Argentina indicated that maize, rice and soybean the summer crops were planted in approximately October -November and harvested in approximately March -April, while planting and harvesting wheat the winter crop occurred in approximately May -June and November -December, respectively, with some variations by source of crop calendar information Fig. 2 a . The NARO-APCC forecasts were available from approximately just after planting to three months before the harvesting of each crop: October -January for maize, November -February for rice and soybean, and June -September for wheat Fig. 2 a . For soybean, the skill score values of the NARO-APCC forecasts increased as the harvest approached. The correlation values increased from 0.275 in November to 0.512 in February, and the RMSE values decreased from 0.83 t ha -1 to 0.70 t ha -1 Fig. 3 c .
However, no tendency similar to that of the soybean forecasts was found for the remaining crops in Argentina. For rice and wheat, the skill score values were almost unchanged over the months. in December Fig. 3 a . Last, the NARO-APCC forecasts overestimated the rice yields and underestimated the maize and soybean yields relative to the reported data, although no systematic error was found for the wheat yields Fig. 3 ad .

Brazil
The crop calendars in Brazil are complex and substantially vary by geographic region within the country and source of information. If simplified for descriptive purposes, then rice and soybean were planted in approximately October -November and harvested in approximately March -April, while wheat was planted in approximately April -May and harvested in approximately October Fig. 2 b . Double maize cropping was performed; the first season spanned approximately October -April, and the second season spanned approximately February -July. The NARO-APCC forecasts for Brazil were available October -May for maize, October -April for rice, November -February for soybean, and April -August for wheat Fig. 2  in February -April Fig. 4 February Fig. 3 c .
The results of the NARO-APCC wheat forecasts were not similar to either of the two cases mentioned above. The number of states where the NARO-APCC wheat forecasts were available increased from seven in April to eight in May -July and then decreased to three in August Fig. 4 d . The skill score values fluctuated by month, and the best values appeared in July correlation, 0.854; RMSE, 0.46 t ha -1 .

Paraguay
For maize in Paraguay, the minor season spanned approximately September -January, whereas the main season spanned February - July Fig. 2 c . To cover these two seasons, the NARO-APCC forecasts for maize in Paraguay were available from September -May. Rice, and soybean was planted in approximately October -November and harvested in approximately April. Wheat was planted in approximately April and harvested in approximately November. The NARO-APCC forecasts were available October -February for rice and November -February for soybean. Although the NARO-APCC forecasts for wheat in  Paraguay were available March -September, the March forecasts were one month earlier than the reported planting month, implying that the crop calendar information used in the NARO-APCC system may have been incorrect. The NARO-APCC maize forecasts showed lower AE values than those of the simple forecasts for most months except January and April Fig. 5 a . Although the NARO-APCC forecasts were found to be better than the simple forecasts for a few cases, such as rice in December Fig. 5 b and wheat in September Fig. 5 d , the simple forecasts did a better job for almost all of the months. For soybean, the NARO-APCC forecasts did not outperform the simple forecasts in any of the months Fig. 5 c .

Uruguay
The summer crops in Uruguay were planted in approximately October -November and harvested in approximately April, while the winter crop was planted in approximately June -July and harvested in approximately November -December Fig. 2 d . The NARO-APCC forecasts for Uruguay were available for October -January for maize, October -February for rice, November -February for soybean, and June -August for wheat.
The NARO-APCC rice forecasts outperformed the simple forecasts throughout the months in terms of the AE values Fig. 6 b . The NARO-APCC forecasts for maize in October and for wheat in July were better than the simple forecasts for maize but not for the remaining months Fig. 6 a and d . For soybean, the NARO-APCC forecast performed worse than the simple forecast for all the months Fig. 6 c .

Australia
The crop calendars in Australia indicated that the summer crops were planted in approximately November and harvested in approximately April, while planting and harvesting the winter crop occurred in approximately May and December, respectively Fig. 2 e . The NARO-APCC forecasts for Australia were available for November -February for maize and rice and May -October for wheat. The NARO-APCC forecasts were not provided for soybean in Australia due to the lack of soybean calendars for the country in the NARO-APCC system.
For wheat, the skill score values of the NARO-APCC forecasts decreased as the harvesting approached. The correlation values drastically decreased from 0.914 -0.923 in May -June to 0.124 in October Fig. 7 Fig. 7 a .
The AE values were calculated for rice instead of the correlation and RMSE due to the small number of rice-producing states in Australia. The AE values for NS, where the only state in which the evaluation of the NARO-APCC rice forecasts in Australia was possible, decreased as the harvest approached 2.47 -2.60 t ha -1 in November -December to 1.43 -1.93 t ha -1 in January -February Fig. 8 .

Comparisons with the simple forecasts at the state level
The comparisons between the NARO-APCC forecasts and simple forecasts across the crops and states are shown in Fig. 9, with the detailed monthly time courses of the AE values in Figs. S1 -S10. The overall results can be summarized as follows: in 72 cases 59 out of the 122 crop-state combinations studied here, the NARO-APCC forecasts showed smaller APE values than those of the simple forecasts Table 2 .
The country-by-country results are also shown in Table 2. In Argentina, the NARO-APCC forecasts were better than the simple forecasts for 40 or 6 states out of 15 states 6/15 for maize Fig. S1   NARO-APCC forecasts were found to be the most skillful for wheat 72 of the wheat-producing states in the three countries assessed here , followed by rice 69 and maize 60 Table 2 . In comparison to those for the other crops considered here, the NARO-APCC forecasts for soybean 36 were found to be least skillful. It was worth to mention that the skills of the NARO-APCC state-level yield forecasts evidently depended on the APCC MME temperature and precipitation forecast skills see Supplemental text for details .

Discussion
The skill assessment results of the NARO-APCC Joint Crop Forecasting service presented here are based on a single season and therefore should be interpreted as the preliminary outcome. Although a previous study assessing the skills of a prototype of the NARO-APCC service was based on the data for a 32-year period 1984 -2015 Iizumi et al., 2018 , this study focuses on    the skills of the service that is currently being tested Iizumi et al., 2021a . Assessments with multiyear samples are desired but are currently not available for two main reasons. The NARO-APCC service just started in June 2019, and national agricultural statistics, especially in developing countries, are not readily available after the completion of a season. Given these limitations, we compared the NARO-APCC yield forecasts with the reference forecasts derived using the simple method. Yield forecasts provided by any agency independent of the NARO-APCC service in an operational manner are not available for the countries studied here, whether they are derived by satellite remote sensing or crop model simulations.
Decreasing forecast errors with the progress of crop growth are often reported for the existing regional crop forecasting services, including the USDA Egelkraut et al. 2003;Holland, 2011 andJRC van der Velde andNisini, 2019 . However, this is not a major feature of the NARO-APCC forecasts, although a decrease in forecast error was observed for some cases, as described earlier. This characteristic notably contrasts with those of existing regional services. The design of the NARO-APCC forecasting system relies solely on seasonal climate forecast information, and therefore, no information obtained from satellite remote sensing, weather observations or field measurements is used to update crop conditions in the forecasting system when providing yield forecasts, explaining this feature.
The NARO-APCC forecasts can capture the major characteristics of the reported state yields three to six months before harvesting, when yield forecast information from existing regional services is not available. More accurate forecasts than the NARO-APCC forecasts derived from existing regional services using satellite remote sensing, weather observations or crop model simulations become available one to two months before harvesting Schauberger et al., 2017;Skakun et al., 2017;van der Velde and Nisini, 2019 . USDA forecasts are based on field measurements and producer surveys and thus are extremely accurate, but the first release of these forecasts for the summer crop in the US occurs midseason August Egelkraut et al. 2003;Holland, 2011;Iizumi et al., 2021a . Using the NARO-APCC forecasts together with existing regional service forecasts enables users to access objective yield forecast information anytime during the entire season from planting to harvesting.
The NARO-APCC state-level forecasts were often found to be more skillful than the simple forecasts for quite a few states within a country throughout the season, even when the NARO-APCC's national-level forecasts failed to outperform the simple forecasts Figs. S1 -S10 . The potential to use the NARO-APCC state-level forecasts for user-specific applications, such as forecast-based risk transfer and financing Carriquiry and Osgood, 2008;FAO, 2021 , is worth exploring, in addition to the current goal of the NARO-APCC joint research program of inputting national-level forecasts for the intergovernmental agricultural market monitoring initiative AMIS through its technical partners, including the Group on Earth Observations GEO Global Agricultural Monitoring GEOGLAM initiative Becker-Reshef et al., 2019 andAsia-RiCE Oyoshi et al., 2014 . There is room for further improvements to the NARO-APCC forecasts. The season length of a crop sometimes spans over six months when multicropping occurs. Double maize cropping in Brazil and Paraguay is an example. However, the NARO-APCC forecasting system uses seasonal temperature and precipitation forecasts for the six months, and thus, it is currently impossible to cover the entire time period of multiple seasons of an intended crop that is required to properly calculate the national average yield anomaly. Temperature and precipitation forecasts over six months are available for some major weather centers but not at the APCC at the time of writing. For instance, the Japan Meteorological Agency/Meteorological Research Institute-Coupled Prediction System version 2 JMA/MRI-CPS2; Takaya et al., 2018 provides forecasts covering 241 days, including the initial dates approximately eight months . Although the skills of temperature and precipitation forecasts at lead times longer than six months are not as high as those with shorter lead times Takaya et al., 2018 , climate oscillation forecasts that predict indices of oceanic and atmospheric oscillations, such as the El Niño-Southern Oscillation, are accurate even at lead times longer than six months Luo et al., 2005 . Such forecasts could be utilized instead of temperature and precipitation forecasts when predicting yields over six months before harvesting, as discussed in the literature Yuan and Yamagata, 2015;Ceglar et al., 2017;Nobre et al., 2019;Iizumi et al., 2021b . Although crop calendars are a key input to the NARO-APCC system, reliable information is hard to access at the global scale, as highlighted in the discrepancies between the different sources of information. Apart from this limitation, we note that two specific issues need to be addressed in future research. The crop calendars used in the NARO-APCC system Sacks et al., 2010 contain no information on the spatial variations in harvesting dates for the crops grown in the countries studied here Fig. S11 . Using a single planting month within the NARO-APCC system may cause forecast errors when a long harvesting window plays a role in determining regional and national average yields Hasegawa et al., 2008 . Satellite remote sensing is powerful in mapping crop type and phenology, but within-season mapping remains challenging Becker-Reshef et al., 2018 . Annual crop type mapping has recently become feasible for specific regions Song et al., 2021 . More importantly, crop forecasting based on seasonal climate information is conducted at the beginning of the season, for instance, before heading, at which point crop phenological patterns are not yet identified from satellite vegetation indices. Satellite-based forecasting is currently conducted 1.5 -2.5 months before harvesting Skakun et al., 2017, Becker-Reshef et al., 2018 Last, soybean harvest areas have been rapidly expanding in the last decade Song et al., 2021 compared to those of the other crops considered here. The soybean harvested area map in 2000 used in the NARO-APCC system might not well represent the current conditions. Replacing the harvested area maps currently used in the NARO-APCC system Monfreda et al., 2008 with more recent maps Yu et al., 2020 might increase the forecast skills, especially for soybean.

Conclusions
This study assesses the NARO-APCC yield forecast skills of major commodity crops at the state level for the 2019/20 season in Australia and the 2018/19 season in Argentina and Brazil.
The national-level assessment is presented for the 2019/20 season in Uruguay and the 2018/19 season in Paraguay. The results indicate that the NARO-APCC forecasts can capture the major characteristics of the reported state yields even just after planting. In three-fifths of the crop-state combinations evaluated here, the NARO-APCC forecasts outperform the simple forecasts derived solely based on the reported yields. These findings underline the value of long-range crop forecasts that long-range forecasting services such as the NARO-APCC forecasts and existing regional services together provide to users so that they can access objective yield forecast information for almost the entire season from planting to harvesting, ultimately contributing to increasing the preparedness of players in global food systems for production shocks and supply disruptions induced by seasonal climate extremes. The NARO-APCC forecasts predominantly rely on climate forecast information, enabling forecasts to be provided even just after sowing. However, notably, this advantage comes at the cost of relatively larger forecast errors in the NARO-APCC forecasts compared to those in more accurate forecasts conducted close to harvesting based on multiple sources of information. The low skills of these forecasts for soybean need be improved as soybean is a major crop in many Southern Hemisphere countries.