To answer questions about how good a forecast is, the forecast is compared to what actually happened. With weather forecasts this is not a simple process. It is not always a case of being right or wrong; it is useful to know how close the forecast was to the actual weather. Therefore we use a method of verification that measures how close the forecast is to the observed weather.

The Monthly Outlook provides forecasts of expected temperature, rainfall and sunshine categories. Temperature and rainfall forecasts use five categories; (1) well below average, (2) below average, (3) near average, (4) above average or (5) well above average conditions for the time of year. Sunshine forecasts use three categories.

To assess the accuracy of the forecast we compare the predicted category with the category that was actually observed to occur. We use a points-based scoring system in which maximum points are awarded to forecasts that are 'spot on' (i.e. the forecast category exactly matches the category that actually occurred), fewer points are awarded for 'near misses' (e.g. the forecast is wrong by one category), and points are subtracted for misleading forecasts (i.e. a forecast of above normal when below normal is observed). The score used is called the Gerrity Skill Score (GSS), and is one of the scores recommended by the World Meteorological Organization (WMO) for evaluation of long-range forecasts. The score is designed so that forecasts that are always 'spot-on' would achieve a score of 1.0, and forecasts based on simply 'forecasting' the long-term average (category 3) would receive a score of zero. Thus a positive score means the forecast is better than guesswork and better than assuming future conditions will be similar to the long-term average.

It is important to assess the performance of long-range prediction systems over a large number of forecasts, since good (or bad) performance over one or two forecasts may not reflect the long-term performance. The bar chart (Fig. 1) shows Gerrity Skill Scores calculated over 45 forecasts issued for each of the 10 UK regions between May 2002 and January 2004. The scores shown are for mean temperature and precipitation for the three periods used in the Monthly Outlook: days 5-11 ahead, days 12-18 ahead and days 19-32 ahead.

Fig. 1: Gerrity Skill Scores for mean temperature and rainfall

Scores are positive over all periods, showing better skill than assuming conditions will be close to the long-term average. Better accuracy is found for temperature in comparison to rainfall and, as expected, best accuracy is for the 5-11 day ahead period. For temperature, the slightly better scores for the longer-range period arise because it is often easier to forecast two-week averages compared to one-week averages. For precipitation, the assessment shows there is benefit to be gained from forecasts out to the 12-18 day range, but that scores for the 19-32 period are only slightly better than climatology.

To gain a more visual impression of the performance, Fig. 2 gives a comparison of forecasts for mean temperature (12-18 days ahead for south-east England) with the temperature that actually occurred. The orange lines show the forecast values and black lines the observed temperature. The four thin black lines show the temperatures that define the five temperature categories.

Fig. 2: Comparison of forecast mean temperature for the period 12-18 days ahead for south-east England with the actual mean temperature during the period. Forecasts shown are those issued between 29 January 2003 and 10 March 2004. Thin black lines divide the five categories: well below average, below average, average, above average and well above average for the time of year.

It may be seen that the forecast is rarely wrong by more than one category, with generally good advance guidance given for the above average or well-above average conditions prevailing throughout Summer 2003, and the variable conditions during Winter 2003/4.

More about the Monthly Outlook features