To answer questions about how good a forecast is, the forecast is compared to what actually happened. With weather forecasts this is not a simple process. It is not always a case of being right or wrong; it is useful to know how close the forecast was to the actual weather. Therefore we use a method of verification that measures how close the forecast is to the observed weather.

The Monthly Outlook includes forecasts of expected temperature and rainfall categories. Five categories are used; (1) well below average, (2) below average, (3) near average, (4) above average or (5) well above average conditions for the time of year.

To assess the accuracy of the forecast we compare the predicted category with the category that was actually observed to occur. We use a points-based scoring system in which maximum points are awarded to forecasts that are 'spot on' (i.e. the forecast category exactly matches the category that actually occurred), fewer points are awarded for 'near misses' (e.g. the forecast is wrong by one category), and points are subtracted for misleading forecasts (i.e. a forecast of above normal when below normal is observed). The score used is called the Gerrity Skill Score (GSS), and is one of the scores recommended by the World Meteorological Organization (WMO) for evaluation of long-range forecasts. The score is designed so that forecasts that are always 'spot-on' would achieve a score of 1.0, and forecasts based on simply 'forecasting' the long-term average (category 3) would receive a score of zero. Thus a positive score means the forecast is better than guesswork and better than assuming future conditions will be similar to the long-term average. Although the theoretical maximum score is 1.0, best scores achieved at the monthly range are of order 0.6, and found in the more predictable tropical regions.

Long-term assessment

It is important to assess the performance of long-range prediction systems over a large number of forecasts, since good (or bad) performance over one or two forecasts may not reflect the long-term performance. The bar chart shows Gerrity Skill Scores calculated over 115 forecasts issued for each of the 10 UK regions between June 2002 and September 2005. The scores shown are for mean temperature (Tmean) and precipitation for the three periods used in the Monthly Outlook: days 5-11 ahead, days 12-18 ahead and days 19-32 ahead.

Best skill is found for the temperature forecasts and, as expected, for the 5-11 day period. At longer ranges, scores for Tmean in 12-18 day period show best skill, and an example of a successful forecast at this range is given below. At the 19-32 day range scores are positive but indicate at best only marginal benefit over use of climatology.

Gerrity Skill Scores for mean temperature and rainfall

Case study - late-winter 2004/5 cold snap

The late-winter 2004/5 cold snap over the UK in 2005 was anticipated by the Monthly Outlook nearly two weeks in advance. The Monthly Outlook issued on 11 February stated for the 12-18 day period (21-27 February) that '...a sudden change to below or well-below average temperatures is expected...'. The left-hand figure shows the predicted category for maximum temperature for the 10 UK districts, the right-hand figure shows the category that later occurred. The correct category was predicted in the southern and central districts. In northern districts cold conditions were predicted, but underestimated by one category.

Fig. 1: Gerrity Skill Scores for mean temperature and rainfall

More about the Monthly Outlook features