Met Office Accuracy II

Guest Post by Charles Duncan

An update to previous analysis and also this time looking at temperature.

A few days ago I posted an analysis of the Met Office’s rainfall forecasts.  I realised after the event that I had used actuals from England and Wales, not the UK, so here’s the revised version.  This time we are comparing apples with apples!

Using the 3-month outlooks[i], published monthly by the Met Office, I looked at the most likely rainfall and temperature for the first month of each; in other words their forecasts in a window of one to six weeks out.

I then compared these to the actuals, also from the Met Office website[ii],[iii].

Rainfall

Rainfall is very variable, and therefore harder to predict, than temperature, which varies little year on year.  The Met Office’s forecast seems to be based on previous years (1910 – 2011), with a correlation coefficient between the two of 0.77.  In the event the actuals were, of course, much higher than either previous averages or the forecast:

rain comparisonAs a result, the correlation between the forecast and actual was a poor 0.14:

rain correlationTemperature

By contrast, a scatter plot of forecast vs actual temperatures yields an impressive correlation coefficient (R2) of 0.95:

temperature correlationIn the graph below, the average is the mean temperature from 1910 to 2011, with the standard deviation shown as error bars.  Also plotted are the Met Office’s forecast and the 2012 actual temperatures:

temperature comparison

The correlation coefficient between the historical average and the forecast is 0.985, and between the mean and the actual is 0.94.  This suggests the forecast for 2012, whilst accurate, was not significantly better than just taking the average of previous years’ temperatures.

However it is not as simple as that; the average is affected by the post 1970 warm period.  The average of the whole data set is 8.5°C, but that from 1997 is 9.25°C.  Furthermore 2011 was a warm year (average 9.64°C).

Generally temperature varies little from year to year; the standard deviation of year-on-year changes is just 0.35°C.   The Met Office’s forecast for 2012 of 8.8°C was not only more that 2 standard deviations from 2011, but surprisingly accurate; 2012 turned out to have an average of 8.7°C.

Summary

The forecasts and actuals for 2012 suggests the Met Office are reasonably accurate at predicting UK temperatures for the next month, but woeful at predicting rainfall.

About these ads
This entry was posted in Weather and tagged , , , , . Bookmark the permalink.

16 Responses to Met Office Accuracy II

  1. j ferguson says:

    Verity,
    Is it possible that the difference in accuracy of prediction is because rainfall can vary greatly over short distances, more so than temperature? The idea would be that the collection points are specific but might not, because of their locations, accurately indicate what had happened. It makes sense that this might be recognized and washed out (so to speak) in the analysis. It also makes sense that the variation would not be the same from rainfall to rainfall, but ???

    I mention this because of an experience I had during a construction project 60 miles northeast of Houston, TX. The forms for a foundation pour had been washed away by what was reported to be fourteen inches of rain over 24 hours – sounded like a lot. We were in Chicago, and didn’t believe it. The official report for Cleveland, TX about 4 miles away was seven inches.

    The superintendent realized we didn’t believe him. Next morning a box arrived Fed-Ex with a photograph of the overflowing job-site rain gauge tied to a rock and a note directing us to go outside and throw the rock with photo attached through the boss’s window.

    The photos of the job site underwater were pretty convincing.

    maybe this is all wet.

    • Charles Duncan says:

      Good point. But hopefully there are enough weather stations for this to average out. If there aren’t then I guess it questions the value of any of their rainfall forecasts or measurements!

      • Verity Jones says:

        IIRC the UK has the highest density of rain gauges in the world although I noted that there was a dramatic increase over the 20thC that might have affected the chances of catching a heavy downpour – https://diggingintheclay.wordpress.com/2012/12/30/the-rain-gain-refrain/#comment-3301.

        It occurs to me that size and location are so important. For example the UK is about a third the size of Texas (for example) in area but in the same ball park of length (650 vs 790miles) and half the width (300 vs 660miles). The northern hemisphere polar jet stream wobbles between latitudes 30°N and 60°N ~2000miles. If models rely on prediction of air mass movement and the location of the jet stream, and perhaps the AO or NAO their accuracy will be based on historical data. If as I tend to think natural cycles of more than 30 year climatology are at work and we are into a new cycle, then forecasting does have to get used to a ‘new norm’, but that would not strongly (if at all) be connected to CO2 concentrations.

      • j ferguson says:

        I supposed it possible that the forecast wasn’t bad, but the reports of amount of rain defective for the reasons suggested. Obviously reality not agreeing with the models is not impossible and the error not necessarily in the model. Wouldn’t it be nice if we could get good measurements of volume for water and energy instead of temperature?

  2. j ferguson says:

    Charles,
    Do forgive me for not addressing my question to you – damned bi-focals again. Thank you for an interesting post.

  3. Perhaps the difference in accuracy is simply that temperature is environmental and rainfall is incidental (requires a trigger). Wind is also environmental, and relatively easy to predict, particularly if a met chart with isobars is to hand. It’s likely no coincidence that GCMs can predict (sorry, project – silly me) temperature far better than rainfall, though neither at all well. Anyone who believes GCMs “project” climate well need to bear in mind they don’t “do” clouds well at all, and as clouds virtually control climate….

    I’d never want to get into the prediction business. If you’re proved right it’ll be put down to luck or coincidence, and if you’re proved wrong you’ll be laughed at – unless your surname is Erlich or Hansen in which case sycophants will just ignore your signal failure, and wait breathlessly for your revised prediction. “Scientists say it’s worse than they thought” doesn’t say much for what they originally thought, and not very much for what they think now. “Scientists were surprised to find….” begs the question “why were they surprised?”. Was it preconceived notions, lack of knowledge, misunderstanding, or what?

  4. Guy Leech says:

    Could you explain why rainfall for a large area like the UK is reported in linear units not volume? A measurement in mm for a single point seems to mean something but it doesn’t have a meaning for a large area over which a volume of rain will have fallen, not a length. Seems an obvious question but I can’t find an answer anywhere?

  5. Charles Duncan says:

    Guy,
    I guess because it’s just the average of the rain gauge readings. Also I’m not sure saying that the UK had 243,000,000,000 cubic metres of rain means much to people either! ;-)

  6. j ferguson says:

    Floodplain capacity is measured in acre-feet in the US which seems a pretty obvious volume measurement which ought to be meaningful to most people. One might suppose that rain gauges could be calibrated by comparison between rain-gauge readings in an area and the amount of water collected in basins such as lakes, etc. once the ground is saturated and all other water must run-off.

    Maybe the error in gauge readings is low enough not to make this worth doing.

    • Verity Jones says:

      I can visualise a cubic metre, or foot, but acre-feet are too big to mean anything really.

      • j ferguson says:

        Acre-feet are what you walk through when your neighborhood is flooded, although I have to agree with you that it’s a pretty abstract metric – unless you plan industrial subdivisions (estates?) and have to worry in those figures.

  7. Guy Leech says:

    Re Charles Duncan’s comment, I do think that a volume of rainfall over an area like the UK would mean much more than an average of linear single point measurements, which simply does not tell anyone how much rain has fallen in a large area. Rain is water which has a volume! A volume of rainfall has a physical meaning which the average of a number of single point linear measurements, expressed as a linear measurement, simply does not.

    Having just had to pump water our of our cellar because the groundwater levels are higher than I have known them for years, I can see that this is a result of the volume of water entering the ground being larger than the volume descending from the ground under our house to the river at the bottom of our valley and from there to the sea.

  8. tsuhtt1 says:

    [Thanks for commenting, however your comment was off-topic for this thread. I have moved it to here: https://diggingintheclay.wordpress.com/tips-and-ot/#comment-4131. Verity]

  9. Doug Cotton says:

    [Snip - off topic]

  10. Joel Heinrich says:

    as to why 1 mm. It is actually 1 mm / m², or 1 litre per m². But it doesn’t matter what your ground area is, it is always the same hight (or depht) of the watercolumn that will be falling. So 25.4 mm is just the same as 25.4 mm / m² or 25.4 mm / ft² or 1 inch / ft² or 1 inch / acre. Your acre will be flooded by 1 inch of water.
    What I find more interesting is that the UK has the lowest amount in spring and the most amount in winter, while in much of central europe it’s the highest amonut of rain in summer and the least in winter.

Comments are closed.