MetService gets call for an inquiry: One News

Willoughby · Unread post by **Willoughby** » Fri 30/12/2005 00:46

Seen it a lot on here on the forum the last couple of weeks, especially for the Upper North Island, other people see it too!

There's a claim the MetService is getting the weather forecast wrong half the time and now a man who keeps a close eye on the weather watchers is calling for an independent inquiry.

According to a study by a pilot of 40 years and an education in meteorology, MetService gets it's forecast wrong 50% of the time.

Whenever the forecast is wrong it is frustrating in the least, but potentially disastrous if your livelihood depends upon the accuracy of the weather.

"We can get caught with a gale force wind which will blow the heads of this crop on to the ground, so it's very important that we have an accurate forecast," says farmer Andrew Gallanders.

The MetService forecast is a topic of fascination for Auckland helicopter instructor John Clements.

He's done his own survey, comparing weather predictions for Auckland with what actually happened and found the forecast and reality are worlds apart.

"On 50% of occasions the forecast were as if they were for a different planet...50% were not right.," Clements says.

A state owned enterprise, the MetService rates itself much higher at more than 90% accurate.

Forecaster Bob McDavitt says that's good going for an island country with wild geography, making it difficult to predict how a weather pattern will behave.

"Our job is to capture the pattern and evolve it in to the future with the computers, which we do quite nicely but chaos keeps changing things, so weather forecasters are never going to be 100% right," McDavitt says.

Clements says while the Metservice may consider it a success if they forecast showers and there is a shower in the Auckland area, he says he would count that as a failure

In Britain and Japan, which are geographically similar to New Zealand, forecasts are more detailed. Here the Metservice concentrates on an area of 60 nautical miles and is yet to get bigger better computer power.

"We're getting better but resources are always a drag point for meteorology," McDavitt says.

But is that good enough for the thousands of people that rely on the weather for their living? From builders to boaties, farmers and contractors, is it good enough when considering the financial cost to the country in lost earnings?
-One News
Video link: http://www.tvnz.co.nz/view/video_popup_ ... width=128k

Come on Bob, the resources are there for sure. The BoM uses pretty much the same technologies as the MetService from what I gather.
Maybe the champs model needs a review?- Seen far too many occasions when the Global models beats the Champs/MM5 model in localised heavy rain/thunderstorm events and Tmax's during summer.

As for '90% accuracy' where do they get this info from - METAR observers?!

I could see that as a reality for Wellington.. not up here.

It would be good if some MetService employees post here on this matter...? (Bob?)

TonyT · Unread post by **TonyT** » Fri 30/12/2005 08:04

Foggy Hamilton wrote:
Clements says while the Metservice may consider it a success if they forecast showers and there is a shower in the Auckland area, he says he would count that as a failure

Thats the funniest thing I have read in ages. If I ask him to fly his helicopter to Wellington and he lands at Rongotai airport, thats a failure too? These crackpots surface from time to time and always manage to blow their credibility with statements like this, usually because they are obsessed with the subject and have nothing better to do than complain. To them the glass is always half empty, never half full. I too suspect that MS forecasts are less than 90% accurate, but I'm not going to very far provinbg it with statements like that!

I think the broader issue is that forecasting has become much more accurate in the last 10-20 years, and people are starting to rely on it. This is a dangerous thing, and something we always caution our clients against. Yes, we strive to do the best we can, but we know, and so should all forecast users, that forecasts will be wrong sometimes. Our knowledge of the atmosphere is not perfect. The problem is, that forecasts are more likely to be wrong at times of severe (ie high energy)weather, and those are precisely the times when forecast failures are more keenly felt.

Users might go for 10-15 days with accurate forecast after accurate forecast, and then get 2-3 days in a row when the forecast misses the mark. Because they have become conditioned to being able to rely on the forecast being right, they can come badly unstuck on the days when it goes wrong.

I think another problem is the destructive language used by TV weather presenters, and perhaps this is in part a problem MS has brought upon itself. Even though we all know the TV weather is the worst forecast, its still the case that for most people its their main (and often only) source of a forecast. Now when we have presenters talking about "scraps of cloud to upper regions", "a few scraggy showers lingering", "frontal debris blotting the landscape for most of the day", etc etc its not surprising that people struggle to work out what their weather forecast is. The meaning inherent in the words so carefully chosen by forecasters is totally destroyed in the name of entertainment and one-upmanship. And not helped by graphics which are at best confusing and at worst downright misleading (why do we have a forecast for Wellington of fine weather which shows rain falling at the top of the graphic?)

Finally, have any of these people who like to complain about the forecasts actually tried doing the job? Its not easy, as I'm sure every one on this forum knows. You dont become a weather forecaster if you cant handle being wrong sometimes. By the same toekn, you shouldnt use a forecast unless you accept it will be wrong sometimes too. For every person that complains we have dozens and dozens of clients who are satisfied (and I'm sure MS do as well), because they know the limitations of what we can do, and use the products accordingly.

RWood · Unread post by **RWood** » Fri 30/12/2005 11:25

Well said. My experience locally suggests that MetServ. is "right" over 90% of the time for here. At shortish time ranges I find I can hear their early morning prediction, look at conditions at say 8 AM and on a rainfall basis for that day, get surprised on at most 1-2 days per month, though I regularly make my own re-estimate of when rain is likely to start. The only vulnerability I've noticed regularly is the over-forecasting of rain in NE situations where the promised rainfall either never arrives, or is late, or is very short-lived. The forecasting of situations here with critical SW flows is a lot better than it was 20 years ago.

I do sympathize with anyone in the forecasting business, there's no doubt that many now regard forecasting as something that should be infallible.

RWood · Unread post by **RWood** » Fri 30/12/2005 11:27

Well said. My experience locally suggests that MetServ. is "right" over 90% of the time for here. At shortish time ranges I find I can hear their early morning prediction, look at conditions at say 8 AM and on a rainfall basis for that day, get surprised on at most 1-2 days per month, though I regularly make my own re-estimate of when rain is likely to start. The only vulnerability I've noticed regularly is the over-forecasting of rain in NE situations where the promised rainfall either never arrives, or is late, or is very short-lived. The forecasting of situations here with critical SW flows is a lot better than it was 20 years ago.

I do sympathize with anyone in the forecasting business, there's no doubt that many now regard forecasting as something that should be infallible.

Gary Roberts · Unread post by **Gary Roberts** » Fri 30/12/2005 13:27

TonyT wrote:Finally...You dont become a weather forecaster if you cant handle being wrong sometimes.

My problem with your statement is that Met people refuse to admit they are ever wrong. Everybody else is always wrong, but never Met. In their world, nobody else is capable of doing the job; archaic Met service data collected by somebody long dead, applying who-knows-how rigorously enforced methodologies, standards and practices is always 100% accurate, yet data collected by anybody else under any circumstances is apparently inaccurate by definition, and to be safely ignored.

Yes, meteorology and weather forecasting can be a fiendishly difficult discipline, and one I'm not interested in having a crack at...but Met people need to admit to themselves sometimes that they aren't the utterly infallible intellectual superior beings they apparently like to believe themselves to be. They claim to practise - and demand - the highest of standards, yet we have no reason to believe they practise what they preach, other than their own claims.

The Met Service dismisses out of hand anything that doesn't originate with the Met Service. I'll give a sh*t what they think once they discover a little humility.

TonyT · Unread post by **TonyT** » Fri 30/12/2005 16:16

OK. Thats your position very clear then Gary... Seems like a raw nerve has been touched.

tich · Unread post by **tich** » Fri 30/12/2005 17:56

The MetService forecasts freely issued to the publc are mostly the short ones, which often summarise what's predicted for a general area, instead of mentioning differences within an area, which can be quite variable. For example a short forecast of southerlies and showers for Malborough often ignores the fact that showers are much more likely about the Kaikoura Coast than inland Malborough.
Also, timing in these forecasts can be vague, eg a forecast of 'showers developing tomorrow' - when are they expected to start developing tomorrow? Morning, midday, afternoon?

Storm Struck · Unread post by **Storm Struck** » Fri 30/12/2005 18:22

The only thing i have to complain about at the moment is the CHCH rain radar once again down

and also to the fact of when there are storms it goes down.
Not that we need it because we can judge for ourselves anyway.
Cheers
Jason.

Willoughby · Unread post by **Willoughby** » Fri 30/12/2005 23:12

Well said gary! don't fully agree with you're first statement however.

NZ Thunderstorm Soc · Fri 30/12/2005 23:19

tich wrote:The MetService forecasts freely issued to the publc are mostly the short ones, which often summarise what's predicted for a general area, instead of mentioning differences within an area, which can be quite variable. For example a short forecast of southerlies and showers for Malborough often ignores the fact that showers are much more likely about the Kaikoura Coast than inland Malborough.
Also, timing in these forecasts can be vague, eg a forecast of 'showers developing tomorrow' - when are they expected to start developing tomorrow? Morning, midday, afternoon?

Yes Ben Some of the short forecasts are quite vague.
Sometimes they forecast southerlies and showers clearing when they haven't forecasted them to happen in the first place???.
I hate the term 'Thundery showers'. It's either thunderstorm or not.

JohnGaul
NZTS

Weathermad · Unread post by **Weathermad** » Sat 21/01/2006 07:00

Since this report aired,I have been keeping my own records in Christchurch and evaluating them with forecasts prepared by the MetService and also Blue Skies.

At 7am, I have jotted down the forecasts and temperatures for the day ahead and the following morning, written down what has actually occurred. I have not included any updated forecasts during the day, as I have taken the stance of someone wanting to know what the weather was going to do before they went to work,school etc..

The results are still a little premature but forecast providers are in the 75-80% range.

When it comes to what the weather will do for the following day and the longer range forecasts, that accuracy falls away quite dramatically

I will continue to collate data for a while as I believe much more is required to become somewhat useful and accurate.

Weathermad

GraemeWi · Unread post by **GraemeWi** » Sat 21/01/2006 07:43

Weathermad wrote:Since this report aired,I have been keeping my own records in Christchurch and evaluating them with forecasts prepared by the MetService and also Blue Skies.

At 7am, I have jotted down the forecasts and temperatures for the day ahead and the following morning, written down what has actually occurred. I have not included any updated forecasts during the day, as I have taken the stance of someone wanting to know what the weather was going to do before they went to work,school etc..

The results are still a little premature but forecast providers are in the 75-80% range.

When it comes to what the weather will do for the following day and the longer range forecasts, that accuracy falls away quite dramatically

I will continue to collate data for a while as I believe much more is required to become somewhat useful and accurate.

Weathermad

How about making one of those 'spin the wheel' type devices to produce a random forecast as a control test. Would be interesting to see how that stacks up as well.

dogmelon · Unread post by **dogmelon** » Sun 22/01/2006 13:24

The chc rain radar is an excellent indicator of thunderstorm activity, when it goes off you know there are thunderstorms about.

NZ Thunderstorm Soc · Sun 22/01/2006 19:23

dogmelon wrote:The chc rain radar is an excellent indicator of thunderstorm activity, when it goes off you know there are thunderstorms about.

Ah..well, if thats the case it should be alright for the next few weeks, whatever, well, they might as well switch it off anyway, nothing on the horizon anyway

JohnGaul
NZTS

TonyT · Unread post by **TonyT** » Mon 23/01/2006 18:55

Weathermad wrote:Since this report aired,I have been keeping my own records in Christchurch and evaluating them with forecasts prepared by the MetService and also Blue Skies.

Where are you getting the Blue Skies forecast from Richard?

03Stormchaser · Unread post by **03Stormchaser** » Mon 23/01/2006 19:36

TonyT wrote:
Weathermad wrote:Since this report aired,I have been keeping my own records in Christchurch and evaluating them with forecasts prepared by the MetService and also Blue Skies.

Where are you getting the Blue Skies forecast from Richard?

Didnt think Blueskies did a Christchurch forecast??

TonyT · Unread post by **TonyT** » Mon 23/01/2006 20:13

There is one on our Weatherline, 09004CAST, but we dont have one on the 'net. Thats why I'm curious where Richard is getting it from.

Thunder · Unread post by **Thunder** » Mon 23/01/2006 23:25

I'm not giving any to him!

I think Richard is just looking at our Canterbury forecasts, that's all. I guess?

Cheers

spwill · Unread post by **spwill** » Tue 24/01/2006 09:21

Aaron said,

our Canterbury forecasts

Another person we can blame.

Weathermad · Unread post by **Weathermad** » Wed 25/01/2006 06:33

Hi Tony,

Sorry about the delay in replying!

I have been using the Blue Skies forecast that is posted on the Fencepost site each day.Is this the best one or are there more detailed forecasts available elsewhere?

At this stage,you have been doing a great job with nearly 80% accuracy(for the 3 and a half weeks of January).

Hope all is well with you and the family

Richard

Weathermad · Unread post by **Weathermad** » Wed 25/01/2006 06:38

Thanks...

Just saw your 0900 number and I see that the forecasts are for Canterbury on the site rather than Christchurch....The minimum and maximum temperature provided, where is this location?

Well if you are getting this sort of accuracy for the region, I'm sure it'll be even higher with the city forecasts you provide!

Richard

Manukau heads obs · Wed 25/01/2006 07:01

tonyt did underesti this low that just hit the upper NI by a fair degree....
why was thart tonyt, when it was discussed even here that it was going to pan out just like it did?

TonyT · Unread post by **TonyT** » Wed 25/01/2006 07:53

I dont think we underestimated it at all, just got the timing wrong by about 12 hours. Thought the heaviest rain would be late in the day and at night, it came through earlier. In all other respects I think we got the winds right and the rain amounts pretty right (as much as you can with the variability that these systems bring across a region).

TonyT · Unread post by **TonyT** » Wed 25/01/2006 08:02

Weathermad wrote:Hi Tony,

Sorry about the delay in replying!

I have been using the Blue Skies forecast that is posted on the Fencepost site each day.Is this the best one or are there more detailed forecasts available elsewhere?

At this stage,you have been doing a great job with nearly 80% accuracy(for the 3 and a half weeks of January).

Hope all is well with you and the family

Richard

Things are going very well with us thanks. Eddie is doing well on the treatments and we only have to go into hospital once a month now.

Just so you realise, the Fencepost forecasts are not updated by 7am, so if you use that one at that time, it was made at 6pm the previous evening. So, a comparison with MS's forecast would be fairer if you used their prediction from the previous evening, not their morning update. There is also the forecast at http://www.ats.co.nz which is often a little more detailed, and specific for Mid Canty.

As for the max and min, well you can imagine the issues trying to do max/min temps for a region rather than a location, but thats what we were asked to do. For instance, on Monday it was 30 in Culverden, 32 in Waipara, 24 in Amberley, 21 at Amberley Beach... We predicted 27 and feel that was a good result! I guess its aimed for the middle plains. I think you will usually find that predicted max will be higher than what Chch gets (the cool easterly effect) and the min possibly a little lower, but they should be close.

Manukau heads obs · Wed 25/01/2006 08:33

i disagree tony
yesterday the fence post site only said fresh NE winds and the rain recorded inmany places was 2x that you had there

NZ Weather Forum