Standard forecasts used to measure skill
|
|
For continuous variables, there are two common standard (unskilled) forecasts used to measure skill,
"climatology" and "persistence". The climatology standard is widely used and provides a simple indication
on the limits of predictability. If, for example, the skill against climatology reduces to 0 after 7 days, then it
can be said that the phenomenon being evaluated has a predictability limit of 7 days according to this
particular verification dataset.
A "climatology" forecast means a forecast of the same value on every occasion, the climatological mean of
the forecast variable. When evaluating skill scores, it is important to know which climatology is used: The
climatology may simply be the average over the data used in the verification, the "sample climatology".
Or, it may be the published long term average for a particular location and season or day of the year. This
is referred to as "long term climatology". The former is far easier to compute, because it can be obtained
during the verification process using only the verification data sample. Long term climatology must be
obtained from another source, usually the climatological records of a national meteorological service. The
skill score calculated with respect to sample climatology is sometimes called the "variance explained" or
the "reduction of variance". From a forecaster’s point of view, the use of sample climatology can be
regarded as unfair because this quantity is unknown when the forecast is made.
The persistence standard is used to assess the ability of the forecast to predict change in the weather. As
the forecast projection gets longer, change in the weather becomes inevitable, which means persistence
forecasts are most useful to assess very short range forecasts. A persistence forecast by definition is the
same as the last observation, that is, forecast no change. For maximum temperature, persistence would be
"yesterday’s maximum". A forecast will be skilful against persistence if changes in the weather are forecast
with some accuracy.
"Tricks of the trade" (strategies which make skill scores look better):
If climatology is used as the standard score, one should know or ask "which" climatology. Skill scores
against climatology will look better (higher values) if
1. Climatology is averaged over many stations or over a long period of time. Then one gets credit
for knowing that the average weather at one’s favourite station is different from other stations.
Ask if it was the climatology of the day and station that was used.
2. Sample climatology is used instead of long-term climatology. The result will be
a better (higher) score, because the portion of the error attributable to the difference between the sample
climatology and the long term climatology has already been removed before the skill is computed.
If persistence is used as the standard, one should make sure the persistence forecast chosen does not contain
known changes such as diurnal temperature variations. For example, the persistence forecast for the 00 UTC
temperature should be yesterday’s 00 UTC temperature.
Loading Questions
...
The above verification results show skill scores against climatology (left) and
against persistence (right) for two years. The skill score used is based on the mean absolute error.
Look carefully at the above verification results, averaged over the same three stations and indicate
whether the following statements are true or false.
Probably correct, but cannot be assumed. The MAE for climatology could have been worse (higher) in 1995,
giving higher skill even if the MAE for the forecasts was also higher.
Correct, according to the right hand graph. Both curves indicate positive skill at all forecast ranges.
Correct. The skill with respect to persistence decreases with forecast projection. This is different from
the usual situation, where persistence is difficult to beat at very short ranges but easier at longer ranges.
Incorrect. There is no clear evidence for this. Highly variable weather lowers the skill of (constant)
climatological forecasts, so the balance of evidence would favour the opposite conclusion. But one has
to consider the accuracy of the forecast too. Greater variability should also lower the accuracy of
persistence forecasts.
Probably true, since that is the usual case. If so, then the skill score shows the percentage of variance
explained each year. The difference between the sample and long term climatology is not counted, so the
forecaster does not get credit for correctly forecasting the average departure from climatology over the
verification sample.
Correct. Definitely. Questions to ask:
1. Long-term or sample climatology?
2. Climatology averaged over the 3 stations or separate for each? (That is very important)
3. Sample size on which each point is based?
Can you think of others?
Likely. However, the skill involves a ratio, so higher skill can be obtained either by lower
forecast error or higher error for climatology forecasts or both.
Correct. There is more evidence in favour of false than true (All else being equal, higher
variability lowers the climate score, and raises skill). However, one needs to know the MAE
of the forecasts too.
Probably incorrect. It is not stated which climatology is used. The graphs could be better
interpreted if this information is given.
