The Continuous Rank Probability Score (CRPS): Definition and use
|
|
The CRPS is related to the rank probability score, but compares a full distribution with the observation,
where both are represented as cumulative distribution functions (cdfs). The equation for calculation of the CRPS is,
Where,
is the forecast probability cdf for the ith forecast case and
is the observation, expressed as a cdf.
If the observation is of a specific value, as shown on the left panel below, then the corresponding cdf is a single step-function
with the step from 0 to 1 at the observed value of the variable. (This is the Heaviside function.)
The diagram on the right below shows schematically the computation of the CRPS, which is the total area
between the cdf of the forecast and the cdf of the observation.
The calculation of the CRPS will result in a value in the units of the forecast variable, temperature in the example below.
In practice, the CRPS is often computed discretely, since observations and forecast distributions are reported in discrete intervals.
The reference Hersbach (2000) gives useful guidance on the computation of the score using a discrete representation of the forecast cdf.
Computer programs are now available for computation of the score. If the forecast is fit to a distribution such as the normal
distribution, then a more continuous computation is possible.
One advantage of the CRPS is that it reduces to the mean absolute error (MAE) if the forecast is deterministic.
To understand this, picture the right hand graph below as the cdf for the forecast becomes a step function, like the observation.
Then the area between the forecast and observation is given by the rectangle formed by the two step functions,
which has width equal to the distance between the two (in the units of the abscissa), and unit length. This is the MAE.
In practice, this makes it possible to compare an ensemble forecast with a deterministic forecast of the same variable
in a consistent fashion.
Now let’s look at the interpretation of the CRPS via an exercise.
This graph shows CRPS results for 4 different ensemble post-processing methods that were being compared.
These were comprised of two different techniques, Bayesian Model Averaging (BMA) and simple Gaussian dressing of the
ensemble mean ("Gauss"), each applied to 16 member ensemble forecasts and 18 member ensemble forecasts formed by adding
the unperturbed control and the full resolution deterministic forecast to the 16 member ensemble.
The accuracy of the 4 ensemble forecasts was compared to the accuracy of the ensemble mean for the 16 and 18 member ensembles
treated as a deterministic forecast, all using the CRPS. The forecast variable is temperature in degrees Celsius and
results were computed for 24 hour intervals out to 10 days. Each point is computed from about 8000 cases.
Please answer the following questions with respect to this graph:
Loading Questions
...
1. Which of the following statements best characterizes the relative accuracy of the 4 sets of forecasts?
Incorrect, but based on lack of objective information. Strictly speaking, there should be confidence intervals
plotted on the graph. If there were, then this would be the best answer if the confidence intervals
were significantly overlapping. Since each point on the graph is supported by about 8000 cases,
one would expect at least some of the differences among the techniques to be significant.
Correct. The CRPS is negatively oriented, that is, lower is better.
2. Do these results suggest that the advantages of ensemble distribution forecasts with respect
to deterministic forecasts using the ensemble mean increase for longer projection and
greater uncertainty?
Correct. The MAE of the ensemble mean is at least 0.8 degrees higher than the CRPS for the four forecast techniques,
rising to 1.3 degrees at 10 days. Thus the advantage increases for longer range higher uncertainty forecasts.
