The Brier Score – Accuracy of a probability forecast

The Brier Score is probably the most commonly used verification measure for assessing the accuracy of probability forecasts. The score is the mean squared error of the probability forecasts over the verification sample and is expressed as:

where N is the sample size. The observations oj are all binary, 1 if the event occurs and 0 if it doesn’t. The Brier score ranges from 0 for a perfect forecast to 1 for the worst possible forecast. Although the score can be computed on a single forecast, the result wouldn’t be very meaningful because the observation is binary and the forecast is a probability.

The following table shows ten forecasts of the probability of precipitation from each of four forecasters, "Mr. Prob", "Mr. Sharp", "Mr. Climat", and "Mr. Cats." Mr. Prob believes he can distinguish the likelihood of rain to within 10% intervals, so feels free to use all probability values, to the nearest 10%. Mr. Sharp believes that one should give clear guidance; that forecasting near 50% is useless to everyone. Mr. Climat has no confidence whatsoever in his ability to discern greater and lesser chances of rain, but he knows that rain happens on average on 4 days out of 10, 40% of the occasions. Mr. Cats is a traditional deterministic forecaster, and thinks that the duty of the forecaster is to give his best estimate of what will happen. He forecasts categorically, forecasting rain if he thinks it is likely to happen. Using the observations given in the last row, compute the Brier scores for these forecasts and answer the following questions.

To aid the computations, the square of each of the possible error values is shown below.

1. Drag and drop the correct Brier scores at right into the table, based on your calculations:

correct

incorrect - try again

Please put the labels on one of the existing boxes.