Accuracy of CustomWeather
Probability of Precipitation Forecasts
(February 2004 through January 2005)

By ForecastWatch.com, a Service of Intellovations, LLC
March 1, 2005

Introduction to ForecastWatch.com

ForecastWatch.com has been helping meteorologists improve their weather forecasts for over two years. Created by Intellovations, LLC, a full-service technology consulting company, ForecastWatch.com provides weather forecast accuracy information for over 800 locations within the United States and Canada. ForecastWatch.com has been used by meteorologists and companies whose business depends on the weather to:

• Evaluate weather forecast providers

• Improve weather forecasts by providing useful feedback

• Compare themselves to other weather forecast providers

• Educate their customers

• Improve the quality of their weather forecast websites

ForecastWatch.com is the only company that provides ongoing weather forecast accuracy information to meteorologists, utilities and energy companies, the agriculture industry, and anyone whose reputation or business depends on being right about the weather. In 2003, ForecastWatch.com released the results of the largest public weather forecast accuracy study undertaken to that point. ForecastWatch.com's ongoing weather forecast accuracy information system has been used to perform custom analysis and reporting, and was instrumental in making this report possible.

How Accuracy Was Measured

The accuracy of a probability of precipitation (POP) forecast is difficult to evaluate. If a forecast states that there is a 30% chance of precipitation, is it right if it doesn't rain? The only way to evaluate the accuracy of a probability forecast is to look at a large number of forecasts that predicted a 30% chance of rain or snow, and see how often it actually rained or snowed. Then, do the same for all other forecasted percentages, from 0% to 100%, and see how closely it matches the forecast.

Unfortunately, it takes a lot of forecasts to get a good picture of accuracy. This is because for every POP or POP range evaluated, a statistically valid sample of actual precipitation events must be captured. For this study, each POP forecast, from 0% to 100% was evaluated individually, not grouped into ranges (say 10%-20% chance, 20%-30% chance, etc.).

From February 2004 through January 2005, ForecastWatch.com collected nearly one and a half million POP forecasts from CustomWeather's website, MyForecast.com. These forecasts were from one- to five-days out, and spanned 839 locations within the United States. Each of the POP forecasts was grouped together. Then, for each location on each day, the actual number of precipitation events was counted. ForecastWatch.com compared the forecasts to the National Weather Service ASOS weather network. An actual daily observation was considered a precipitation event if more than a trace of either rain or snow was reported by the station.

POP Accuracy Results

The results of the POP weather forecast accuracy evaluation can be seen in Figure 1. The dashed line indicates the “perfect” forecast. A perfect forecast means that for each probability of precipitation forecast, precipitation was observed exactly that percent of time. For example, precipitation was observed exactly 50% of the time a 50% chance of precipitation was forecast.

The blue line is the actual observed precipitation percent for each POP forecast value. If the blue line is above the dashed line, it indicates it rained or snowed more often than forecast. For example, it actually rained or snowed 23.8% of the time when a 20% probability of precipitation was forecast. If the blue line is below the dashed line, it indicates it rained or snowed less often than forecast.

Figure 1: CustomWeather's POP Forecast versus Actual Precipitation Event Percentage.

As Figure 1 demonstrates, there is a high correlation between CustomWeather's POP forecast and what actually occurs. The average error between forecast and actual precipitation probability was 3.51 percentage points. The greatest error occurs when CustomWeather forecasts 100% probability of precipitation. Only 94.47% of the time did it actually rain or snow. The least error occurs at the 80% probability of precipitation forecast. With an error of only 0.12 percentage points, it actually rained or snowed 80.12% of the time that forecast was made. CustomWeather's POP forecasts are accurate and closely reflect what actually happened.