Who was asleep at the wheel at the NWS on August 4? They had an issue with wildly bad low forecasts in NDFD in and around the Los Angeles area, forecasting lows between 3 and 22 degrees (the 50's or 60's would be more appropriate). These bad lows even propagated and were published by both Weather Underground and WeatherBug, passing through their QA checks. Interestingly, though, the forecasts were correct on NWS's own website.

Generally, either the forecast is so wrong it fails the automated checks (like a low less than -150F), or its missing completely. The hardest to audit are the ones that are "almost" good (like these). Here is a little insight as to how and when I manually invalidate forecasts.

As part of each month's aggregation, I run a number of queries that hopefully will make forecasts and observations that are "unusual" or that warrant a further look stand out. I take a look at these reports and spend a day (or longer, if necessary, as was the case this month) going through each one and determining if it should be marked invalid.

I am very hesitant to mark forecasts invalid. The forecast is the forecast, and if that's what went out, sorry. However, invalid forecasts, being so obviously not based on the process and skill of the meteorologist but rather a mistake or processing error, will skew the valid results and make them less useful and valuable.

So my first rule is, it must be both a obviously bad forecast and one where a meteorologist at another company/organization would have no question that the forecast should be marked invalid. I generally look at forecasts from the same provider and for the same location, but made on different days. If the forecast made the day before and the day after are nearly the same, and wildly different than the one I'm looking at, then it's likely there was an error and the forecast is invalid. For these, I always look at the actual source file to verify that the forecast was successfully retrieved and that it wasn't an error on ForecastWatch's side.

I will also look to see what other providers forecasts look like. If they are not too far off, then it's probably just a really bad model or a total miss on front timing. Generally though, the errors seen with invalid forecasts are well above errors from missed forecasts, so it's pretty obvious.

In the case with August 4, the low forecast was obviously invalid, but not so blatantly wrong to trigger QA checks at other providers like Weather Underground and WeatherBug that do very little other than repackage the NWS forecast. So both those providers carried on the NWS invalid forecasts, whereas in many cases the invalid forecasts seen at the NWS are generally caught before making it to the other providers.

In any event, August 2011 data is now available in ForecastWatch. Have a great month! Fall is in the air!