The September 2010 aggregations are complete and available online. The re-aggregations of June 2009-January 2010 MOS high and May 2010-July 2010 precipitation data are complete. Soon, we will start back on 2009 and earlier precipitation reaggregations to generate the new combined precipitation statistics.

It's hard to believe that in March of this year we moved to a far more capable server than we had been using, and that it is now straining. We calculated almost double the statistics this month that we did back in March. The strain has been showing, as monthly data releases that were occurring on the 15th or 16th are now happening on the 20th or 21st. We are closing in on 200 million forecasts and 90 months of aggregated statistics!

So with that, at the end of the month we will be moving to another server as the first step to improving performance. The current server we are on is a 4GB Dell PowerEdge 2950 with two 500GB 7200RPM SATA drives in a RAID 0 configuration. We are moving to a 12GB Dell PowerEdge R710 with four 300GB 15K SAS drives in a RAID10 configuration. This increase in memory and disk throughput should take us to the next level in database performance.

Additionally, the ForecastWatch database is seven years old. We've done a lot to improve the table structure and the queries to keep performance up. But when it was created it was created with MyISAM tables, because there wasn't a better option at the time. Now, however, the InnoDB engine matches the performance of the MyISAM engine and provides a number of benefits. So we will also be migrating a number of the tables to InnoDB. This will give us one big benefit that has been hindering performance: no more table locking. MyISAM tables are unavailable for reading when they are being written to. This causes performance issues on the website and slows forecast collection when aggregations and other write tasks are being performed. InnoDB allows for row-level locking, so the parts of the table that are not affected by the write can still be read.

Additionally, I've spent some time, and will be spending more time over the next few months, to take the audit process to the next level. It is currently very robust, and catches most invalid forecasts and observations, but doesn't do as good a job validating the aggregated statistics. There are some, but as the issue with MOS high stats showed, there are some holes. Some of those holes have already been patched, and others will over the next few months. Additionally, there are a number of manual audit steps that I hope to automate, saving time for other audit tasks and making things less error-prone.