If you've been following my Twitter you know that the web server at Schweitzer got hacked. Whoever broke in didn't do a very solid job, and managed to kill the network before doing anything terribly malicious. The site was only down about 12 hours. After looking through the system it looks like the vulnerability was caused by my not patching ssh reliably (it was several versions out of date). The server rebuild was time consuming, but easy. The biggest hurdle was slow mirrors for downloading the most recent linux ISOs. This sort of thing is a known risk of running our servers in house. The good news is that our backup strategy was rock solid and there wasn't any data loss (maybe a couple of hours). As an expansion of our current approach to backups I'm going to set up a duplicate server that mirrors the home tree of the main server hourly. In the event of the main system going offline for whatever reason we'll be able to bring up the mirror just by adjusting the firewall. Hopefully we'll never have to use it, but having a hot spare web system seems like a good idea.