A Near Death Experience
July 15th, 2008 Chris Posted in Operations | 3 Comments »
This weekend, Colleen and I were sitting around in our family room, each with our laptops on, well, our laps. I knew something was going horribly wrong when she leaned over and asked why her blog was only showing posts up to March 08. Of course, I automatically assumed it was user error and made some crack. But then I looked at this blog, a noted the complete absence of, well, this blog.
I went to the support forum for our hosting company, Hosting Zoom, and in terror read:
“There are some ongoing issues with the database after the primary node was rebooted. It looks like the databases has rolled back to a previous date. Unfortunately, the daily backups are not up to date.“
In fact, daily backups hadn’t been working since March. That’s why all Colleen’s blog entries since March were gone and this blog (which started in May) was vaporized.
As a side-note, I had to laugh at the response of the hoster. In my previous life, we made mission-critical security software and when a bug was found, all hell broke lose. Nothing is grumpier than a big bank on a Friday afternoon when their security software isn’t working. All you can do is stay professional, develop a well thought out plan and grovel. Hosting Zoom’s first response was not quite along those lines:
“It is ultimately your responsibility to keep current back ups of your data. This is noted in our terms of service: ‘HOSTING ZOOM DOES NOT GUARANTEE THE AVAILABILITY, COMPLETENESS, CURRENCY, OR INTEGRITY OF THESE BACKUPS OR THE DATA THEY CONTAIN. Consequently, you must not rely upon the availability, completeness, currency, or integrity of these backups.’ “
In other words, it is all my fault that they didn’t keep backups. From a PR perspective – complete disaster.
But you know what? They are right. It was my fault.
As a micro business, I’m at the mercy of a variety of service providers to make our business run. A business that is the only source of income for Colleen and I. Not making sure that I had contingencies in case one of those service providers blow up was negligent or at least dumb.
I sat down and made a list of all the places I’m vulnerable:
- Data on PCs in the office and our laptops
- Customer and marketing data in our CRM system
- Web content and databases on our hoster
- Financial information in our payroll system
Not to mention a few non-IT related insurance things:
- Disability and life insurance for Colleen and I (we do ride motorcycles after all…)
- Content insurance for the home and office
- Liability insurance for the business
As it turns out, we’ve actually got business continuity or at least back-up processes in place for most – apart from the one that blew up. And believe me, we now have back-up stuff for the web out the wazoo.
The story has a happy ending. It turns out, we were one of the few Hosting Zoom accounts whose back up processes were running correctly. So we had our backups recovered and we were back in business. Actually – it was really a good thing as it exposed a current weakness in our processes.
When is the last time you sat down and thought through all the systems your business relies on and what would happen if those systems failed or your data was lost? Don’t wait for a near death experience, fix it now.
C.
P. S. I’ve not linked any of the references to Hosting Zoom because you should not do business with them under any circumstances. More on that later…
July 15th, 2008 at 10:52 am
Sounds like you might use a service like Mozy.com I like the idea of scheduled backups happening automatically in the background to a remote site owned by a reputable company (EMC). It beats the removable drive process that requires human intervention and won’t recover from disaster. Just my .02 cents!
July 15th, 2008 at 11:12 am
Steve,
Great point. We use Dell for our computer and file servers (although Mozy sounds better but is much more expensive). The specific challenge is that our MySQL databases can’t be accessed remotely without going through the web hosters interface so a different solution is required. Especially different than the one I was using – doing nothing. For all you bloggers using WordPress – there is a great backup plug-in (http://www.ilfilosofo.com/blog/wp-db-backup/) that automatically does regular back ups of your MySQL database.
C.
July 17th, 2008 at 10:46 am
[...] been much worse if I had to be re-creating our blog and member site databases. As I mentioned in my last post, our web hoster had major backup issues and we almost lost a wack of [...]