Back online. New Server. Lost Data.

February 12, 2009 at 12:11 am

HarddriveIt’s been a crazy week for us here. If you haven’t noticed, WineLog.net has been offline for almost an entire week. And before that our uptime was spotty. I’ll try to cover the key points of interest below, but feel free to comment here or contact us if you have any questions or concerns.

What Happened?

The company we were hosting our site with secretly sold their business. With no prior notice, our servers were migrated to a new location. This caused a bit of downtime as we updated the winelog.net domain to point to the new servers.

This wouldn’t have been so bad, but it turns out the new setup was not really ready. A few days later, the server restarted randomly, was down completely at times, and finally “disappeared”. The support team cited “hardware issues”. We lost access to the server and had to fight hundreds (thousands?) of other clients for the attention of the support staff to get our servers back.

When the servers came back, we realized they had been restored using backups from January 27th. We have lost all data updated on the site between January 27th and February 7th. This is probably about 250 new users, 500 newly added/logged wines, a few blog posts, and numerous other bits of data that we know if very important to our users.

We are very sorry that your data may have been lost, and we are doing everything we can to get it back.

How come you didn’t have a more recent backup?

This is our fault. We had opportunities to create more frequent backups. And we should have been more careful after the initial migration a couple weeks ago. The old server made daily backups on-site and I would move the latest of these off-site once a week. After the initial migration (when I should have been more cautious) I missed a weekly backup because the IPs had changed.

What have I done?

I’ve moved WineLog to a new dedicated server with another company. One bonus from all this is that the site may be a bit snappier on this new hardware.

I am working with the old host to possibly recover some of that 1/27-2/7 data. Cross your fingers. If we get this data, we will be able to “splice” it back into the site.

By the end of the week, I will have a new backup system in place with DAILY OFF-SITE backups. And off-site here means in my home office, in my hands, where I will have control over it.

What can you do?

If you are wondering what that wine was that you tasted last week. You can try searching Google and viewing the “cache” for your wine log. For example a search for “jason site:winelog.net”, let’s me get to this page to see my wine log as it was a few days ago.

You can also search the WineLog twitter feed for the post made there. Most logged wines show up in that feed. And you can at least recover the name of the wine and your rating. Do us a favor and relog the wine. (If we eventually recover your previous data, we will be able to merge the records.)

You can also help us by reporting any bugs you happen across on the site. WineLog is complicated software. I’ve been testing and will continue to test, but I’m sure there are scenarios we’ll miss where errors may crop up on the site. Do us a favor and contact us if you see anything funny going on.

What more can I do?

We have a few nifty features that have been lingering in development. I will try to fast track those to get something exciting out there as soon as possible. I will try to come up with a good way to repay you all… maybe dig up some good deals with our ecommerce partners. And I will try to never let this happen again.

Thanks for your patience. Thanks for your understanding. Thanks for supporting WineLog.

Jason Coleman is Co-founder of and Lead Developer for WineLog.net. Check out his wine log, read more of his blog posts, or contact him here.

2 Responses to “Back online. New Server. Lost Data.”

  1. info says:

    p.s. Also working to get WineLog.mobi back online soon.

  2. I feel your pain. As entrepreneurs running social site we are 100% dedicated to making sure our systems are available to people who love our communities. Good luck and I would to learn more details on the corporate blog on how you physically scaled to the new system.