Think all your data is safe and sound in The Cloud? Maybe you shouldn’t be so sure. Yesterday, a system administrator at a cloud service data centre accidentally rebooted every single server at once by accident. It’s a handy reminder that you’re only one idiot away from total data annihilation.
The cock-up happened at Joyent’s US-East-1 data centre yesterday. The company’s own acknowledgment of the outage uses the euphemism “operator error”, but in a post on Hacker News, Joyent’s CTO Bryan Cantri is a little more frank:
It should go without saying that we’re mortified by this. While the immediate cause was operator error, there are broader systemic issues that allowed a fat finger to take down a datacenter. As soon as we reasonably can, we will be providing a full postmortem of this: how this was architecturally possible, what exactly happened, how the system recovered, and what improvements we are/will be making to both the software and to operational procedures to assure that this doesn’t happen in the future (and that the recovery is smoother for failure modes of similar scope).
In short, the problem wasn’t just that some system operator pressed the wrong button, but that there was even a “shut it all down right now” button in the first place.
Granted, Joyent is no Google. Or Amazon. Or Dropbox. But it did host Twitter once upon a time, so it should know better than this. Turning off all the servers at once is a pretty reparable problem, sure, but if one typo can shut down a data center, it’s not hard to imagine that one might also be able to, I don’t know, delete everything.
Any company doing business in the cloud should have plenty of precautions in place to keep that from happening, but as we’ve seen, even big companies can be pretty damn bad at keeping data secure from the outside. And you have to worry about the inside too.
There’s a massive cloud data massacre just waiting to happen. It’s less a question of “if”, but rather “when” and “to whom”. So let this klutz be a reminder that you should keep a local copy too, and maybe when the cloudpocalypse comes you’ll still have backups of all your delightful photos of meals. [Joyent via The Register]