In 2011 Facebook reported that their first data centre in (Prineville in the US) had a high humidity issue. Probably not the best condition for servers, sure. But it turns out that wonky temperature controls were actually causing condensation in the data centre. Like indoor rain. Like it was literally raining in the server room.
The cloud on cloud action was reported by The Register, which quoted Facebook’s VP of Engineering, Jay Parikh as saying:
I got a call, ‘Jay, there’s a cloud in the data centre.’ ‘What do you mean, outside?’ ‘No, inside’.”
Though it’s not impossible to interpret what was actually going on from Facebook’s 2011 explanation (“cold aisle supply temperature exceeding 80°F and relative humidity exceeding 95%”) they definitely left a lot of the drama out. As air circulated and cooled, it took on so much moisture that it was forming a cloud and raining down on the servers, which automatically shut down or just shorted out.
Parikh says Facebook’s data centres have changed their protocols so such an incident will never reoccur. But just in case, all servers are now additionally equipped with “a rubber raincoat” around their power supplies. [The Register]