On Friday night, the internet fell victim to a thunderstorm outside of Washington D.C., taking out a chunk of Amazon’s servers, which plays host to sites and services such as Instagram and Netflix. Last night, the internet died again, taking down sites such as Yelp, Reddit, and even the Gawker network. But it wasn’t because of the elements. This time it was because of a leap second.
Long story short, at 12:00am Grenwich Meridian Time, all of the atomic clocks across the world inserted a leap second (or in simpler terms, paused for a second) so that they could remain in unison with the rotation of the planet (this is something that has occurred 24 times since 1972). As it happens, many pieces of technology, ranging from servers, to networks, to laptops, sync up their clocks with the atomic clocks. Problem is, they don’t know how to handle things when a leap second gets thrown in to the mix because they see the same second twice in a row.
The end result was pretty ugly. Sites were down for hours, as administrators worked to clean up the havoc wreaked by one little stray second. But not everyone was affected. In fact, one site was well-prepared: Google. As Wired points out, Google was anticipating this moment, outlining months ago its strategy for handling the leap second threat with Leap Smears.
The solution we came up with came to be known as the “leap smear.” We modified our internal NTP servers to gradually add a couple of milliseconds to every update, varying over a time window before the moment when the leap second actually happens. This meant that when it became time to add an extra second at midnight, our clocks had already taken this into account, by skewing the time over the course of the day. All of our servers were then able to continue as normal with the new year, blissfully unaware that a leap second had just occurred. We plan to use this “leap smear” technique again in the future, when new leap seconds are announced by the IERS.
So in case you were wondering why the internet has been god awful all weekend, now you know why.
Image via Shutterstock/iofoto