As some of my readers already know, I changed jobs in Novembre: I left Opera Software to join Telenor Digital. We have decided not to run any leap second simulation here, so I am not going to publish anything on the subject this year. You can still refer to the post The leap second aftermath for some suggestions I wrote after the latest leap second we had in June/July 2015.
Good luck!
A new leap second will be introduced at the end of 2016. We have six months to get ready, but this time it may be easier than before as several timekeeping software have implemented some “leap smear” algorithm, which seems to be a very popular approach nowadays; e.g.: ntpd, the reference implementation for NTP, seems to have implemented leap smear from version 4.2.8p3 onward.
We’ll see how it goes. Until then… test!
The leap second is finally behind us, and for the first time it has been transformed in an event. That had the unfortunate consequence that many channels where useful information had flown in the previous events were now flooded with bullshit. But it’s over. A giant army of idiots has finally stopped asking “what will you do with your extra second?”, a smaller but still noticeable army of inaccurate writers and journalists won’t write for a while that the atomic clocks need to be stopped for a second to realign with the Earth (?!?!?!?!?!?). We can now sit, look back and save some take-aways for the next edition of the event.
Update: Watch out for public servers not announcing the leap second! In the last few minutes we have been observing a number of public servers (even stratum 1) that don’t announce the leap second. If the majority of your upstream doesn’t announce the leap second, your clients won’t trigger it. If that’s your case, you can use ntpd’s leapfile directive and a leap second file to provide your own servers with the correct information. Check the ntpd documentation for more information.
Update: Miroslav Lichvar has counted the public servers that are announcing the leap second on a per-country basis. You can find his stats on pastebin.
I have been running simulations for the upcoming leap second for a few weeks now. While some mysteries haven’t been solved yet, I was finally able to put together a configuration for our servers and clients that satisfies to the following requirements (where do these requirements come from? That is explained further down in the article):
What this solution doesn’t provide: this is neither Google’s leap smear nor Amazon’s: you use standard ntpd code with no changes; this is not a fast clock slew as chrony’s either. Servers/clients have evolved predictably during most of the simulations and shouldn’t diverge too much from each other, but there are conditions where you may observe offsets between them in the order of magnitude of 0.1s. That should still be bearable though and will still save you from the headache of kernel bugs or jumps back in time. In order to work properly, this solution must make a few assumptions:
Let’s look at the implementation on both server and client side, which is pretty similar but with a few important differences. Continue reading
I’m starting to write a detailed description about how we’ll tackle the leap second. In the meanwhile you can well read this very beautiful and informative piece by Miroslav Lichvar. His observations about the behaviour of ntpd match mine and will give you an idea of how our solution works.
The leap-lab at Opera (2015)
I’ve been quite busy with my experiments to prevent possible side effect of the leap second. As those who follow me on twitter know I am quite close to finalise a recommendation: all going well, that should come by the end of the week. Stay tuned!
The leap-lab at Opera (2015)
After one month spent on other high priority tasks it was about time to get back to the leap second lab. The fated day is coming and we need to have a strategy in place.
I spent this week running tests, tuning the scripts that support them, and improving the CFEngine policies that manage the lab today and will implement our strategy tomorrow. Besides, I structured my tests a bit better to ensure that the “false start” I had one month ago doesn’t happen again.
On Friday I finally got to run some crucial tests and the results of one of them were scary to say the least.
Yesterday I reported about Debian Wheezy stepping back one second, despite the settings in ntpd prohibiting step changes and the leap second not armed in the kernel. The clock in Debian Jessie didn’t step.
At first, I thought it depended on a different ntpd version shipped in the two distributions, but it turned out to be the same. That suggested me that I should have tried a new experiment: run two tests in parallel on wheezy, one with an ntpd running and the other without, to see if the one without ntpd would still step back.
To my biggest surprise, no step happened in either.
This suggests that there must have been something odd in yesterday’s experiment and I should repeat it, while watching the configurations and set up more closely. As always, I’ll keep you posted. Until then, take care.
No, I’m not going to run 42 kilometres jumping 🙂 I’ve started my leap second tests today. The goal of the tests is to find a configuration, or a procedure, or both, to avoid a backwards step of the clock at the insertion of the leap second at the end of June 30th (UTC).
I ran the first test today. In the ntpd configuration I set two directives: tinker step 0 and disable kernel. The first directive disables step adjustments, the latter disables the kernel discipline: ntpd will manage the clock all by itself instead of “asking” the kernel to make corrections to the clock speed; it is not really necessary as the first one should be enough to automatically disable the kernel discipline, so it’s there just for good measure.
So I installed the leap seconds file, installed the new configuration for ntpd, reset the clock to June 30th, 2015 and started the test. For the whole duration of the test the leap second was never armed in the kernel. Everything went as planned in Debian Jessie:
2015/06/30 23:59:59.998489934 2015/06/30 23:59:59.999009849 2015/06/30 23:59:59.999536585 2015/07/01 00:00:00.000063781 2015/07/01 00:00:00.000589560 2015/07/01 00:00:00.001109634
but not so in Wheezy:
2015/06/30 23:59:59.998049126 2015/06/30 23:59:59.998788657 2015/06/30 23:59:59.999572132 2015/07/01 00:00:00.000316483 2015/07/01 00:00:00.001051262 2015/07/01 00:00:00.001792934 2015/07/01 00:00:00.002529339 2015/06/30 23:59:59.004499757 2015/06/30 23:59:59.005266331 2015/06/30 23:59:59.006014975
That means that for wheezy we have two possible “branches”:
The second case is, of course, unlikely as the kernel didn’t know about a leap second. Therefore, the branch to follow first is to use the same ntpd as in jessie in wheezy and see if the results match or not. I’ll keep you posted. Take care.
Update: apparently jessie and wheezy sport the same version of ntpd. Oh well…
It’s going to happen again: IERS’ Bulletin C was published yesterday announcing that we’ll have a leap second at the end of June this year. It’s time to get ready, but we this time we are luckier than in 2012: leapocalypse’s scars were deep and still hurt, and only inexperienced sysadmins or experienced idiots will deny the need of preparing a strategy to mitigate the unavoidable bugs that will be triggered by the leap second insertion. Or even by the leap second announcement, like it happened in 2012.
Which bugs? Nobody knows. We have six months left during which you could review the source code of your OS’ and software. Or maybe not, but you can at least test what happens on your systems when the leap second is announced and when it is inserted.
Like in 2012, I’ll soon start testing the behavior of both ntpd and Linux. The test framework will be the same as last time, but with the added wisdom gained fighting against leapocalypse. This is what I plan to add to the picture:
Like last time I’ll share my results. If you’ll be doing similar tests in your environment, I’ll be more than glad to see your findings. This is going to affect us all, the more information we share, the more chances we have to overcome the bugs.
Image from http://writing.wikinut.com/img/3bguk3_7i7al8s9b/If-We-Could-Turn-Back-Time
A sysadmin's logbook 