No leap second in December

I’ve just got the Bulletin C from the IERS, and with good news: there won’t be a leap second in December. The next leapocalypse will have to wait at least until June 2014.

For your reference, here’s the full bulletin:

     INTERNATIONAL EARTH ROTATION AND REFERENCE SYSTEMS SERVICE (IERS) 

SERVICE INTERNATIONAL DE LA ROTATION TERRESTRE ET DES SYSTEMES DE REFERENCE

SERVICE DE LA ROTATION TERRESTRE
OBSERVATOIRE DE PARIS
61, Av. de l'Observatoire 75014 PARIS (France)
Tel.      : 33 (0) 1 40 51 22 29
FAX       : 33 (0) 1 40 51 22 91
Internet  : services.iers@obspm.fr

                                             Paris, 8 July 2013

                                             Bulletin C 46

                                             To authorities responsible
                                             for the measurement and 
                                             distribution of time             

                          INFORMATION ON UTC - TAI

 NO leap second will be introduced at the end of December 2013.
 The difference between Coordinated Universal Time UTC and the 
 International Atomic Time TAI is :		

     from 2012 July 1, 0h UTC, until further notice : UTC-TAI = -35 s

 Leap seconds can be introduced in UTC at the end of the months of December 
 or June,  depending on the evolution of UT1-TAI. Bulletin C is mailed every  
 six months, either to announce a time step in UTC, or to confirm that there 
 will be no time step at the next possible date.
                                            Daniel GAMBIS
                                            Director			
                                            Earth Orientation Center of IERS
					    Observatoire de Paris, France
__________________________________________

An eye on the clock

Recently I was trying to make sane a system clock that, for some reason, suddenly slowed down to a crawl. I started to fiddle with adjtimex, and I needed a way to verify the reaction of the clock itself, and ntpd’s. On one window, I had a watch ntpq -c pe -c as running. On another, there was an ntpdate -q in a loop. In a third one, I wanted to monitor the changes in frequency (set by ntpd) and ticks (set by me). I found this one-liner pretty useful:

# while true ; do adjtimex -p | perl -alne '/frequency/ and $f=sprintf("%3.2f%%",100*$F[1]/32768000) ; /tick/ and $t=$F[1]-10000 ; END { print scalar(localtime),"\tf=$f\tt=$t" }' ; sleep 30 ; done

This snippet assumes that the frequency tolerance of the clock is 32768000, and that the normal value for ticks is 10000. Check the output of adjtimex -p and the man page to verify this fits your system, too.

The dirty dozen

I still remember very well the first (and only, for now) time I held a seminar for my colleagues in Opera: “NTP, a misunderstood protocol” in February 2011. Now fast forward to 2012. We had one leapocalypse between June and July, one bogus leap second one month later, and a bit of disaster about one week ago: nearly two years have passed since I held that seminar, and NTP is still a misunderstood protocol. … Continue reading

Know your tools: adjtimex

Cosimo was one of the speakers at Velocity Europe 2012 last week. He gave a nice talk, and he even dared to show my face in his presentation (slide 32 if you're curious).

However, what caught my attention was the very next slide, where it was evident that he was blindly installing adjtimex on all his puppet-managed nodes. I warned him that it could be a bad idea, and he seemed to be quite surprised. Sadly, I was proved to be correct just a few hours later: they had an handful of virtual machines whose clocks were going crazy in all possible ways, and the root cause was tracked down to be the installation of the adjtimex package.

But let's take a step back: what is adjtimex, and why it can switch from a useful tool to an evil rapist of system clocks?
Continue reading

How to be a timelord, or rather not

I think it's quite ironic that someone considers me an NTP guru. Because I am not. I am just a sysadmin who, once upon a time, had an NTP service to rebuild and wanted to get it right.

What happened then is: instead of doing what a bunch of the sysadmins out there did and still do (use the default configurations of the OS, or trust one of the misleading sources you can find with google), I downloaded the RFC and read most of it. And then, I downloaded the (then) two Sun blueprints about NTP and read them too. Only when I understood how a good architecture ought to be, I started with the design and the implementation.

I must say that I didn't get the architecture right the first time, but at least I learnt what was wrong with it. I kept studying on the subject, doing experiments, and discussing with people who had a much better understanding of the internals (how ntpd really works) and… externals 🙂 (all the electronics engineering theory behind the algorithms used in ntpd). And I ended up building a (hopefully) decent NTP infrastructure for all the employers I worked for.

As you see, I am just a person who tackled a problem more seriously than the average. That was enough to stand out compared to "the mass" and make me look like a guru… not being it!

When Cosimo asked for a photo of mine for his talk at Velocity Europe 2012, and he was going to talk about last July's leapocalypse, I spent one hour with my wife to get some nice photos with the right amount of irony in them. In the end, we selected two photos and sent them to Cosimo.

Of course, Cosimo chose for his presentation the one I liked less, that's why I'm doing justice to my favourite by publishing it here:


If there is as much irony as clocks in the photos, the goal is achieved 😉

The hectic week of the leap second

Last week has been an hectic period:

  • we’ve been hit by the leap second announcement bug
  • others around the internet have been, as well
  • …not to mention those hit by the leap second itself
  • Wired put my name on an article twice, which started a “citation spree” around the globe
  • I’ve finally realized my proposition to start on Twitter (@brontolinux)
  • I’ve been appointed as one of the CFEngine champions 2012

I’ll try to sum up, and conclude with a take-away lesson for next leap second.

Continue reading

An humble attempt to work around the leap second

Note: this article is now obsolete, please have a look at A humble attempt to work around the leap second, 2015 edition. Thanks.


Some background
Back in March, I talked about the experiments I was conducting to manage the leap second coming at the end of June 30th, 2012. Despite the fact that the leap second was first introduced in the early 70s, and that we never had a negative leap second up to date, a number of applications and systems still rely on some wrong assumptions, namely:

  • every minute always lasts 60 seconds
  • time read from the system clock is monotonic
  • two consecutive reads of a UNIX timestamp, happening at least one second after the other, will result in the second timestamp being bigger than the first one (rephrase of the previous point in the UNIX/POSIX world)

So bad that, after exactly fourty years from the first leap second, systems and applications still rely on these assumptions and can crash badly when, during a leap second insertion, they find themselves in a situation they didn’t expect.

David Mills, the inventor of NTP, in his document “The NTP Timescale and Leap Seconds” suggests how it should implemented on all systems that always assume 60-second minutes. If that was correctly implemented in, e.g., the Linux kernel, we’d have no need to work around any issue, as time would still be monotonic during the leap second transition. Unfortunately, that is not the case, and Linux will suddenly step back one second when the clock reaches July 1st, 2012, 00:00:00.

The procedure described below will help you to avoid the step, and to recover for the excess second the clock will find itself to have compared to its time sources. However, this procedure is far from ideal in a number of situations, and if you decide to apply it on your systems you do it at your own risk. My advice is: go for this procedure only where the risk of having a system crash due to a leap second is higher than the risk of a misbehavior due to two systems having an offset of some tenths of a second; and do that only after some testing. … Continue reading

No step back!

Summary

We'll have a leap second insertion at the end of June this year. My tests and other evidence shows that the Linux kernel in Debian Squeeze (but possibly in newer versions of the Kernel, too, I didn't check the sources) doesn't handle the leap second in the best way, and steps back one second around midnight between June 30th and July 1st. This is a big pain in the… grass 🙂 for time-sensitive applications (e.g.: clusters), or those that rely on a monotonic clock (usually RDBMSs; e.g. older versions of Oracle crashed badly, dunno about newer).

I am trying to find feasible workarounds for this, and I'll keep you posted. … Continue reading