Syslog

How to make CFEngine recognize if systemd is used in Debian

/ bronto / Leave a comment

CFEngine 3.6 tries to understand if a Linux is using systemd as init system by looking at the contents of /proc/1/cmdline, that happens in bundle common inventory_linux. That’s indeed a smart thing to do but unfortunately fails on Debian Jessie, where you have:

root@cf-test-v10:~# ls -l /sbin/init
lrwxrwxrwx 1 root root 20 May 26 06:07 /sbin/init -> /lib/systemd/systemd

the pseudo-file in /proc will still report /sbin/init and as a result the systemd class won’t be set. This affects services promises negatively and therefore I needed to make our policies try to outsmart the inventory 😉 These promises, added in a bundle of ours, did the trick:

bundle common debian_info {
  vars:
    init_is_link::
      "init_link_destination"
          string => filestat("/sbin/init","linktarget") ;

  classes:
    init_is_link::
      "systemd"
          expression => regcmp("/lib/systemd/systemd",
                               "$(init_link_destination)"),
          comment => "Check if /sbin/init links to systemd" ;

    debian::
      "init_is_link"
          expression => islink("/sbin/init"),
          comment => "Detect if init is a link" ;
}

Notice that our bundle is actually bigger, I cut off all the promises that were not relevant for this post. Enjoy!

A humble attempt to work around the leap second, 2015 edition

/ bronto / 7 Comments

TurnBackTimeUpdate: Watch out for public servers not announcing the leap second! In the last few minutes we have been observing a number of public servers (even stratum 1) that don’t announce the leap second. If the majority of your upstream doesn’t announce the leap second, your clients won’t trigger it. If that’s your case, you can use ntpd’s leapfile directive and a leap second file to provide your own servers with the correct information. Check the ntpd documentation for more information.

Update: Miroslav Lichvar has counted the public servers that are announcing the leap second on a per-country basis. You can find his stats on pastebin.

I have been running simulations for the upcoming leap second for a few weeks now. While some mysteries haven’t been solved yet, I was finally able to put together a configuration for our servers and clients that satisfies to the following requirements (where do these requirements come from? That is explained further down in the article):

  1. it works on Debian Linux Squeeze, Wheezy and Jessie
  2. it keeps the Linux kernel out of the game, in order to avoid triggering unknown kernel bugs
  3. it avoids backward steps of the clock
  4. the clock converges to the right time in an acceptable amount of hours
  5. it doesn’t hog public services

What this solution doesn’t provide: this is neither Google’s leap smear nor Amazon’s: you use standard ntpd code with no changes; this is not a fast clock slew as chrony’s either. Servers/clients have evolved predictably during most of the simulations and shouldn’t diverge too much from each other, but there are conditions where you may observe offsets between them in the order of magnitude of 0.1s. That should still be bearable though and will still save you from the headache of kernel bugs or jumps back in time. In order to work properly, this solution must make a few assumptions:

  1. you have at least four internal NTP servers, synchronized with at least four public servers and/or internal specialized time sources
  2. your clients use at least four of your own internal NTP servers and no external NTP server
  3. you use unicast NTP packets (broadcast and multicast will probably work as well or even better, but they haven’t been tested in my simulations)
  4. you are using ntpd (the reference implementation) version 4.2.8p3 (earlier versions have a bug that will make our countermeasures against clock stepping ineffective)

Let’s look at the implementation on both server and client side, which is pretty similar but with a few important differences. Continue reading →

Scary times at the leap second lab

/ bronto / 1 Comment
The leap-lab at Opera (2015)

The leap-lab at Opera (2015)

After one month spent on other high priority tasks it was about time to get back to the leap second lab. The fated day is coming and we need to have a strategy in place.

I spent this week running tests, tuning the scripts that support them, and improving the CFEngine policies that manage the lab today and will implement our strategy tomorrow. Besides, I structured my tests a bit better to ensure that the “false start” I had one month ago doesn’t happen again.

On Friday I finally got to run some crucial tests and the results of one of them were scary to say the least.

Continue reading →

Wheezy and the leap second: a mystery?

/ bronto / Leave a comment

Yesterday I reported about Debian Wheezy steppiTurnBackTimeng back one second, despite the settings in ntpd prohibiting step changes and the leap second not armed in the kernel. The clock in Debian Jessie didn’t step.

At first, I thought it depended on a different ntpd version shipped in the two distributions, but it turned out to be the same. That suggested me that I should have tried a new experiment: run two tests in parallel on wheezy, one with an ntpd running and the other without, to see if the one without ntpd would still step back.

To my biggest surprise, no step happened in either.

This suggests that there must have been something odd in yesterday’s experiment and I should repeat it, while watching the configurations and set up more closely. As always, I’ll keep you posted. Until then, take care.

The leap marathon has started

/ bronto / 1 Comment

TurnBackTimeNo, I’m not going to run 42 kilometres jumping 🙂 I’ve started my leap second tests today. The goal of the tests is to find a configuration, or a procedure, or both, to avoid a backwards step of the clock at the insertion of the leap second at the end of June 30th (UTC).

I ran the first test today. In the ntpd configuration I set two directives: tinker step 0 and disable kernel. The first directive disables step adjustments, the latter disables the kernel discipline: ntpd will manage the clock all by itself instead of “asking” the kernel to make corrections to the clock speed; it is not really necessary as the first one should be enough to automatically disable the kernel discipline, so it’s there just for good measure.

So I installed the leap seconds file, installed the new configuration for ntpd, reset the clock to June 30th, 2015 and started the test. For the whole duration of the test the leap second was never armed in the kernel. Everything went as planned in Debian Jessie:

 2015/06/30 23:59:59.998489934
 2015/06/30 23:59:59.999009849
 2015/06/30 23:59:59.999536585
 2015/07/01 00:00:00.000063781
 2015/07/01 00:00:00.000589560
 2015/07/01 00:00:00.001109634

but not so in Wheezy:

2015/06/30 23:59:59.998049126
2015/06/30 23:59:59.998788657
2015/06/30 23:59:59.999572132
2015/07/01 00:00:00.000316483
2015/07/01 00:00:00.001051262
2015/07/01 00:00:00.001792934
2015/07/01 00:00:00.002529339
2015/06/30 23:59:59.004499757
2015/06/30 23:59:59.005266331
2015/06/30 23:59:59.006014975

That means that for wheezy we have two possible “branches”:

  • it was ntpd to request the step back
  • it was the kernel to request the step back.

The second case is, of course, unlikely as the kernel didn’t know about a leap second. Therefore, the branch to follow first is to use the same ntpd as in jessie in wheezy and see if the results match or not. I’ll keep you posted. Take care.

Update: apparently jessie and wheezy sport the same version of ntpd. Oh well…

Bug or feature? Dereferencing of arrays and namespaces

/ bronto / Leave a comment

CFEngineAgentNow that the upgrade from 3.4 to 3.6 is advancing slowly but steadily I am starting to check the features that are new in 3.6 compared to 3.4.  According to the docs namespaces were actually introduced in 3.4.0, but I didn’t take advantage of them yet, and it’s time to start.

When something is declared in a namespace (a bundle, a variable or whatnot) it must be referred to with its namespace. For example, if you declare a bundle test in the namespace nstest, you’ll refer to that bundle from outside the namespace (e.g. in the bundlesequence) as nstest:test. If you declare a variable, for example an array called conf in that bundle, that will be nstest:test.conf outside the namespace. So far so good.

Now, what happens inside the namespace? Well, I found one fact that is indeed surprising.

Continue reading →