systemd unit files for CFEngine

systemd logoLearning more of systemd has been on my agenda since the release of Debian 8 “Jessie”. With the new year I decided that I had procrastinated enough, I made a plan and started to study according to the plan. Today it was time for action: to verify my understanding of the documentation I read up to now, I decided to put together unit files for CFEngine. It was an almost complete success and the result is now on GitHub for everyone to enjoy. I would appreciate if you’d give them a shot and report back.

Main goals achieved:

  1. I successfully created three service unit files, one for each of CFEngine’s daemons: cf-serverd, cf-execd and cf-monitord; the units are designed so that if any of the daemon is killed for any reason, systemd will bring it back immediately.
  2. I successfully created a target unit file that puts together the three service units. When the cfengine3 target is started, the three daemons are requested to start; when the cfengine3 target is stopped, the three daemons are stopped. The cfengine3 target completely replaces the init script functionality.

Goal not achieved: I’ve given a shot at socket activation, so that the activation of cf-serverd was delayed until a connection was initiated to port 5308/TCP. That didn’t work properly: systemd tried to start cf-serverd but it died immediately, and systemd tried and tried again until it was too much. I’ll have to investigate if cf-serverd needs to support socket activation explicitly or if I was doing something wrong. The socket unit is not part of the distribution on GitHub but its content are reported here below. In case you spot any problem please let me know.

Continue reading

How to make CFEngine recognize if systemd is used in Debian

CFEngine 3.6 tries to understand if a Linux is using systemd as init system by looking at the contents of /proc/1/cmdline, that happens in bundle common inventory_linux. That’s indeed a smart thing to do but unfortunately fails on Debian Jessie, where you have:

root@cf-test-v10:~# ls -l /sbin/init
lrwxrwxrwx 1 root root 20 May 26 06:07 /sbin/init -> /lib/systemd/systemd

the pseudo-file in /proc will still report /sbin/init and as a result the systemd class won’t be set. This affects services promises negatively and therefore I needed to make our policies try to outsmart the inventory 😉 These promises, added in a bundle of ours, did the trick:

bundle common debian_info {
          string => filestat("/sbin/init","linktarget") ;

          expression => regcmp("/lib/systemd/systemd",
          comment => "Check if /sbin/init links to systemd" ;

          expression => islink("/sbin/init"),
          comment => "Detect if init is a link" ;

Notice that our bundle is actually bigger, I cut off all the promises that were not relevant for this post. Enjoy!

A humble attempt to work around the leap second, 2015 edition

TurnBackTimeUpdate: Watch out for public servers not announcing the leap second! In the last few minutes we have been observing a number of public servers (even stratum 1) that don’t announce the leap second. If the majority of your upstream doesn’t announce the leap second, your clients won’t trigger it. If that’s your case, you can use ntpd’s leapfile directive and a leap second file to provide your own servers with the correct information. Check the ntpd documentation for more information.

Update: Miroslav Lichvar has counted the public servers that are announcing the leap second on a per-country basis. You can find his stats on pastebin.

I have been running simulations for the upcoming leap second for a few weeks now. While some mysteries haven’t been solved yet, I was finally able to put together a configuration for our servers and clients that satisfies to the following requirements (where do these requirements come from? That is explained further down in the article):

  1. it works on Debian Linux Squeeze, Wheezy and Jessie
  2. it keeps the Linux kernel out of the game, in order to avoid triggering unknown kernel bugs
  3. it avoids backward steps of the clock
  4. the clock converges to the right time in an acceptable amount of hours
  5. it doesn’t hog public services

What this solution doesn’t provide: this is neither Google’s leap smear nor Amazon’s: you use standard ntpd code with no changes; this is not a fast clock slew as chrony’s either. Servers/clients have evolved predictably during most of the simulations and shouldn’t diverge too much from each other, but there are conditions where you may observe offsets between them in the order of magnitude of 0.1s. That should still be bearable though and will still save you from the headache of kernel bugs or jumps back in time. In order to work properly, this solution must make a few assumptions:

  1. you have at least four internal NTP servers, synchronized with at least four public servers and/or internal specialized time sources
  2. your clients use at least four of your own internal NTP servers and no external NTP server
  3. you use unicast NTP packets (broadcast and multicast will probably work as well or even better, but they haven’t been tested in my simulations)
  4. you are using ntpd (the reference implementation) version 4.2.8p3 (earlier versions have a bug that will make our countermeasures against clock stepping ineffective)

Let’s look at the implementation on both server and client side, which is pretty similar but with a few important differences. Continue reading

Safer package installations with APT and CFEngine

CFEngineAgentPackage installation can be tricky sometimes when using configuration management tools, as the order in which package operations are performed can have an impact on the final result, sometimes a disastrous impact. Months ago I had been looking for a way to make apt-get a bit less proactive when trying to solve dependencies and removing packages and came up with the following package method that we now have in our library.

To use it, just put it in your policies and use apt_get_safe in your policies instead of apt_get wherever you want a more prudential approach to package installations.

I’m putting it here in the hope that it may be useful for everyone. I have used it successfully in Debian 5, 6, 7, 8 and on CFEngine 3.4.4 (where I borrowed parts of the masterfiles from 3.5 and 3.6 like, e.g., the debian_knowledge bundle) and 3.6.x. Enjoy!

Continue reading

A small leap for a clock, a giant leap for mankind

TurnBackTimeIt’s going to happen again: IERS’ Bulletin C was published yesterday announcing that we’ll have a leap second at the end of June this year. It’s time to get ready, but we this time we are luckier than in 2012: leapocalypse’s scars were deep and still hurt, and only inexperienced sysadmins or experienced idiots will deny the need of preparing a strategy to mitigate the unavoidable bugs that will be triggered by the leap second insertion. Or even by the leap second announcement, like it happened in 2012.

Which bugs? Nobody knows. We have six months left during which you could review the source code of your OS’ and software. Or maybe not, but you can at least test what happens on your systems when the leap second is announced and when it is inserted.

Like in 2012, I’ll soon start testing the behavior of both ntpd and Linux. The test framework will be the same as last time, but with the added wisdom gained fighting against leapocalypse. This is what I plan to add to the picture:

  • more than one NTP test server: either three or four, to measure the convergence speed more accurately;
  • test under heavy load;
  • test on at least some of the software that we run;
  • test with the kernel discipline disabled to compensate for kernel bugs;
  • test on both stock ntpd and the recently released 4.2.8;
  • test on Debian Jessie, since it will be surely released before the leap second takes place;
  • prepare CFEngine policies to handle the leap second and test those as well.

Like last time I’ll share my results. If you’ll be doing similar tests in your environment, I’ll be more than glad to see your findings. This is going to affect us all, the more information we share, the more chances we have to overcome the bugs.

Image from

Managing apt_preferences with CFEngine

Linux Mint Debian EditionSince when I started using Linux Mint Debian Edition (LMDE), it has become increasingly important to have my apt_preferences set up correctly, so that I could pull packages from Debian testing (currently codenamed “jessie”) if I wanted, but use LMDE packages by default. It took me some time and research to find the right settings.

Once found the right settings, and discovered that I needed to “hardcode” the hostname of the LMDE mirror I was using, there came the problem to have the apt sources and apt preferences in sync. It seemed a nice task for CFEngine, and I started to work on managing my apt preferences. The first solution I had in mind didn’t work properly due to a bug in CFEngine that wasn’t yet fixed in version 3.4.4, the one I’m using. The solution below works. It’s a bit more cumbersome than the one that I had in mind, but it should be useful nonetheless 🙂 It’s an edit_line bundle:

# This bundle is used to compile an apt-preferences file (usually
# /etc/apt/preferences, or another file in /etc/apt/preferences.d.
# It receives in input the name of an array with the following
# structure:
# pin[id][Package] string => "package_identifier"
# pin[id][Pin]     string => "pin for this package"
# pin[id][Prio]    string => "priority for this package"
# We use preserve_block in insert_lines, so all the three items must
# be present (Package, Prio, and Pin). Besides, it is advisable that
# the whole file is managed from a blank state.

bundle edit_line apt_preferences(pins) {
      "n" string => "$(const.n)" ;
      "ids" slist => getindices("$(pins)") ;

      "Package: $($(pins)[$(ids)][Package])$(n)Pin: $($(pins)[$(ids)][Pin])$(n)Pin-Priority: $($(pins)[$(ids)][Prio])$(n)",
	  insert_type => "preserve_block" ;


Put the bundle in one of your libraries, import it, and then you can use it like this:

bundle agent test {
      "ids" slist => {"lmde","mint","jessie","wheezy"} ;

      "pin[lmde][Prio]"   int => "700" ;
      "pin[mint][Prio]"   int => "700" ;
      "pin[jessie][Prio]" int => "-1"  ;
      "pin[wheezy][Prio]" int => "1"   ;

      "pin[lmde][Pin]"   string => "origin" ;
      "pin[mint][Pin]"   string => "origin" ;
      "pin[jessie][Pin]" string => "release n=jessie"  ;
      "pin[wheezy][Pin]" string => "release n=wheezy"   ;

      "pin[$(ids)][Package]" string => "*" ;

	  create        => "yes",
	  edit_defaults => empty,
	  perms         => owner("bronto"),
	  edit_line     => apt_preferences("") ;


The test bundle above will create a file like this:

Package: *
Pin: origin
Pin-Priority: 700

Package: *
Pin: origin
Pin-Priority: 700

Package: *
Pin: release n=jessie
Pin-Priority: -1

Package: *
Pin: release n=wheezy
Pin-Priority: 1