cfengine

A dockerized policy hub

/ bronto / 1 Comment

DockerThis is a quick post, apologies in advance if it will come out a bit raw.

I’ve been reading about docker for a while and even attended the day of docker in Oslo. I decided it was about time to try something myself to get a better understanding of the technology and if it could be something useful for my use cases.

As always, I despise the “hello world” style examples so I leaned immediately towards something closer to a real case: how hard would it be to make CFEngine’s policy hub a docker service? After all it’s just one process (cf-serverd) with all its data (the files in /var/cfengine/masterfiles) which looks like a perfect fit, at least for a realistic test. I went through the relevant parts of the documentation (see “References” below) and I’d say that it pretty much worked and, where it didn’t, I got an understanding of why and how that should be fixed.

Oh, by the way, a run of docker search cfengine will tell you that I’m not the only one to have played with this 😉

Continue reading →

systemd unit files for CFEngine

/ bronto / Leave a comment

systemd logoLearning more of systemd has been on my agenda since the release of Debian 8 “Jessie”. With the new year I decided that I had procrastinated enough, I made a plan and started to study according to the plan. Today it was time for action: to verify my understanding of the documentation I read up to now, I decided to put together unit files for CFEngine. It was an almost complete success and the result is now on GitHub for everyone to enjoy. I would appreciate if you’d give them a shot and report back.

Main goals achieved:

  1. I successfully created three service unit files, one for each of CFEngine’s daemons: cf-serverd, cf-execd and cf-monitord; the units are designed so that if any of the daemon is killed for any reason, systemd will bring it back immediately.
  2. I successfully created a target unit file that puts together the three service units. When the cfengine3 target is started, the three daemons are requested to start; when the cfengine3 target is stopped, the three daemons are stopped. The cfengine3 target completely replaces the init script functionality.

Goal not achieved: I’ve given a shot at socket activation, so that the activation of cf-serverd was delayed until a connection was initiated to port 5308/TCP. That didn’t work properly: systemd tried to start cf-serverd but it died immediately, and systemd tried and tried again until it was too much. I’ll have to investigate if cf-serverd needs to support socket activation explicitly or if I was doing something wrong. The socket unit is not part of the distribution on GitHub but its content are reported here below. In case you spot any problem please let me know.

Continue reading →

Managing system services with CFEngine

/ bronto / Leave a comment

An important system service is not running...

I have experienced that when people talk about a system’s configuration, they mostly think of software to be installed and configuration files to be deployed. That’s true, they are part of a system configuration, but there’s more to it — if Configuration Management was only that, you could rightfully call it “provisioning” instead. For example, another part of a system’s configuration is that certain critical services must be running and/or certain other services must not be running. And in fact, any configuration management tool has provisions to manage system services and ensure they are in the desired state (while they may differ a lot on the “when” and “how” and “how often” the state is checked).

CFEngine is no exception. You can take advantage of ready-to-use frameworks like NCF or EFL, or  roll your own checks. What I’m presenting you today is a simple bundle that I wrote called watch_service, that you can use to ensure that certain system services are up or down.

My approach is similar to NCF’s bundle called service_action in that it tries to provide a generic, system-agnostic bundle to manage services but with a few differences:

  • while service_action relies on information in NCF itself to make the bundle simpler to use, my watch_service relies only on CFEngine’s standard_services knowledge as available in the standard library;
  • while service_action returns information to the agent in the form of namespace-scoped classes (e.g.: the service was in the desired state, or the service was not in the desired state and the problem has been fixed successfully), watch_service only reports about the events by means of another bundle called report, whose code will be also provided in the last part of this post.
  • service_action supports many different actions, watch_service only supports “up” (ensure the service is running) or “down” (ensure the service is not running).

Continue reading →

Rudimentary compliance report for CFEngine

/ bronto / Leave a comment

In CFEngine community you don’t have a web GUI with compliance report. You can get them via EvolveThinking’s Delta Reporting, but if you can’t for any reason, you need to find another way.

A poor man’s compliance report at the bundle level can be extracted via the verbose output. This is how I’ve used it to ensure that a clean-up change in the policies didn’t alter the overall behavior:

cf-agent -Kv 2>&1 | perl -lne 'm{verbose: (/.+): Aggregate compliance .+ = (\d+\.\d%)} && print "$1 ($2)"'

These are the first ten lines of output on my workstation:

bronto@brabham:~$ sudo cf-agent -Kv 2>&1 | perl -lne 'm{verbose: (/.+): Aggregate compliance .+ = (\d+\.\d%)} && print "$1 ($2)"' | head -n 10
/default/banner (100.0%)
/default/inventory_control (100.0%)
/default/inventory_autorun/methods/'proc'/default/cfe_autorun_inventory_proc (100.0%)
/default/inventory_autorun/methods/'fstab'/default/cfe_autorun_inventory_fstab (100.0%)
/default/inventory_autorun/methods/'mtab'/default/cfe_autorun_inventory_mtab (100.0%)
/default/inventory_autorun/methods/'dmidecode'/default/cfe_autorun_inventory_dmidecode (100.0%)
/default/inventory_autorun (100.0%)
/default/inventory_linux (100.0%)
/default/inventory_lsb (100.0%)
/default/services_autorun (100.0%)

Not much, but better than nothing and a starting point anyway. There is much more information in the verbose log that you can extract with something slightly more elaborated than this one-liner. Happy data mining, enjoy!

hENC version 3 released

/ bronto / Leave a comment

github-logo Today I am releasing the version 3 of hENC, the radically simple hierarchical External Node Classifier (ENC) for CFEngine (version 2 was released at the end of May and added support for data containers).

This version adds new features and bug fixes, namely:

  • implemented !COMMANDS: a ! primitive is added to specify commands; three commands exist currenty: !RESET_ACTIVE_CLASSES to make hENC forget about any class that was activated up to that point, !RESET_CANCELLED_CLASSES ditto for cancelled classes, and !RESET_ALL_CLASSES that makes hENC forget about any class that was activated or cancelled;
  • fixed enc.cf, so that it is possible to run the henc module more than once during the same agent run;
  • added a Changelog;
  • improved tests: tests have been added for the new features and the whole test suite has been improved to support the TAP protocol; for example, it’s now it’s possible to use the prove utility to verify if hENC works correctly on your system before trying the installation.

See the README and Changelog for more information.

The leap second aftermath

/ bronto / 1 Comment

TurnBackTimeThe leap second is finally behind us, and for the first time it has been transformed in an event. That had the unfortunate consequence that many channels where useful information had flown in the previous events were now flooded with bullshit. But it’s over. A giant army of idiots has finally stopped asking “what will you do with your extra second?”, a smaller but still noticeable army of inaccurate writers and journalists won’t write for a while that the atomic clocks need to be stopped for a second to realign with the Earth (?!?!?!?!?!?). We can now sit, look back and save some take-aways for the next edition of the event.

Continue reading →

How to make CFEngine recognize if systemd is used in Debian

/ bronto / Leave a comment

CFEngine 3.6 tries to understand if a Linux is using systemd as init system by looking at the contents of /proc/1/cmdline, that happens in bundle common inventory_linux. That’s indeed a smart thing to do but unfortunately fails on Debian Jessie, where you have:

root@cf-test-v10:~# ls -l /sbin/init
lrwxrwxrwx 1 root root 20 May 26 06:07 /sbin/init -> /lib/systemd/systemd

the pseudo-file in /proc will still report /sbin/init and as a result the systemd class won’t be set. This affects services promises negatively and therefore I needed to make our policies try to outsmart the inventory 😉 These promises, added in a bundle of ours, did the trick:

bundle common debian_info {
  vars:
    init_is_link::
      "init_link_destination"
          string => filestat("/sbin/init","linktarget") ;

  classes:
    init_is_link::
      "systemd"
          expression => regcmp("/lib/systemd/systemd",
                               "$(init_link_destination)"),
          comment => "Check if /sbin/init links to systemd" ;

    debian::
      "init_is_link"
          expression => islink("/sbin/init"),
          comment => "Detect if init is a link" ;
}

Notice that our bundle is actually bigger, I cut off all the promises that were not relevant for this post. Enjoy!

A humble attempt to work around the leap second, 2015 edition

/ bronto / 7 Comments

TurnBackTimeUpdate: Watch out for public servers not announcing the leap second! In the last few minutes we have been observing a number of public servers (even stratum 1) that don’t announce the leap second. If the majority of your upstream doesn’t announce the leap second, your clients won’t trigger it. If that’s your case, you can use ntpd’s leapfile directive and a leap second file to provide your own servers with the correct information. Check the ntpd documentation for more information.

Update: Miroslav Lichvar has counted the public servers that are announcing the leap second on a per-country basis. You can find his stats on pastebin.

I have been running simulations for the upcoming leap second for a few weeks now. While some mysteries haven’t been solved yet, I was finally able to put together a configuration for our servers and clients that satisfies to the following requirements (where do these requirements come from? That is explained further down in the article):

  1. it works on Debian Linux Squeeze, Wheezy and Jessie
  2. it keeps the Linux kernel out of the game, in order to avoid triggering unknown kernel bugs
  3. it avoids backward steps of the clock
  4. the clock converges to the right time in an acceptable amount of hours
  5. it doesn’t hog public services

What this solution doesn’t provide: this is neither Google’s leap smear nor Amazon’s: you use standard ntpd code with no changes; this is not a fast clock slew as chrony’s either. Servers/clients have evolved predictably during most of the simulations and shouldn’t diverge too much from each other, but there are conditions where you may observe offsets between them in the order of magnitude of 0.1s. That should still be bearable though and will still save you from the headache of kernel bugs or jumps back in time. In order to work properly, this solution must make a few assumptions:

  1. you have at least four internal NTP servers, synchronized with at least four public servers and/or internal specialized time sources
  2. your clients use at least four of your own internal NTP servers and no external NTP server
  3. you use unicast NTP packets (broadcast and multicast will probably work as well or even better, but they haven’t been tested in my simulations)
  4. you are using ntpd (the reference implementation) version 4.2.8p3 (earlier versions have a bug that will make our countermeasures against clock stepping ineffective)

Let’s look at the implementation on both server and client side, which is pretty similar but with a few important differences. Continue reading →

Bug or feature? Dereferencing of arrays and namespaces

/ bronto / Leave a comment

CFEngineAgentNow that the upgrade from 3.4 to 3.6 is advancing slowly but steadily I am starting to check the features that are new in 3.6 compared to 3.4.  According to the docs namespaces were actually introduced in 3.4.0, but I didn’t take advantage of them yet, and it’s time to start.

When something is declared in a namespace (a bundle, a variable or whatnot) it must be referred to with its namespace. For example, if you declare a bundle test in the namespace nstest, you’ll refer to that bundle from outside the namespace (e.g. in the bundlesequence) as nstest:test. If you declare a variable, for example an array called conf in that bundle, that will be nstest:test.conf outside the namespace. So far so good.

Now, what happens inside the namespace? Well, I found one fact that is indeed surprising.

Continue reading →

Bug or feature? Change in behaviour in CFEngine templates

/ bronto / 3 Comments

CFEngineAgentToday I stumbled upon an unexpected behaviour change in CFEngine templates when upgrading from version 3.4 to 3.6. The change is not documented anywhere in the Changelog, so I am really not sure if it’s a bug or a feature. In any case, it is something to be aware of.

Take this template:

Normally, each line expands to an insert lines promise, which means that
duplicated lines may not be printed more than once. That behavior has
changed with CFEngine versions.
[%CFEngine cfengine_3_4:: %]
In this version of CFEngine, $(sys.cf_version), duplicates are printed once
In this version of CFEngine, $(sys.cf_version), duplicates are printed once
In this version of CFEngine, $(sys.cf_version), duplicates are printed once


that holds for blank lines, too

[%CFEngine cfengine_3_6:: %]
In this version of CFEngine, $(sys.cf_version), duplicates are preserved
In this version of CFEngine, $(sys.cf_version), duplicates are preserved
In this version of CFEngine, $(sys.cf_version), duplicates are preserved


[%CFEngine any:: %]
End of the story!

and this policy:

bundle agent test_example_cf3_template
{
  vars:
    cfengine_3_4::
      "templatedir" string => execresult("/bin/pwd","noshell") ;
      "testfile"    string => "/tmp/cf_34.txt" ;


    cfengine_3_6::
      "templatedir" string => "$(this.promise_dirname)" ;
      "testfile"    string => "/tmp/cf_36.txt" ;

  files:
      "$(testfile)"
	  create => "yes",
	  edit_template => "$(templatedir)/example_cf3_template.tmpl";

  reports:
    cfengine_3::
      "Check output in $(testfile)" ;
}

Now run the policy in both CFEngine 3.4 and 3.6. You’d expect to get two identical files: /tmp/cf_34.txt and /tmp/cf_36.txt. But they’re not.

Continue reading →