External node classification, the CFEngine way

CFEngineAgentExternal node classification is a Puppet functionality, where it is left to a program external to Puppet (the external node classifier, ENC) to decide which contexts apply to the node being configured. This approach is opposed to the “standard”, basic one, where the configuration applied to a node is completely defined in Puppet files (manifests), and site.pp in particular. ENC programs really show their power and usefulness where the same ruleset is used to manage a large number of servers, with many different combinations of configurations are applied. In this case, pulling the configuration information from a structured data base instead of plain files scales much better. One can write his own ENC, or use one of the several available, hiera being a well-known one.

Note: files and ENC are just two possible ways to classify nodes in Puppet; besides, classification happens on the puppetmaster, while in CFEngine all configuration decisions are taken on the node running the agent. You may read more information on node classification in puppet here, but then we’ll leave you alone and keep reading this post 🙂

Now transpose to CFEngine. Being it generally more “low-level” than Puppet, it provides no ENC mechanism out of the box, but plenty of possibilities to implement one yourself. I first checked what were the available options. I got very nice suggestions, notably one by LinkedIn’s Mike Svoboda, where they use a Yahoo! open source product called Range to store the data about the nodes, then they dump the data in JSON format, and finally they use a bash script run as a CFEngine module to raise the relevant classes. As scalable and sophisticated as it is, it was way too much than what I needed.

ENC, in my case, had two purposes: the first: allow us to scale better (and that’s a common trait in all ENC mechanisms) the second: take as much configuration information as possible out of the policies and in plain text files. This way, the access barrier for the non-CFEngine savvy people in the company would be lowered significantly. This approach is actually closer to Neil Watson’s and EvolveThinking’s CFEngine library (based on CSV files) than to the otherwise wonderful LinkedIn approach.

I sowed this information and ideas in my brain and let them sprout in the background for a couple of weeks, letting the most complex solutions drop. The last one standing was, in my opinion, the best combination of power and simplicity. We’d use plain text files, the CFEngine’s own module protocol, and extra-simple scripts: a bash script for simple external node classification, and a Perl script for hierarchical node classification. I’ll summarize the module protocol first, and then show how we leverage it to achieve ENC.

Continue reading


Why I gave up puppet and chose cfengine 3

But before even beginning to tell the story, I'd like to say that I have nothing against the puppet community, which is in fact a great one: be it on IRC or in a mailing list, they were always helpful, never rude; yes, I hated when they said "well, puppet is not designed to do that", but at least they were honest 😉

The fact is: I never fell in love with puppet. I've always been frustrated by its unpredictability, badly implemented functionalities (fileserver anyone?), bad design decisions (e.g.: why use the hostname as a key in the hosts file, where all systems use the IP as the key?), and no scalability out-of-the-box (e.g.: no provisions for hierarchical puppetmaster architectures, need for external resources like an nginx reverse proxy to make it scale…) to mention the first ones that come to mind.

With all this bad, how did it happen that I got trapped into puppet?

It's a long story, so take your time and relax. … Continue reading

My first puppet facts

An apparently simple problem: have puppet manage an host's own entries: the localhost one, and the hostname's one. Well, we have plenty of facts to help us with this: we have ipaddress, and hostname, and fqdn. It should be simple, right?

But think. What happens if the host is multihomed? What if one of the interfaces is a bond interface? Does facter choose a value for ipaddress in a smart way?

Unfortunately, it doesn't. And to make it smart, you have to feed it the right thing. And to feed it, you need to write code in Ruby. Oh oh…

I decided to give it a try, and to pull in some shell script to actually do the job (bad thing, I know, but I really don't have time to learn ruby; and when I'll use some time to learn another programming language, I'd go through python and perl 6 first). … Continue reading

Drawing puppet class relationships with graphviz

In the last few days, I got the impression that my puppet class hierarchy was growing a bit out of control. In particular, it looked like "glue" classes aimed to simplify stuff were actually causing more harm than benefit. How I found out? Well, it was getting too complicated to understand what included what, and walking the classes by hand quickly became a terrible task.

So, I decided to go for something that could do the work for me: go through my modules, get the include statements, and draw the relationships. Luckily, I remembered I saw that graphviz was designed explicitly to ease that latest task. … Continue reading

Using a makefile to automate a puppet installation

The first make will back up some files. It is important to run it only once.

The second run, "make" alone, will go through the needed steps for installation: if rsync is not installed, it will install rsync; if the needed files are not present in the /puppet hierarchy, it will download the files using rsync; if puppet is not installed, it will install it; then, it will run puppetd with a bootstrap configuration file and… fail!

Yes, fail. Because you need to sign the host certificate, don't you? Once you signed it on the master node, then you'll run make config. If everything is properly configured, this will set up your host according to the manifests.

The last command, make test will just run a puppetd --test to ensure that everything is properly set up.

And that's all. Installing one host this way takes about 5 minutes, downloads included. And using cssh I could even install puppet on several machines in parallel.

Am I missing something? Oh, yes, the real Makefile 🙂


all: install config

install: /usr/bin/puppetd

        puppetd --config /puppet/common/files/bootstrap/client.conf --server $(PUPPETMASTER) --test

        puppetd --test

        -rm -rf /puppet
        -apt-get remove puppet facter
        -apt-get autoremove

        cp /etc/apt/sources.list{,.dist}

        apt-get install rsync

/puppet/usa/files/sources.list: sync-puppet-conf

/puppet/volatile/keys/apt: sync-puppet-conf

update: /puppet/usa/files/sources.list /puppet/volatile/keys/apt
        cp /puppet/usa/files/sources.list /etc/apt/sources.list
        apt-key add /puppet/volatile/keys/apt 
        apt-get update
        apt-key update

sync-puppet-conf: /usr/bin/rsync
        rsync -zav $(SYNCSERVER)::PuppetConf /puppet

/usr/bin/puppetd: update
        apt-get install puppet

What is a “Configuration Management Tool” (and why you want to use it)


A number of contrasting definitions exist today for "configuration management"; it is enough to search "what is configuration management?" in Google and skim through the results. Since different definitions would lead to different interpretations of what a configuration management tool is, it's better to stop introduce the subject before we even start talking. Besides, since "Configuration Management System" would lead to the acronym "CMS" which is widely used now for stuff that has nothing to do with configuration management, we'll abdicate and talk about "Configuration management tools" or CMTs.

What is a CMT?

Configuration management tools are, well, tools that aim to empower system administrators to efficiently manage large installations of systems. They generally try to achieve this goal by classifying servers and applying specific configurations to each class, so that hosts in a chosen class are configured in the same way and work in the same way[1].

A strong recommendation (read: a requirement) of these tools is that the tool's configuration files are centralized and versioned in a version control system of some kind. The VCS is not part of the CMT, and the CMT usually doesn't impose any specific one. This is good news, since you can keep using your favourite one. By the way, in large installations you often have different people managing different parts. This may require different access rights to the CMT configuration, which may lead to the choice of a specific VCS that implements some sort of access control.

[1] More or less. We all know from experience that it is not enough for two servers to share the same configuration to work the same way. Besides, more other “noise” can affect the systems (e.g.: a badly hand-configured service, out of the control of the configuration management tool, may disrupt an otherwise well-working service under control).

But the most important thing comes when the human brain is involved. System administrators often tend to think at each system as a single entity that they configure by tweaking configuration files, and tuning the operating system here and there. Sysadmins have the concept of “class of servers” (a set of servers that perform the same function, e.g.: machines x, y and z are web servers). But the class is often created by repeating the same configuration job by hand on each machine.

This is not the case with CMTs. You now manage classes, not (just) servers: you don't configure servers (one or one hundred they may be): you configure a class. And going at the level of, say, the particular service's configuration file is left to the CMT.

When you are using a CMT you should limit low-level activities, directly accessing the operating system entities (files, devices, processes…), and leave it to the CMT to do such things, for a) your changes may be overridden by the CMT, and you'll get the (wrong) impression that it gets in the way instead of helping you, and 2) you'd get back to patch the single instance of the server instead of configuring the whole class.

In other words: if you are going to use a CMT, any CMT, you'll also need to change the way you think about server management. Without that change, a CMT may never fully work for you, since you'll feel it getting in your way rather than helping you.

The tools we'll compare, and how

The tools we'll compare are cfengine and puppet.

With its birth dating back in 1993, Cfengine is probably the oldest, best known, and more powerful tool around. It lays its foundation on the scientific theories of his creator, Marc Burgess. The current version of cfengine is 3.0.5.

Puppet is a younger project, started by Luke Kanies. It is being actively developed and has a community of enthusiast around it. The current stable version is 0.25.5, while the next stable will see a numbering change and become 2.6 (instead of 0.26.x).

Each tool will be analysed under three points of view: architecture (how the functionalities of the tool are organized), philosophy (the principles behind the tool) and peculiarities (things that make the tool special). This comparison will be rather high-level, and will focus on features. Rather than, say, showing examples about how to perform a specific task, we'll try to asnwer the following questions:

  1. Can this tool handle multiple service clusters (say 10 clusters of a hundred machine each), allowing different policies and access restrictions to be applied for configuration files (per cluster, per team, per user)?
  2. Can this tool handle multiple datacenters, where a single service is typically deployed in more than one datacenter?
  3. Does this tool natively integrate with monitoring tools (such as Nagios, Munin…) so that changes in the CMT configuration are reflected in the monitoring tools?
  4. Is it easy to maintain configuration repositories so that templates can be stored for easy modification and duplication?
  5. Is configuration change history supported?
  6. Is change request/approval/notification supported?

Some of this questions can get an answer right away.

Questions 4 and 5 all depend to the chosen VCS. Answer to question 5 is “yes” for any VCS, and answer to question 4 is “yes, once/given that you know the chosen VCS”.

Answer to question 6 is: “No or partially; that much depends on people role's and procedures they follow”. More on this later, along with the answers to questions 1, 2, 3.

Once done with the analysis of both tools, I'll try to answer to the questions above, and I'll talk about things that didn't find an answer or a solution. This will be a separate post.

And that's all for now. I'll see you next time with cfengine.