Why I gave up puppet and chose cfengine 3

But before even beginning to tell the story, I'd like to say that I have nothing against the puppet community, which is in fact a great one: be it on IRC or in a mailing list, they were always helpful, never rude; yes, I hated when they said "well, puppet is not designed to do that", but at least they were honest ๐Ÿ˜‰

The fact is: I never fell in love with puppet. I've always been frustrated by its unpredictability, badly implemented functionalities (fileserver anyone?), bad design decisions (e.g.: why use the hostname as a key in the hosts file, where all systems use the IP as the key?), and no scalability out-of-the-box (e.g.: no provisions for hierarchical puppetmaster architectures, need for external resources like an nginx reverse proxy to make it scale…) to mention the first ones that come to mind.

With all this bad, how did it happen that I got trapped into puppet?

It's a long story, so take your time and relax. …
One of my first task at $WORK was to evaluate puppet and cfengine to understand which one was more fit to be used company-wide. One of the reasons why the task was assigned to me was my previous experience with cfengine 2 at $PREVIOUS_EMPLOYER, which gave me some insight in cfengine's philosophy and architecture. Puppet was chosen as competitor because it was already used in some departments of the company, although in completely separate installations.

The comparison was just "on paper": check features, architecture, language from the documentation, and decide which one was a better choice for us. The target was to create a company-wide configuration management infrastructure, with a library of easy, ready-to-use, pre-canned "modules" (be them puppet modules or cfengine policies). Besides, the infrastructure must allow for "federation": it should have been possible for existing implementations to join the infrastructure, allowing their administrators to retain full control of their stuff.

cfengine turned out to be a better product overall: well designed, secure, robust, with solid theoretical foundations; on the other hand, it was quite clear that puppet was not designed to scale. Cfengine had a huge disadvantage though: compared to puppet, the language was overly complicated and the learning curve much steeper at the beginning: not exactly the features you want for a tool that should prompt people to use it.

In the end, it looked like puppet could do everything cfengine could, the learning curve seemed easier, and puppet was –as said– already in use in the company. We decided to go for that.

The infrastructure was designed an planned. As soon as I started the implementation, problems started to surface: puppet was not designed to support a hierarchy of puppetmasters, the file serving functionality was crappy, puppetd hogged the CPU doing nothing due to a bad interaction between it and the ruby interpreter packaged with Debian Lenny… every time I moved on to accomplish what was supposed to be an easy task, there I was stuck for 10 times the time I had estimated in the beginning. Every single sensible functionality I was trying to implement required a workaround to work properly.

When it was finally time to put manifests together, more problems surfaced. Easy things were actually very easy, much easier than accomplishing the same tasks with cfengine. But more difficult was the problem, much and much harder was the solution: sometimes, to adapt a module to a slight variation of the problem it was supposed to solve, it was necessary to completely redesign it. In the end, I had to make a module of mine completely parametric, because it was simply impossible to fully solve the problem at hand otherwise. I often become frustrated by puppet's limitations and the constant need to work around its limitations and bad design choices.

It was a bad time. Nevertheless, I kept working, sometime dedicating a lot of my work to the puppet infrastructure, sometimes sparingly due to other high priority tasks. Eventually, the task became too big to be managed successfully by myself alone: I was stuck with version 0.25.4, when puppet 2.6 and then 2.7 came out. Then Debian Squeeze was out, and I was stuck with Lenny.

When the infrastructure was finally out, all the worst problems were worked around, the infrastructure could scale indefinitely and allow for "federations", there was a core library that supported all the basic needs, and enough documentation to kick-start a single client or a full project inside the infrastructure.

But, as said, the infrastructure was big, obsolete, over-engineered for the small set of machines I was managing. Early puppet implementers didn't want to enter the infrastructure, not even in a "federated" mode. New implementers started on Squeeze and with the puppet 2.6 in bundle with that distribution, and it was to hard for them to both enter the infrastructure and adapt existing manifests to the new environment.

In the end, both early and new puppet implementers borrowed something from the infrastructure, but chose to go their own way, and out of it. My infrastructure was a wonderful piece of system engineering that no one wanted to use, and the only user (me) was not glad at all to use puppet.

At that point I said to myself: "If I am the only one who will use this puppet infrastructure, I am not going to use it". Soon after, I started working on cfengine. It was the end of 2011.

And it was hard.

Things hadn't changed in cfengine land since the last time I had visited it: lots and lots of documentation, but almost impossible to find a correct path through it. And the tutorial, honestly, wasn't: much theory and very little and very basic practical examples made it sound like: "See, this is how you put together a bolt and a nut. Understood? Good! Now go build a plane".

It was an hard time again, everything was a trial and error, and often the error was due to my ignorance of something (e.g.: a function, a bundle or body in the standard library, or how much normal ordering really matters…); or, in the worst case, it was a philosophical problem, e.g.: something that was really important to me was deemed as not important by those who tried to help me (for example: formatting a configuration file in a certain way).

Yes, that was frustrating, too. Until something happened, and things changed like when the night turns into a new day.

When the early release of "Learning cfengine 3" was announced, I think I was one of the first people who bought it. I started reading it, and it was amazing: all the things that were confusingly buzzing in my head started to line themselves orderly. In a very little time, I was able to do much more than I had been able to do in months. And I finally could appreciate the good qualities of cfengine I liked from version 2: powerful, predictable, lightweight.

Where am I today? Well, talking by example: the gun is almost ready, and it's now time to arm, deploy, and fire. When I'll be finished with that, cfengine will have kicked puppet out of my way, hopefully for a long time.


Workaround image from Web Analytics Inside
Gears image from clintonio:
Information overload image from Controlled chaos


9 thoughts on “Why I gave up puppet and chose cfengine 3

  1. You're absolutely right, and I'd be grateful if you could back-tweet to that. I don't have a twitter account, so I can't do that myself; however, I'll comment in cfengine's help list later today.Besides, I hope my post is clear about that. I am not saying that we gave up puppet as a company (that would be a lie, in fact). That's a story about myself and my personal experience, not about the company.

  2. James Turnbull writes:To be fair you're also not making an apples and apples comparison – you were comparing Puppet 0.25.4 running on an older Debian release with CFEngine 3. A LOT has changed in the 20 odd releases since then.

  3. Laurent Raufaste writes:I experienced the same as you, except that I did not even gave Puppet a try, and instead I spent 1 month to think about which one to choose: Puppet, CFEngine or Chef. In the end I chose CFEngine and we are managing hundreds of servers on AWS with git a single t1.micro CFEngine server. Amazing.The book from Diego changed everything for me too. It should be the official tutorial.

  4. Originally posted by anonymous:

    James Turnbull writes:To be fair you're also not making an apples and apples comparison – you were comparing Puppet 0.25.4 running on an older Debian release with CFEngine 3. A LOT has changed in the 20 odd releases since then.

    Hi James!I could not compare 2.6/2.7 to cfengine 3 simply because I didn't manage to use those. This happened for two interconnected reasons: the first is that, as I said, the infrastructure was growing too big to allow me to both develop it (developing more manifests/functionalities, and improve existing ones), and the second was that switching from 0.25 to 2.6 required more planning and testing and development than I had time for.I tried to describe my frustrations with puppet in the most objective way I could. I didn't cut it to a short "puppet sucks". Instead, I said exactly which version of puppet I was using, on which OS, and mentioned a few specific problems I've had. Again, I didn't cut it to a short "it doesn't work" or "it has problems".Choosing a tool for your work is often a philosophical question. What I think is clear from my post is that puppet's philosophy didn't fit mine. I got mad when I couldn't use the fileserver the way it was supposed to be used because that could bring the puppetmaster process to its knees; I got mad when I saw people running puppetd in cron or by hand only because, on certain platforms, it had problems when running as a daemon (if a daemon has problems running as a daemon, isn't it badly broken?); I got mad when I had to run puppet three times by hand to have a configuration completely applied to a system (letting puppet do it with its own schedule would have meant waiting 90 minutes…), I got mad when I got three different orders of execution in three different run, even when each run didn't change the system in any way, I got mad after too many times I couldn't succeed doing something sensible and the answer was "puppet is not designed to do that".Puppet has surely improved since 0.25, and has a big community of users (a very nice community, I restate). But I don't see that its philosophy has changed. It makes its own level of abstraction on top of the system: use it, or lose it. I've been bitten by that too many times, and I chose to lose it, with no regrets. I have to implement my own level of abstraction on cfengine: it's not a workaround, and I'm OK with that.Sincerely– bronto

  5. Anonymous writes:I got mad when I had to run puppet three times by hand to have a configuration completely applied to a system

    Do you not know how to specify dependencies? Doesn't seem like a problem with Puppet here…

  6. Originally posted by anonymous:

    Anonymous writes:> I got mad when I had to run puppet three times by hand to have a configuration completely applied to a systemDo you not know how to specify dependencies? Doesn't seem like a problem with Puppet here…

    Of course I do: notify, require and stuff. But anything else that was not dependency-bound was scheduled for execution by puppet in a seemingly random order — I am sure it was not random, of course, but the logic seemed quite mysterious even for people more puppet-savvy than I was.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s