But before even beginning to tell the story, I'd like to say that I have nothing against the puppet community, which is in fact a great one: be it on IRC or in a mailing list, they were always helpful, never rude; yes, I hated when they said "well, puppet is not designed to do that", but at least they were honest 😉

The fact is: I never fell in love with puppet. I've always been frustrated by its unpredictability, badly implemented functionalities (fileserver anyone?), bad design decisions (e.g.: why use the hostname as a key in the hosts file, where all systems use the IP as the key?), and no scalability out-of-the-box (e.g.: no provisions for hierarchical puppetmaster architectures, need for external resources like an nginx reverse proxy to make it scale…) to mention the first ones that come to mind.

With all this bad, how did it happen that I got trapped into puppet?

It's a long story, so take your time and relax. …
One of my first task at $WORK was to evaluate puppet and cfengine to understand which one was more fit to be used company-wide. One of the reasons why the task was assigned to me was my previous experience with cfengine 2 at $PREVIOUS_EMPLOYER, which gave me some insight in cfengine's philosophy and architecture. Puppet was chosen as competitor because it was already used in some departments of the company, although in completely separate installations.

The comparison was just "on paper": check features, architecture, language from the documentation, and decide which one was a better choice for us. The target was to create a company-wide configuration management infrastructure, with a library of easy, ready-to-use, pre-canned "modules" (be them puppet modules or cfengine policies). Besides, the infrastructure must allow for "federation": it should have been possible for existing implementations to join the infrastructure, allowing their administrators to retain full control of their stuff.

cfengine turned out to be a better product overall: well designed, secure, robust, with solid theoretical foundations; on the other hand, it was quite clear that puppet was not designed to scale. Cfengine had a huge disadvantage though: compared to puppet, the language was overly complicated and the learning curve much steeper at the beginning: not exactly the features you want for a tool that should prompt people to use it.

In the end, it looked like puppet could do everything cfengine could, the learning curve seemed easier, and puppet was –as said– already in use in the company. We decided to go for that.

The infrastructure was designed an planned. As soon as I started the implementation, problems started to surface: puppet was not designed to support a hierarchy of puppetmasters, the file serving functionality was crappy, puppetd hogged the CPU doing nothing due to a bad interaction between it and the ruby interpreter packaged with Debian Lenny… every time I moved on to accomplish what was supposed to be an easy task, there I was stuck for 10 times the time I had estimated in the beginning. Every single sensible functionality I was trying to implement required a workaround to work properly.

When it was finally time to put manifests together, more problems surfaced. Easy things were actually very easy, much easier than accomplishing the same tasks with cfengine. But more difficult was the problem, much and much harder was the solution: sometimes, to adapt a module to a slight variation of the problem it was supposed to solve, it was necessary to completely redesign it. In the end, I had to make a module of mine completely parametric, because it was simply impossible to fully solve the problem at hand otherwise. I often become frustrated by puppet's limitations and the constant need to work around its limitations and bad design choices.

It was a bad time. Nevertheless, I kept working, sometime dedicating a lot of my work to the puppet infrastructure, sometimes sparingly due to other high priority tasks. Eventually, the task became too big to be managed successfully by myself alone: I was stuck with version 0.25.4, when puppet 2.6 and then 2.7 came out. Then Debian Squeeze was out, and I was stuck with Lenny.

When the infrastructure was finally out, all the worst problems were worked around, the infrastructure could scale indefinitely and allow for "federations", there was a core library that supported all the basic needs, and enough documentation to kick-start a single client or a full project inside the infrastructure.

But, as said, the infrastructure was big, obsolete, over-engineered for the small set of machines I was managing. Early puppet implementers didn't want to enter the infrastructure, not even in a "federated" mode. New implementers started on Squeeze and with the puppet 2.6 in bundle with that distribution, and it was to hard for them to both enter the infrastructure and adapt existing manifests to the new environment.

In the end, both early and new puppet implementers borrowed something from the infrastructure, but chose to go their own way, and out of it. My infrastructure was a wonderful piece of system engineering that no one wanted to use, and the only user (me) was not glad at all to use puppet.

At that point I said to myself: "If I am the only one who will use this puppet infrastructure, I am not going to use it". Soon after, I started working on cfengine. It was the end of 2011.

And it was hard.

Things hadn't changed in cfengine land since the last time I had visited it: lots and lots of documentation, but almost impossible to find a correct path through it. And the tutorial, honestly, wasn't: much theory and very little and very basic practical examples made it sound like: "See, this is how you put together a bolt and a nut. Understood? Good! Now go build a plane".

It was an hard time again, everything was a trial and error, and often the error was due to my ignorance of something (e.g.: a function, a bundle or body in the standard library, or how much normal ordering really matters…); or, in the worst case, it was a philosophical problem, e.g.: something that was really important to me was deemed as not important by those who tried to help me (for example: formatting a configuration file in a certain way).

Yes, that was frustrating, too. Until something happened, and things changed like when the night turns into a new day.

When the early release of "Learning cfengine 3" was announced, I think I was one of the first people who bought it. I started reading it, and it was amazing: all the things that were confusingly buzzing in my head started to line themselves orderly. In a very little time, I was able to do much more than I had been able to do in months. And I finally could appreciate the good qualities of cfengine I liked from version 2: powerful, predictable, lightweight.

Where am I today? Well, talking by example: the gun is almost ready, and it's now time to arm, deploy, and fire. When I'll be finished with that, cfengine will have kicked puppet out of my way, hopefully for a long time.

Cheers!

Workaround image from Web Analytics Inside
Gears image from clintonio:
Information overload image from Controlled chaos