I have already posted links on Twitter, but here we go. I hope you’ll enjoy it!
A blog post with a sort-of transcript of the full talk (the long version, presented at cfgmgmtcamp) is almost ready. I’ll post it as soon as I finish my sources double-checking.
I needed to remove all fancy ANSI graphics and colors from the output of tree, and just print something that fit a plain text file. This did the trick:
tree -L 1 --noreport -S -n --charset=US-ASCII
E.g.:
bronto@brabham:/var/cfengine/git/common$ tree -L 1 --noreport -S -n --charset=US-ASCII . |-- controls |-- libraries |-- modules |-- services |-- sources |-- templates |-- tools |-- unit `-- update.cf
Thanks to all those who attended my talks at FOSDEM and cfgmgmtcamp, and double thanks to all those who also showed their appreciation: you made that experience a memorable one!!!
I have uploaded my slides on SpeakerDeck soon after my talks were over. Here you are the FOSDEM deck:
A video of the talk has also been released, but it’s unfortunately mute 😦 I hope that it will be fixed soon.
At FOSDEM I had only 20 minutes (that is: 20 plus questions, up to 25). However, the subject deserved to be dealt with with more attention. At cfgmgmtcamp I had 40 minutes — well, in the end they were only 35, but that’s Jonathan Clark’s fault ;-) so both the talk and the slide deck were more detailed than the ones from FOSDEM:
The plan is now to “unroll” the cfgmgmtcamp’s talk, and write an article from the transcript of the talk. It will take some more days however, but it will definitely come. Stay tuned!
I’ll be presenting at two conferences in two weeks.
On February 1st and 2nd I’ll be at FOSDEM in Brussels, where I’ll present the seminar “The classification problem: challenges and solutions“. In 20 minutes I’ll try to give you an idea of why classification in configuration management is critical, why it is difficult, and how you can make it sane.
Since 20 minutes may not be enough to treat the subject in detail, I got the opportunity to explain things in full at the Configuration management camp in Gent/Ghent/Gand, that will take place on February 3rd and 4th. I’ll be in the CFEngine / Rudder Community Room and will have 40 minutes to present “External node classification, the CFEngine way”. It’s the same seminar as FOSDEM; however, I’ll be able to show some examples of what makes classification difficult when you try to do it only “internally” with your CM tool of choice, and I’ll be able to show analogies and differences between Puppet and CFEngine when it comes to external node classification.
I hope to see some friends from the community there. If you’re going to any of these conference and want to say hello, just reach out 😉
I am working hard on my talk for FOSDEM, and don’t have much time to write detailed posts, so you’ll have to put up with me and be contempt with this “one liners” series 😉
In a memory shortage situation, it’s handy to understand which processes are memory hogs. Not only the ps command can tell you that: it can even return the process list sorted by memory occupation. At that point, getting the “top N” is a piece of cake. The following example shows the top 3 on my machine this morning, after shutting down and restarting a few of them:
bronto@brabham:~$ ps -e -o pid,user,rss,vsz,args --sort -vsz | head -n 4  PID USER      RSS   VSZ COMMAND 27971 bronto  213896 3099640 /usr/bin/python /usr/bin/hotot 27884 bronto  44572 2306604 liferea 24470 bronto  202724 2164244 /usr/bin/gnome-shell
Hope it helps!
On the policy hub, run:
cf-key -s | grep "`date +'%a %b %e'`" | awk '/^Incoming/ { print $3 }'
Beware that hostnames are truncated to 25 characters. However, you can replace $3 with $2 and get IPs instead.
A two of spades, to represent the bad moments from this year.
An ace of hearts, a big red heart to represent our love.
A four of diamonds, to wish you richness — not necessarily money, but a wealth of what everyone of us really needs.
And a face-down card, the uncertainty of the new year, wishing you that it will be a good one.
Best wishes!!!
External node classification is a Puppet functionality, where it is left to a program external to Puppet (the external node classifier, ENC) to decide which contexts apply to the node being configured. This approach is opposed to the “standard”, basic one, where the configuration applied to a node is completely defined in Puppet files (manifests), and site.pp in particular. ENC programs really show their power and usefulness where the same ruleset is used to manage a large number of servers, with many different combinations of configurations are applied. In this case, pulling the configuration information from a structured data base instead of plain files scales much better. One can write his own ENC, or use one of the several available, hiera being a well-known one.
Note: files and ENC are just two possible ways to classify nodes in Puppet; besides, classification happens on the puppetmaster, while in CFEngine all configuration decisions are taken on the node running the agent. You may read more information on node classification in puppet here, but then we’ll leave you alone and keep reading this post 🙂
Now transpose to CFEngine. Being it generally more “low-level” than Puppet, it provides no ENC mechanism out of the box, but plenty of possibilities to implement one yourself. I first checked what were the available options. I got very nice suggestions, notably one by LinkedIn’s Mike Svoboda, where they use a Yahoo! open source product called Range to store the data about the nodes, then they dump the data in JSON format, and finally they use a bash script run as a CFEngine module to raise the relevant classes. As scalable and sophisticated as it is, it was way too much than what I needed.
ENC, in my case, had two purposes: the first: allow us to scale better (and that’s a common trait in all ENC mechanisms) the second: take as much configuration information as possible out of the policies and in plain text files. This way, the access barrier for the non-CFEngine savvy people in the company would be lowered significantly. This approach is actually closer to Neil Watson’s and EvolveThinking’s CFEngine library (based on CSV files) than to the otherwise wonderful LinkedIn approach.
I sowed this information and ideas in my brain and let them sprout in the background for a couple of weeks, letting the most complex solutions drop. The last one standing was, in my opinion, the best combination of power and simplicity. We’d use plain text files, the CFEngine’s own module protocol, and extra-simple scripts: a bash script for simple external node classification, and a Perl script for hierarchical node classification. I’ll summarize the module protocol first, and then show how we leverage it to achieve ENC.
On September 19th and 20th I attended the OSS4B conference, and held the talk “a big project starts with a small plan” (the slides are available on SpeakerDeck). While I was at it, I also held the five-minute lightning talk “the value of personal configuration management”, about why using configuration management on your own machines may be a good idea (especially if you do that with CFEngine 🙂 the slides of the lightning talk are also on SpeakerDeck).
I left Oslo exhausted by the effort of the preparations (I had posted my proposal quite late, and my talk was confirmed only 45 days before the conference), and I was quite nervous for two main reasons; the first: well, it was my first international conference! and the second: Gene Kim, one of the two keynote speakers, was scheduled to hold a talk in parallel with mine. The trip from Oslo to Prato was quite a long one, and didn’t help make me feel more relaxed.
But things went quite smooth. The organizers were really welcoming and helpful, and a friendly atmosphere sprung immediately among the speakers. It made me smile to see a CFEngine junkie (me) sitting side by side with a Puppet junkie (Alessandro Franceschi) and a Chef hippie (Chris McClimans) and discussing without prejudice of the good and bad sides of their respective favorite toys, and about the advantages of configuration management in general. Not to mention the nice chats with the rest of the gang of badasses that was scheduled to talk at the conference. By the way, if you want to get a taste of how good the OSS4B speakers are, you can do it anytime and as long as you want via the OSS4B-2013 list I set up on twitter.
And my talk… well, I expected that everyone would go to Gene’s speech, but I actually managed to have a dozen or so spectators! I had fun giving my speech; I hope they had, too, and that (at least some of -) them found the talk interesting.
All in all, it was a nice “first time”, and a great opportunity to listen to experienced people, get interesting inputs, and discuss nice ideas. I’d like to express my thanks Barbara, Gabriele and Harald, and the whole organizing team through them: you did a great job, I look forward to see what next year’s edition will be!
I didn’t take many photos, and those taken with my phone are horrible quality, but I hope you’ll enjoy them.
Since when I started using Linux Mint Debian Edition (LMDE), it has become increasingly important to have my apt_preferences set up correctly, so that I could pull packages from Debian testing (currently codenamed “jessie”) if I wanted, but use LMDE packages by default. It took me some time and research to find the right settings.
Once found the right settings, and discovered that I needed to “hardcode” the hostname of the LMDE mirror I was using, there came the problem to have the apt sources and apt preferences in sync. It seemed a nice task for CFEngine, and I started to work on managing my apt preferences. The first solution I had in mind didn’t work properly due to a bug in CFEngine that wasn’t yet fixed in version 3.4.4, the one I’m using. The solution below works. It’s a bit more cumbersome than the one that I had in mind, but it should be useful nonetheless 🙂 It’s an edit_line bundle:
# This bundle is used to compile an apt-preferences file (usually
# /etc/apt/preferences, or another file in /etc/apt/preferences.d.
# It receives in input the name of an array with the following
# structure:
#
# pin[id][Package] string => "package_identifier"
# pin[id][Pin] string => "pin for this package"
# pin[id][Prio] string => "priority for this package"
#
# We use preserve_block in insert_lines, so all the three items must
# be present (Package, Prio, and Pin). Besides, it is advisable that
# the whole file is managed from a blank state.
bundle edit_line apt_preferences(pins) {
vars:
"n" string => "$(const.n)" ;
"ids" slist => getindices("$(pins)") ;
insert_lines:
"Package: $($(pins)[$(ids)][Package])$(n)Pin: $($(pins)[$(ids)][Pin])$(n)Pin-Priority: $($(pins)[$(ids)][Prio])$(n)",
insert_type => "preserve_block" ;
}
Put the bundle in one of your libraries, import it, and then you can use it like this:
bundle agent test {
vars:
"ids" slist => {"lmde","mint","jessie","wheezy"} ;
"pin[lmde][Prio]" int => "700" ;
"pin[mint][Prio]" int => "700" ;
"pin[jessie][Prio]" int => "-1" ;
"pin[wheezy][Prio]" int => "1" ;
"pin[lmde][Pin]" string => "origin debian.lth.se" ;
"pin[mint][Pin]" string => "origin packages.linuxmint.com" ;
"pin[jessie][Pin]" string => "release n=jessie" ;
"pin[wheezy][Pin]" string => "release n=wheezy" ;
"pin[$(ids)][Package]" string => "*" ;
files:
"/tmp/prefs"
create => "yes",
edit_defaults => empty,
perms => owner("bronto"),
edit_line => apt_preferences("test.pin") ;
}
The test bundle above will create a file like this:
Package: * Pin: origin debian.lth.se Pin-Priority: 700 Package: * Pin: origin packages.linuxmint.com Pin-Priority: 700 Package: * Pin: release n=jessie Pin-Priority: -1 Package: * Pin: release n=wheezy Pin-Priority: 1
Enjoy!
A sysadmin's logbook 