Perl

External node classification, the CFEngine way

/ bronto / 12 Comments

CFEngineAgentExternal node classification is a Puppet functionality, where it is left to a program external to Puppet (the external node classifier, ENC) to decide which contexts apply to the node being configured. This approach is opposed to the “standard”, basic one, where the configuration applied to a node is completely defined in Puppet files (manifests), and site.pp in particular. ENC programs really show their power and usefulness where the same ruleset is used to manage a large number of servers, with many different combinations of configurations are applied. In this case, pulling the configuration information from a structured data base instead of plain files scales much better. One can write his own ENC, or use one of the several available, hiera being a well-known one.

Note: files and ENC are just two possible ways to classify nodes in Puppet; besides, classification happens on the puppetmaster, while in CFEngine all configuration decisions are taken on the node running the agent. You may read more information on node classification in puppet here, but then we’ll leave you alone and keep reading this post 🙂

Now transpose to CFEngine. Being it generally more “low-level” than Puppet, it provides no ENC mechanism out of the box, but plenty of possibilities to implement one yourself. I first checked what were the available options. I got very nice suggestions, notably one by LinkedIn’s Mike Svoboda, where they use a Yahoo! open source product called Range to store the data about the nodes, then they dump the data in JSON format, and finally they use a bash script run as a CFEngine module to raise the relevant classes. As scalable and sophisticated as it is, it was way too much than what I needed.

ENC, in my case, had two purposes: the first: allow us to scale better (and that’s a common trait in all ENC mechanisms) the second: take as much configuration information as possible out of the policies and in plain text files. This way, the access barrier for the non-CFEngine savvy people in the company would be lowered significantly. This approach is actually closer to Neil Watson’s and EvolveThinking’s CFEngine library (based on CSV files) than to the otherwise wonderful LinkedIn approach.

I sowed this information and ideas in my brain and let them sprout in the background for a couple of weeks, letting the most complex solutions drop. The last one standing was, in my opinion, the best combination of power and simplicity. We’d use plain text files, the CFEngine’s own module protocol, and extra-simple scripts: a bash script for simple external node classification, and a Perl script for hierarchical node classification. I’ll summarize the module protocol first, and then show how we leverage it to achieve ENC.

Continue reading →

Who’s using my Perl modules?

/ bronto / Leave a comment

blog-tpf_logo_150x79It was ten years ago, when the version 0.01 of my module Net::LDAP::Express hit CPAN. And it was early December 2004 when the current version of it, 0.11, landed in the same place. It was my first public module, it was a simple one, it was hit by some criticism in the Perl community and in the perl-ldap community in particular. In short, I always thought that it was just a little, simple module that a very few people would use, and that would never be included in any mainstream product.

Today, for no special reason, I decided to run a search on Google, and I found out I was wrong. Delightfully wrong.

It was strange to see how wrong I was, but nice!

What about you? Are you using any of my Perl modules in your software? If so, it would be so nice if you’d let me know! Continue reading →

…and promise_summary.log met Solarized

/ bronto / Leave a comment

Inspired by the blog post at nanard.org, I’ve spent a few days hacking on promise_summary.log and rrdtool, and I have finally something to show.

Let’s recap quickly: a file promise_summary.log in cfengine’s workdir (usually /var/cfengine) contains a summary of every agent run: when the run started and finished, which version of the policy has run, and how many promises (in percentage) were kept, repaired, and not repaired. The first thing I wanted were graphs for these metrics; a medium-term goal is to bring these metrics into some well known tool like Munin or Cacti — that will come later.

I chose to use RRDtool for many reasons; among all:

  • it takes care for both saving the data and making the graphs;
  • it saves the data at different resolutions, automatically;
  • all aspects of a graph are customizable, and different type of graphs can be embedded in the same picture

I had previous experience with RRDtool, and I knew the downsides of course, mainly: the odd, cryptic syntax. What I had forgotten since such a long time was that it’s actually easier than it looks 🙂 … Continue reading →

Parsing promise_summary.log

/ bronto / 1 Comment

In CFEngine's work directory there is a log file called promise_summary.log. Unsurprisingly, it contains a summary of how agent runs went in the past: how many promises were kept, how many repaired, and how many failed to be repaired. Some weeks ago a blog post on nanard.org showed how this file can be parsed to graph cfengine's activity, and I thought it could be a nice thing to do the same thing in Perl.

For those who never had a peek, the file looks like this:

bronto@murray:/var/cfengine$ tail promise_summary.log 
1367869877,1367869877: Outcome of version Community Failsafe.cf 1.0.0 (agent-0): Promises observed to be kept 95%, Promises repaired 0%, Promises not repaired 5%
1367869877,1367869905: Outcome of version MyOwnPC 1.0.16-1 (agent-0): Promises observed to be kept 97%, Promises repaired 3%, Promises not repaired 0%
1367870164,1367870164: Outcome of version Community Failsafe.cf 1.0.0 (agent-0): Promises observed to be kept 95%, Promises repaired 0%, Promises not repaired 5%
1367870164,1367870191: Outcome of version MyOwnPC 1.0.16-1 (agent-0): Promises observed to be kept 97%, Promises repaired 3%, Promises not repaired 0%
1367870450,1367870450: Outcome of version Community Failsafe.cf 1.0.0 (agent-0): Promises observed to be kept 95%, Promises repaired 0%, Promises not repaired 5%
1367870450,1367870475: Outcome of version MyOwnPC 1.0.16-1 (agent-0): Promises observed to be kept 97%, Promises repaired 3%, Promises not repaired 0%
1367870797,1367870797: Outcome of version Community Failsafe.cf 1.0.0 (agent-0): Promises observed to be kept 95%, Promises repaired 0%, Promises not repaired 5%
1367870797,1367870825: Outcome of version MyOwnPC 1.0.16-1 (agent-0): Promises observed to be kept 97%, Promises repaired 3%, Promises not repaired 0%
1367871083,1367871084: Outcome of version Community Failsafe.cf 1.0.0 (agent-0): Promises observed to be kept 95%, Promises repaired 0%, Promises not repaired 5%
1367871084,1367871111: Outcome of version MyOwnPC 1.0.16-1 (agent-0): Promises observed to be kept 97%, Promises repaired 3%, Promises not repaired 0%

The first thing, and I did it in some ten minutes, is to parse the file to extract the relevant information. Check this one-liner:

perl -F: -alne 'next if m{failsafe.cf}i ; my ($start,$finish) = split(",",$F[0],2) ; my $duration = $finish-$start ; my ($version) = m{version (.+) (} ; my ($kept,$rep,$notrep) = m{Promises observed to be kept (d+)%, Promises repaired (d+)%, Promises not repaired (d+)%} ; print join("t",$start,$duration,$version,$kept,$rep,$notrep)' promise_summary.log

Or, reformatted:

#!/usr/bin/perl -F: -aln

next if m{failsafe.cf}i ;

my ($start,$finish) = split(",",$F[0],2) ;
my $duration = $finish-$start ;
my ($version) = m{version (.+) (} ;
my ($kept,$rep,$notrep) = m{Promises observed to be kept (d+)%, Promises repaired (d+)%, Promises not repaired (d+)%} ;

print join("t",$start,$duration,$version,$kept,$rep,$notrep)

This parses the file and outputs the relevant information tab-separated.

The command line explained

  • -a turns on the "autosplit" feature, where the input is split (normally at spaces) and the chunks are put in the @F array; in this specific case however, the "-F:" switch will make Perl split at colons ":";
  • -l will make it so that newlines are added after each print;
  • -n will wrap the code around a while loop, reading from the file(s) given as argument, and put each line in the $_ "default" variable;
  • -e will execute the code given on the command line;

The code explained

  • we skip the lines generated by the failsafe policy
  • we split the first field (timestamps), and calculate the difference (that is: the duration of the run)
  • we match the version string
  • we match the percentage for kept, repaired, and not repaired promises;
  • we print the bunch in tab-separated format

This is just the start, of course. E.g.: rather than tab-separated values one could print the values in a format suitable to rrdtool, and then use rrdgraph to create the graphs… But that's for later, take care for now. Ciao!

Disabling SSL checks in recent versions of WWW::Mechanize

/ bronto / Leave a comment

I've been scratching my head for a while: a script using WWW::Mechanize, that works perfectly on my workstation's Ubuntu 10.04 LTS, didn't want to work on my new laptop's Linux Mint Debian Edition. The scripts fetches content via HTTPS from an internal site of ours, that uses a certificate from our internal CA.

After checking some obvious stuff, I suspected that the reason was the SSL certificate issued by an unknown certification authority. A quick capture with wireshark confirmed that: "TLSv1 Record Layer: Alert (Level: Fatal, Description: Unknown CA)".

I needed to make WWW::Mechanize's less picky, but the man page didn't say anything about the problem, nor did the FAQ. Some more google'ing showed that the problem was one layer down: LWP::UserAgent (a fundamental component in WWW::Mechanize). As the man page puts it:

           "verify_hostname" => $bool
               When TRUE LWP will for secure protocol schemes ensure
               it connects to servers that have a valid certificate
               matching the expected hostname.  If FALSE no checks are
               made and you can't be sure that you communicate with
               the expected peer.  The no checks behaviour was the
               default for libwww-perl-5.837 and earlier releases.

               This option is initialized from the
               PERL_LWP_SSL_VERIFY_HOSTNAME environment variable.  If
               this environment variable isn't set; then
               "verify_hostname" defaults to 1.

So: the default behaviour has changed, and LWP::UserAgent, now defaults to be picky when it comes to SSL verifications. It's a good thing, but in my case it was breaking my toy.

The fix? Quick and simple: switch from:

my $mech = WWW::Mechanize->new( autocheck => 0 ) ;

to

my $mech = WWW::Mechanize->new( autocheck => 0, ssl_opts => { verify_hostname => 0 } ) ;

Enjoy!

Redoing RabbitMQ’s tutorial – part 4

/ bronto / Leave a comment

RabbitMQ's tutorial 4' scope is: subscribe only to a subset of the messages.

The following shows an implementation of that tutorial (sort of) in Perl with Net::RabbitMQ. As previously, the code is just sketched out and definitely not an example of style: it just aims to show how things work. Once I'll get all the tutorials sorted out in Perl, I'll build on these sketches to create something "real".

Enjoy! … Continue reading →

Redoing RabbitMQ’s tutorial – part 1

/ bronto / 1 Comment

I am experimenting with RabbitMQ and Perl for a pet project of mine: gravity.

The Perl module Net::RabbitMQ seems to be usable, finally. I decided to give it a go, and reproduce with Perl the python scripts shown in the official RabbitMQ tutorial.

Tutorial 1 starts, unsurprisingly, with an "Hello world!" example:

let's send a message, receive it and print it on the screen. To do so we need two programs: one that sends a message and one that receives and prints it.

I am not going to redo the whole tutorial, just show the finished code. Please refer to the tutorial and to Net::RabbitMQ's documentation for the details. … Continue reading →