…and promise_summary.log met Solarized

Inspired by the blog post at nanard.org, I’ve spent a few days hacking on promise_summary.log and rrdtool, and I have finally something to show.

Let’s recap quickly: a file promise_summary.log in cfengine’s workdir (usually /var/cfengine) contains a summary of every agent run: when the run started and finished, which version of the policy has run, and how many promises (in percentage) were kept, repaired, and not repaired. The first thing I wanted were graphs for these metrics; a medium-term goal is to bring these metrics into some well known tool like Munin or Cacti — that will come later.

I chose to use RRDtool for many reasons; among all:

  • it takes care for both saving the data and making the graphs;
  • it saves the data at different resolutions, automatically;
  • all aspects of a graph are customizable, and different type of graphs can be embedded in the same picture

I had previous experience with RRDtool, and I knew the downsides of course, mainly: the odd, cryptic syntax. What I had forgotten since such a long time was that it’s actually easier than it looks 🙂 … Continue reading

Parsing promise_summary.log

In CFEngine's work directory there is a log file called promise_summary.log. Unsurprisingly, it contains a summary of how agent runs went in the past: how many promises were kept, how many repaired, and how many failed to be repaired. Some weeks ago a blog post on nanard.org showed how this file can be parsed to graph cfengine's activity, and I thought it could be a nice thing to do the same thing in Perl.

For those who never had a peek, the file looks like this:

bronto@murray:/var/cfengine$ tail promise_summary.log 
1367869877,1367869877: Outcome of version Community Failsafe.cf 1.0.0 (agent-0): Promises observed to be kept 95%, Promises repaired 0%, Promises not repaired 5%
1367869877,1367869905: Outcome of version MyOwnPC 1.0.16-1 (agent-0): Promises observed to be kept 97%, Promises repaired 3%, Promises not repaired 0%
1367870164,1367870164: Outcome of version Community Failsafe.cf 1.0.0 (agent-0): Promises observed to be kept 95%, Promises repaired 0%, Promises not repaired 5%
1367870164,1367870191: Outcome of version MyOwnPC 1.0.16-1 (agent-0): Promises observed to be kept 97%, Promises repaired 3%, Promises not repaired 0%
1367870450,1367870450: Outcome of version Community Failsafe.cf 1.0.0 (agent-0): Promises observed to be kept 95%, Promises repaired 0%, Promises not repaired 5%
1367870450,1367870475: Outcome of version MyOwnPC 1.0.16-1 (agent-0): Promises observed to be kept 97%, Promises repaired 3%, Promises not repaired 0%
1367870797,1367870797: Outcome of version Community Failsafe.cf 1.0.0 (agent-0): Promises observed to be kept 95%, Promises repaired 0%, Promises not repaired 5%
1367870797,1367870825: Outcome of version MyOwnPC 1.0.16-1 (agent-0): Promises observed to be kept 97%, Promises repaired 3%, Promises not repaired 0%
1367871083,1367871084: Outcome of version Community Failsafe.cf 1.0.0 (agent-0): Promises observed to be kept 95%, Promises repaired 0%, Promises not repaired 5%
1367871084,1367871111: Outcome of version MyOwnPC 1.0.16-1 (agent-0): Promises observed to be kept 97%, Promises repaired 3%, Promises not repaired 0%

The first thing, and I did it in some ten minutes, is to parse the file to extract the relevant information. Check this one-liner:

perl -F: -alne 'next if m{failsafe.cf}i ; my ($start,$finish) = split(",",$F[0],2) ; my $duration = $finish-$start ; my ($version) = m{version (.+) (} ; my ($kept,$rep,$notrep) = m{Promises observed to be kept (d+)%, Promises repaired (d+)%, Promises not repaired (d+)%} ; print join("t",$start,$duration,$version,$kept,$rep,$notrep)' promise_summary.log

Or, reformatted:

#!/usr/bin/perl -F: -aln

next if m{failsafe.cf}i ;

my ($start,$finish) = split(",",$F[0],2) ;
my $duration = $finish-$start ;
my ($version) = m{version (.+) (} ;
my ($kept,$rep,$notrep) = m{Promises observed to be kept (d+)%, Promises repaired (d+)%, Promises not repaired (d+)%} ;

print join("t",$start,$duration,$version,$kept,$rep,$notrep)

This parses the file and outputs the relevant information tab-separated.

The command line explained

  • -a turns on the "autosplit" feature, where the input is split (normally at spaces) and the chunks are put in the @F array; in this specific case however, the "-F:" switch will make Perl split at colons ":";
  • -l will make it so that newlines are added after each print;
  • -n will wrap the code around a while loop, reading from the file(s) given as argument, and put each line in the $_ "default" variable;
  • -e will execute the code given on the command line;

The code explained

  • we skip the lines generated by the failsafe policy
  • we split the first field (timestamps), and calculate the difference (that is: the duration of the run)
  • we match the version string
  • we match the percentage for kept, repaired, and not repaired promises;
  • we print the bunch in tab-separated format

This is just the start, of course. E.g.: rather than tab-separated values one could print the values in a format suitable to rrdtool, and then use rrdgraph to create the graphs… But that's for later, take care for now. Ciao!

Disabling SSL checks in recent versions of WWW::Mechanize

I've been scratching my head for a while: a script using WWW::Mechanize, that works perfectly on my workstation's Ubuntu 10.04 LTS, didn't want to work on my new laptop's Linux Mint Debian Edition. The scripts fetches content via HTTPS from an internal site of ours, that uses a certificate from our internal CA.

After checking some obvious stuff, I suspected that the reason was the SSL certificate issued by an unknown certification authority. A quick capture with wireshark confirmed that: "TLSv1 Record Layer: Alert (Level: Fatal, Description: Unknown CA)".

I needed to make WWW::Mechanize's less picky, but the man page didn't say anything about the problem, nor did the FAQ. Some more google'ing showed that the problem was one layer down: LWP::UserAgent (a fundamental component in WWW::Mechanize). As the man page puts it:

           "verify_hostname" => $bool
               When TRUE LWP will for secure protocol schemes ensure
               it connects to servers that have a valid certificate
               matching the expected hostname.  If FALSE no checks are
               made and you can't be sure that you communicate with
               the expected peer.  The no checks behaviour was the
               default for libwww-perl-5.837 and earlier releases.

               This option is initialized from the
               PERL_LWP_SSL_VERIFY_HOSTNAME environment variable.  If
               this environment variable isn't set; then
               "verify_hostname" defaults to 1.

So: the default behaviour has changed, and LWP::UserAgent, now defaults to be picky when it comes to SSL verifications. It's a good thing, but in my case it was breaking my toy.

The fix? Quick and simple: switch from:

my $mech = WWW::Mechanize->new( autocheck => 0 ) ;

to

my $mech = WWW::Mechanize->new( autocheck => 0, ssl_opts => { verify_hostname => 0 } ) ;

Enjoy!

Redoing RabbitMQ’s tutorial – part 4

RabbitMQ's tutorial 4' scope is: subscribe only to a subset of the messages.

The following shows an implementation of that tutorial (sort of) in Perl with Net::RabbitMQ. As previously, the code is just sketched out and definitely not an example of style: it just aims to show how things work. Once I'll get all the tutorials sorted out in Perl, I'll build on these sketches to create something "real".

Enjoy! … Continue reading

Redoing RabbitMQ’s tutorial – part 1

I am experimenting with RabbitMQ and Perl for a pet project of mine: gravity.

The Perl module Net::RabbitMQ seems to be usable, finally. I decided to give it a go, and reproduce with Perl the python scripts shown in the official RabbitMQ tutorial.

Tutorial 1 starts, unsurprisingly, with an "Hello world!" example:

let's send a message, receive it and print it on the screen. To do so we need two programs: one that sends a message and one that receives and prints it.

I am not going to redo the whole tutorial, just show the finished code. Please refer to the tutorial and to Net::RabbitMQ's documentation for the details. … Continue reading

An eye on the clock

Recently I was trying to make sane a system clock that, for some reason, suddenly slowed down to a crawl. I started to fiddle with adjtimex, and I needed a way to verify the reaction of the clock itself, and ntpd’s. On one window, I had a watch ntpq -c pe -c as running. On another, there was an ntpdate -q in a loop. In a third one, I wanted to monitor the changes in frequency (set by ntpd) and ticks (set by me). I found this one-liner pretty useful:

# while true ; do adjtimex -p | perl -alne '/frequency/ and $f=sprintf("%3.2f%%",100*$F[1]/32768000) ; /tick/ and $t=$F[1]-10000 ; END { print scalar(localtime),"\tf=$f\tt=$t" }' ; sleep 30 ; done

This snippet assumes that the frequency tolerance of the clock is 32768000, and that the normal value for ticks is 10000. Check the output of adjtimex -p and the man page to verify this fits your system, too.

Location detection in cfengine

My employer provided me with a laptop that I usually use when working remotely (whether on a VPN or not); more rarely, that laptop could also be connected to the company network in Oslo. Usually, when working remotely I am either at home in Oslo, or in one of my relatives' house in Sardinia.

Depending on where I am, the laptop's configuration needs to be changed or augmented, in particular for these services:

  • ntpd
  • DNS resolver
  • software updates

Being an NTP junkie, I want my ntpd configured in the best possible way, and no: I am not going to point to some random servers in the pool that could be on the other side of the world. Whether I am in Italy or in Norway, I want to use servers from the pool that are either in the same country, or in a country nearby.

When I am in VPN, the client decides for me the search list for DNS domains: I want that list to be enriched with all the domains I actually need.

When I am on a low bandwidth network, I want the system to refrain from automatically checking for system updates: it's much better if I do it manually when, for example, I am not going to rot my mother's skype calls.

For all these reasons, I decided to implement a simple location detection process in cfengine that keeps my laptop correctly configured wherever I am. Besides, since the agent runs every 5 minutes, I can be sure that the configuration changes (e.g. when I enter the VPN) will take place quickly.

…and there could be more if you think about it! E.g.: what about having specific firewall rules depending on where you are? If you are in the office, you may not care if someone tries to access your laptop via SSH as that someone could be… you 😉 But suppose you were not able to determine your current location: would you leave free access to your laptop? … Continue reading