New module on CPAN: Log::Stderr

I published my fourth module on CPAN today: Log::Stderr. As the name says, it will write timestamped log messages to STDERR.

Like the other three I published back in 2004, it's not rocket science. But it is something that I was using to help me debug small scripts, and I found it very convenient. So I am publishing it in the hope that it will be useful for someone else. Of course this module won't be much useful if you have more than basic needs, but it's not a problem: there are so many good modules in the Log:: hierarchy! And if you are in doubt about which one to use, well, visit "the Monastery" and ask the monks 😉

Enjoy!

Quickly graphing RRD data

Still fiddling with RRD, and I'm about at the end of the tunnel 🙂 I finally collected a significant amount of data, and I have scripts to aggregate different sources. What I was missing was a quick way to generate graphs, so that I could visually check if my aggregated data "looks" OK.

As usual, the syntax of rrdtool graph is quite verbose and cryptic, not exactly what you hope when all you need is a quick one-shot graph of some RRD files.

As always, Perl comes to the rescue, this time with RRD::Simple. RRD::Simple is by default able to generate nice graphs with pre-loaded color schemes –if you suck at choosing colors like me, this is a really appreciable feature. It has a set of pre-defined graphs that it can generate, as well, but since it accepts native RRD options (beside its own set), it's actually easy to bend it at your need, and generating a graph just needs a one-liner:

perl -MRRD::Simple -e 'RRD::Simple->graph("aggr_root_storage.rrd", destination => "/tmp", sources => [ qw{perc_used worst_perc_free} ], width => 1024, height => 768, start => 1288566000, end => 1291158000, periods => [ "daily" ] )'  

Or, reindented:

perl -MRRD::Simple -e 
  'RRD::Simple->graph("aggr_root_storage.rrd", 
    destination => "/tmp", 
    sources => [ qw{perc_used worst_perc_free} ], 
    width => 1024, height => 768, 
    start => 1288566000, end => 1291158000, 
    periods => [ "daily" ] )'

The periods options, in this case, has no purpose but to generate only one graph (otherwise you would get many graphs, all equal; why? go and find out yourself, if you really care 😉

And what about plotting a collection of RRDs? It could be something like:

$ for FILE in aggr*.rrd ; do export FILE ; perl -MRRD::Simple -e 'RRD::Simple->graph($ENV{FILE}, destination => "/tmp", width => 1024, height => 768, start => 1288566000, end => 1291158000, periods => [ "daily" ] )' ; done   

or, clearer:

$ for FILE in aggr*.rrd ; 
do 
  export FILE ; 
  perl -MRRD::Simple -e 
    'RRD::Simple->graph($ENV{FILE}, 
      destination => "/tmp", 
      width => 1024, height => 768, 
      start => 1288566000, end => 1291158000, 
      periods => [ "daily" ] )' ; 
done

THINK!

Reading in one of the blogs I keep an eye on (I didn't say I read it thorougly all the time, I just keep an eye on it) I stumbled upon the acronym THINK. It refers to five qualities that what you write should always have, especially when you're communicating with other people.

Each time you are writing, or at least before you rush to the "Send" button in your MTA, you should consider if what you have just written is:

Thoughtful
Honest
Intelligent
Necessary
Kind

I admit I sometimes sin with T and N, and in the worst case with K 😦

A more general approach for AppConfig

In my previous post about AppConfig I suggested an approach in order to read configuration options from a file, and override them with command-line options. It turns out that it works pretty well in the most common case (that is: only scalar variables are involved), but works really bad when list variables come into play.

The problem lies in the command line parsing happening two times, so while scalar variables are overwritten with the same value they had at the first pass, the list variables add the specified value again. Probably, not what you want. The fix is actually trivial:

# Reading configuration
# The first AppConfig object, we'll throw it away. We'll use it
# just to discover if there is a config file that we should read
# first, and then override config file options on the command line
my $config_file ;
{
  my $c        = AppConfig->new($ACparms,@config_vars) ;
  my $args     = [ @ARGV ] ;

  $c->args($args) ;
  $config_file = $c->get("config") ;
}

my $c = AppConfig->new($ACparms,@config_vars) ;

if (defined $config_file) {
  $c->file($config_file) ;
}

$c->args() ;

So basically, we parse the command line the first time inside a bare block, and just to find out if the config option was specified; that option will tell us if we have to parse a configuration file. Once the bare block is over, the AppConfig object ceases to exist, so when we get to the command line again, we are starting from scratch: no duplicated list values any more 🙂

Using AppConfig in 10 minutes

This is a short HOWTO for the AppConfig module. It assumes some knowledge of how to use Perl modules.

What it is
AppConfig is a Perl module that allows you to manage a program’s configuration in a simple way, with both configuration files and command line options.

AppConfig configurations support scalar variables, arrays and hashes, which naturally translate into Perl variables.

Continue reading

Extracting information from iptables/fwbuilder logs

I use Firewall Builder for fast prototyping of my iptables configuration. When a firewall rule matches and logging for that rule is enabled, one line like this is added to /var/log/messages:

Sep  1 09:48:43 server kernel: [9490931.734574] RULE 12 -- DENY IN=eth0 OUT= MAC=a1:b2:c3:d4:e5:f6:00:11:22:33:44:55:66:77 SRC=1.2.3.4 DST=5.6.7.8 LEN=96 TOS=0x00 PREC=0x00 TTL=58 ID=0 DF PROTO=ICMP TYPE=8 CODE=0 ID=27754 SEQ=0 

(sensitive information has been forged, of course 🙂

Depending on the protocol, the same field is not always in the same position, e.g.: destination port (DPT) could be in position 23 or 24. So if you want to list, say, the inward interface, source address, destination address, protocol and destination port you need a smarter matching. This one-liner worked for me:

perl -alne 'if (m{RULE 12}) { my %field ; foreach $token (@F) { next unless $token =~ /=/ ; my ($k,$v) = split(/=/,$token,2) ; $field{$k} = $v } ; print qq{ @field{ qw{IN SRC DST PROTO DPT} } } }' /var/log/messages | sort | uniq -c | sort -nr   

That perl part means: if the line matches "RULE 12" then I initialize the %field hash. Then I go through the tokens and I select those that contain a "=", I split on the equal sign and fill the hash. Finally, when %field is ready, I print the interesting fields.

I don't need to worry about splitting the line and save the "tokens", because perl's autosplit (-a) takes care of it. And I don't need to bother about printing newlines, because -l takes care of it.

And the sort/uniq/sort dance is the old trick to count the occurrences of he same line in the output.

Improving the pseudo-ini parser

In a previous article I described how to create a parser for a pseudo-INI file. The grammar presented in the post is unfortunately unable to parse inline comments correctly.

I asked for ways to improve it, both on perlmonks and on IRC. After some testing it turns out that a slight change is enough: the first line of the grammar should be changed to:

AsIni: <skip: qr{s*(;[^n]*ns*)?}s> Line(s?) /Z/

Unfortunately, this won't print the skipped content (as CommentLine) does. From the answers I had it's not quite clear if this is actually possible with Parse::RecDescent. Anyway, I am satisfied with the result so far 🙂

Perl 6 is out! Erm… sort of

Nota per gli Italiani: date un'occhiata all'annuncio sul sito Perl.it.

OK, it is not the great day that we all Perlers are longing to see, but it's definitely a notable one. The Rakudo and Perl 6 development teams have just announced the release of Rakudo Star, "a useful and usable distribution of Perl 6".

The announcement says:

Rakudo Star is aimed at "early adopters" of Perl 6. We know that it still has some bugs, it is far slower than it ought to be, and there are some advanced pieces of the Perl 6 language specification that aren't implemented yet. But Rakudo Perl 6 in its current form is also proving to be viable (and fun) for developing applications and exploring a great new language.

It's just like mom bought me a new toy to play with 🙂

Creating a parser with Parse::RecDescent

Thanks God, configuration files are everywhere, so that you don't have configuration information hardcoded into your programs anymore. And if you have… well, at least you know it's bad practice, don't you?

Anyway, configuration files need to be parsed to extract the information they hold. If the format is a well-known one, you will probably find a library for your favourite language that will do the work for you. If it's not, but it's line-based and simple enough, then writing a parser may be easy. If it's an XML file, things get more complicated but still doable: just use one of the many parsing library around, look for the relevant XML elements, and you're done. Or, if it is up to you to choose the file format, you may use one that is tailored for your configuration library of choice –mine happens to be the AppConfig Perl module.

But, what to do when you fall out of the easy cases? … Continue reading