The things I wish I knew before I started using Golang with JSON

A sign sold on EbayThis is not an article about how you can work with JSON in Go: you can easily learn that from the articles and web pages in the bibliography. Rather, this post is about the concepts that you must understand clearly before you set yourself for the task. Don’t sweat, it’s just two concepts two, and I’ve tried to explain them here.

In the last few weeks I have worked together with a colleague to write some automation with Golang and the Atlassian Crowd API. With several separate user databases (and, at the current state, no hope to unify them in a smart way) it would be very handy to take advantage of the APIs offered by, say, G Suite to fetch all the email addresses related to a user and use that information to automatically deactivate that user from all systems.

Coming from a Perl 5 background, I was hoping that decoding and encoding JSON in Go was as simple as it is in Perl.  But it turns out that it wasn’t, and it’s obvious if you think about it: as Perl 5 is weakly typed, decoding any typed data into an “agnostic” data structure must be simple. Encoding a weakly typed data structure into a typed format may be a bit trickier, but as long as you don’t have too many fancy data (i.e., in this context: strings made of only digits or non-obvious boolean representations) this will also work well. But with strongly typed Go and struct field names having side effects depending on upper-/lowercase, that’s a different story.

As it often happens in cases like this, you will not find all the information you need in a single place. This is my attempt to collect it all and hand it to you, so that you won’t have to waste as much time as I did. You will still have to read through stuff though.

Continue reading

Advertisements

Perl to go

I have been using Perl for more than 20 years now, seen Perl 4 bow out and Perl 5 come in and develop in that fantastic language that has helped me uncountable times in my professional life. During those years I’ve also considered learning another language, but I have been unable to take a stand for a long time.

And there came Go and the hype around Go, just like years ago there was a lot of hype around Java. But while whatever written in Java I came across was a big, heavy and slow memory eater, most of the tools I came across that were written in Go were actually good stuff — OK, still a bit bloated in size, but they actually worked. The opportunity came, and I finally gave Go a shot.

Continue reading

Rudimentary compliance report for CFEngine

In CFEngine community you don’t have a web GUI with compliance report. You can get them via EvolveThinking’s Delta Reporting, but if you can’t for any reason, you need to find another way.

A poor man’s compliance report at the bundle level can be extracted via the verbose output. This is how I’ve used it to ensure that a clean-up change in the policies didn’t alter the overall behavior:

cf-agent -Kv 2>&1 | perl -lne 'm{verbose: (/.+): Aggregate compliance .+ = (\d+\.\d%)} && print "$1 ($2)"'

These are the first ten lines of output on my workstation:

bronto@brabham:~$ sudo cf-agent -Kv 2>&1 | perl -lne 'm{verbose: (/.+): Aggregate compliance .+ = (\d+\.\d%)} && print "$1 ($2)"' | head -n 10
/default/banner (100.0%)
/default/inventory_control (100.0%)
/default/inventory_autorun/methods/'proc'/default/cfe_autorun_inventory_proc (100.0%)
/default/inventory_autorun/methods/'fstab'/default/cfe_autorun_inventory_fstab (100.0%)
/default/inventory_autorun/methods/'mtab'/default/cfe_autorun_inventory_mtab (100.0%)
/default/inventory_autorun/methods/'dmidecode'/default/cfe_autorun_inventory_dmidecode (100.0%)
/default/inventory_autorun (100.0%)
/default/inventory_linux (100.0%)
/default/inventory_lsb (100.0%)
/default/services_autorun (100.0%)

Not much, but better than nothing and a starting point anyway. There is much more information in the verbose log that you can extract with something slightly more elaborated than this one-liner. Happy data mining, enjoy!

How we shaved the poodle

CFEngineAgentIn this post I’ll describe how we used CFEngine to apply fixes to apache and nginx to defuse the infamous poodle bug. The post is a bit rushed, in the hope it may still be useful to someone. The policies use bundles and bodies from either the standard library or from our own. The libraries are not shown here but the names speak for themselves… hopefully 🙂

As you’ll probably know, the “trick” on the server side is not to allow secure (erm…) connections to use anything older than TLSv1. In order to do that, we decided to

  • deploy a conf.d snippet to set the appropriate protocol versions as a default;
  • disable the same directive in existing configuration files to avoid weaker directives take priority;
  • restart the server if/when the configuration gets fixed.

Continue reading

New home for my code, new release

Perl (onion)During the past years I’ve published a few Perl modules of mine to CPAN. Nothing big, nothing special, just some small, simple modules that I published in the hope that they would be useful to more people than just me. That code lived, or rather slept, in my hard disk and was not shared anywhere than in CPAN.

At the end of May, a bug was opened against the Net::LDAP::Express module and I decided it was time to bring that code to year 2014. Now, and since a few days ago, you can find the code of all my modules in github. With the code shared on github I was able to share a fix, have it tested by the person who submitted the bug, and confirm the bug was solved. Since one hour ago, the bugfix release 0.12 of Net::LDAP::Express is available on CPAN (on metaCPAN only for now, will hit all the archives in the next few hours).

You are welcome to clone the code from github, fork, branch, open pull requests… Just share the code, make it better, help people, and don’t forget to have fun in the process!

The classification problem: challenges and solutions

Update March 1st, 2015: the latest version of the code for hENC is now on github

It’s been about a month since I came back from FOSDEM and cfgmgmtcamp, a month where I gradually recovered from the the backlog both in the office and at home. It’s been a wonderful experience, especially at cfgmgmtcamp, and I really want to thank all those that helped make it special — more details at the end of this article.

But promise is debt (no pun intended with promise theory here), and I promised to write a long blog post with some (or all) the details from my talks. It’s time to keep that promise. So, without any further ado…

Continue reading

External node classification, the CFEngine way

CFEngineAgentExternal node classification is a Puppet functionality, where it is left to a program external to Puppet (the external node classifier, ENC) to decide which contexts apply to the node being configured. This approach is opposed to the “standard”, basic one, where the configuration applied to a node is completely defined in Puppet files (manifests), and site.pp in particular. ENC programs really show their power and usefulness where the same ruleset is used to manage a large number of servers, with many different combinations of configurations are applied. In this case, pulling the configuration information from a structured data base instead of plain files scales much better. One can write his own ENC, or use one of the several available, hiera being a well-known one.

Note: files and ENC are just two possible ways to classify nodes in Puppet; besides, classification happens on the puppetmaster, while in CFEngine all configuration decisions are taken on the node running the agent. You may read more information on node classification in puppet here, but then we’ll leave you alone and keep reading this post 🙂

Now transpose to CFEngine. Being it generally more “low-level” than Puppet, it provides no ENC mechanism out of the box, but plenty of possibilities to implement one yourself. I first checked what were the available options. I got very nice suggestions, notably one by LinkedIn’s Mike Svoboda, where they use a Yahoo! open source product called Range to store the data about the nodes, then they dump the data in JSON format, and finally they use a bash script run as a CFEngine module to raise the relevant classes. As scalable and sophisticated as it is, it was way too much than what I needed.

ENC, in my case, had two purposes: the first: allow us to scale better (and that’s a common trait in all ENC mechanisms) the second: take as much configuration information as possible out of the policies and in plain text files. This way, the access barrier for the non-CFEngine savvy people in the company would be lowered significantly. This approach is actually closer to Neil Watson’s and EvolveThinking’s CFEngine library (based on CSV files) than to the otherwise wonderful LinkedIn approach.

I sowed this information and ideas in my brain and let them sprout in the background for a couple of weeks, letting the most complex solutions drop. The last one standing was, in my opinion, the best combination of power and simplicity. We’d use plain text files, the CFEngine’s own module protocol, and extra-simple scripts: a bash script for simple external node classification, and a Perl script for hierarchical node classification. I’ll summarize the module protocol first, and then show how we leverage it to achieve ENC.

Continue reading

Who’s using my Perl modules?

blog-tpf_logo_150x79It was ten years ago, when the version 0.01 of my module Net::LDAP::Express hit CPAN. And it was early December 2004 when the current version of it, 0.11, landed in the same place. It was my first public module, it was a simple one, it was hit by some criticism in the Perl community and in the perl-ldap community in particular. In short, I always thought that it was just a little, simple module that a very few people would use, and that would never be included in any mainstream product.

Today, for no special reason, I decided to run a search on Google, and I found out I was wrong. Delightfully wrong.

It was strange to see how wrong I was, but nice!

What about you? Are you using any of my Perl modules in your software? If so, it would be so nice if you’d let me know! Continue reading

…and promise_summary.log met Solarized

Inspired by the blog post at nanard.org, I’ve spent a few days hacking on promise_summary.log and rrdtool, and I have finally something to show.

Let’s recap quickly: a file promise_summary.log in cfengine’s workdir (usually /var/cfengine) contains a summary of every agent run: when the run started and finished, which version of the policy has run, and how many promises (in percentage) were kept, repaired, and not repaired. The first thing I wanted were graphs for these metrics; a medium-term goal is to bring these metrics into some well known tool like Munin or Cacti — that will come later.

I chose to use RRDtool for many reasons; among all:

  • it takes care for both saving the data and making the graphs;
  • it saves the data at different resolutions, automatically;
  • all aspects of a graph are customizable, and different type of graphs can be embedded in the same picture

I had previous experience with RRDtool, and I knew the downsides of course, mainly: the odd, cryptic syntax. What I had forgotten since such a long time was that it’s actually easier than it looks 🙂 … Continue reading