In the last few months I have been working, together with a colleague, on an API client for several well-known systems and cloud providers. When we started, I was a novice in the Go programming language, I had no experience in programming API clients, and I trusted the makers of the APIs enough to have great expectations at them.
Today, a few months later, several hours programming later and a bunch of lines of code later, I am a better novice Go programmer, I have some experience in interfacing with APIs, and my level of trust in the API makers is well beneath my feet.
This article will be a not-so-short summary of the reasons why we started this journey and all the unexpected bad surprises we got along the way. Unfortunately, I will be able to show only snippets of the code we have written because I didn’t get the authorisation to make it public. I will make the effort to ensure that the snippets are good enough to help you get a better understanding of the topics.
OK, enough preface. It’s time to tell the story.
This is not an article about how you can work with JSON in Go: you can easily learn that from the articles and web pages in the bibliography. Rather, this post is about the concepts that you must understand clearly before you set yourself for the task. Don’t sweat, it’s just two concepts two, and I’ve tried to explain them here.
In the last few weeks I have worked together with a colleague to write some automation with Golang and the Atlassian Crowd API. With several separate user databases (and, at the current state, no hope to unify them in a smart way) it would be very handy to take advantage of the APIs offered by, say, G Suite to fetch all the email addresses related to a user and use that information to automatically deactivate that user from all systems.
Coming from a Perl 5 background, I was hoping that decoding and encoding JSON in Go was as simple as it is in Perl. But it turns out that it wasn’t, and it’s obvious if you think about it: as Perl 5 is weakly typed, decoding any typed data into an “agnostic” data structure must be simple. Encoding a weakly typed data structure into a typed format may be a bit trickier, but as long as you don’t have too many fancy data (i.e., in this context: strings made of only digits or non-obvious boolean representations) this will also work well. But with strongly typed Go and struct field names having side effects depending on upper-/lowercase, that’s a different story.
As it often happens in cases like this, you will not find all the information you need in a single place. This is my attempt to collect it all and hand it to you, so that you won’t have to waste as much time as I did. You will still have to read through stuff though.
On March 10th I was in Bologna for Incontro DevOps Italia 2017, the Italian DevOps meeting organized by the great people at BioDec. The three tracks featured several talks in both Italian and English, and first-class international speakers. And, being a conference in Bologna, it also featured first-class local food that no other conference around the world will ever be able to match.
I am guilty for not having considered encrypting my hard drives for too long, I confess. As soon as I joined Telenor Digital (or, actually, early in the process but a bit too late…) I was commanded to encrypt my data and I couldn’t delay any more. To my utter surprise, the process was surprisingly simple in my Debian jessie! Here is a short checklist for your convenience.
As some of my readers already know, I changed jobs in Novembre: I left Opera Software to join Telenor Digital. We have decided not to run any leap second simulation here, so I am not going to publish anything on the subject this year. You can still refer to the post The leap second aftermath for some suggestions I wrote after the latest leap second we had in June/July 2015.
Errata corrige: it’s actually v3! This is what happens when you don’t publish updates for your software for too long…
I took some time this weekend to release an update for cf-deploy. You have now the option to override the configuration hardcoded in the script by means of environment variables. Check the README for the details.
If you don’t know what cf-deploy is, that’s fair 😉 In two words, it’s a Makefile and a Perl front-end to it that makes it easier to pack together a set of files for a configuration management tools and send them to a distribution server. Designed with git and CFEngine in mind, it’s general enough that you can easily adapt it to any version control system and any configuration management tool by simply modifying the Makefile. If it sounds interesting, you are welcome to read Git repository and deployment procedures for CFEngine policies on this same blog. Enjoy!
A new leap second will be introduced at the end of 2016. We have six months to get ready, but this time it may be easier than before as several timekeeping software have implemented some “leap smear” algorithm, which seems to be a very popular approach nowadays; e.g.: ntpd, the reference implementation for NTP, seems to have implemented leap smear from version 4.2.8p3 onward.
We’ll see how it goes. Until then… test!
After some lengthy busy times I’ve been able to restart my work on Docker. Last time I played with some containers to create a Consul cluster using three containers running on the same docker host — something you will never want to do in production.
And the reason why I was playing with a Consul cluster on docker was that you need a key/value store to play with overlay networks in Docker, and Consul is one of the supported stores. Besides, Consul is another technology I wanted to play with since the first minute I’ve known it.
To run an overlay network you need more than one Docker host otherwise it’s pretty pointless. That suggested me that it was time to automate the installation of a Docker host, so that I could put together a test lab quickly and also maintain it. And, as always, CFEngine was my friend. The following policy will not work out of the box for you since it uses a number of libraries of mine, but I’m sure you’ll get the idea.
Here’s another quick post about docker, sorry again if it will come out a bit raw.
In my previous post I talked about my first experiments with docker. There was a number of unanswered questions at first, which got an answer through updates to the blog post during the following days. All but one. When talking about a containerized process that needs to log through syslog to an external server, the post concluded:
if the dockerized process itself needs to communicate with a syslog service “on board”, this may not be enough…
This is a quick post, apologies in advance if it will come out a bit raw.
I’ve been reading about docker for a while and even attended the day of docker in Oslo. I decided it was about time to try something myself to get a better understanding of the technology and if it could be something useful for my use cases.
As always, I despise the “hello world” style examples so I leaned immediately towards something closer to a real case: how hard would it be to make CFEngine’s policy hub a docker service? After all it’s just one process (cf-serverd) with all its data (the files in /var/cfengine/masterfiles) which looks like a perfect fit, at least for a realistic test. I went through the relevant parts of the documentation (see “References” below) and I’d say that it pretty much worked and, where it didn’t, I got an understanding of why and how that should be fixed.
Oh, by the way, a run of
docker search cfengine will tell you that I’m not the only one to have played with this 😉