I am publishing four Terraform modules today. This code has lived into a private repository of mine for two years and I decided that it was time to dig it out, put it in dedicated, public repositories and release it. Two of them were my first experiments in making Terraform modules and probably too simple for general use. The other two are related to CFEngine: one helps making CFEngine test clients, the other making CFEngine servers.
As always, this code is released with a GPL license in the hope that it will be useful to more people than just myself.
Commands like the AWS CLI may return a list of values all in one line, where each item in the list is separated by the nearby items with spaces. Using a plain
read command doesn’t really work:
read will read all the values in one go into the variable. You need to change the delimiter that
read uses to split the input. No need to pipe the output through Perl or other tools,
read got you covered with the
In this example I get the list of the ARNs of all target groups in an AWS account, and then iterate over those ARNs to list all the instances in each target group. The ouput will also be saved into a file through the
aws elbv2 describe-target-groups \
--query 'TargetGroups.TargetGroupArn' \
--output text | \
while read -d ' ' ARN ; do \
echo -n "$ARN: " ; \
aws elbv2 describe-target-health \
--target-group-arn "$ARN" \
--query 'TargetHealthDescriptions.Target.Id' \
--output text ; sleep 1 ; \
done | \
The ouput of this one liner will be in the format:
ARN: instance_ID [instance_ID...]
Things to notice:
- the AWS CLI’s
describe-target-groups command will list all target groups’ ARNs thanks to the
--query option and list as many as possible on single lines, according to the shell’s output buffer capacity; the ouput is piped through a
- the while loop uses
read -d ' ' to split each line at spaces and save each item in the
$ARN variable, one per cycle;
echo command prints the value of
$ARN followed by a colon, a space, but will not output a newline sequence due to the
- the AWS CLI’s
describe-target-health command will list all target IDs thanks to the
--query option and print them out in a single line; it will also provide a newline sequence, so that the next loop will start on a new line;
sleep 1 command slows down the loop, so that we don’t hammer the API to the point that they will rate limit us;
- finally, the
tee command will duplicate the output of the while loop to both the standard output and the file
Just a small bash snippet for those cases where, for example, a command returns AWS instance IDs but not the matching DNS names or an IP addresses. The function id2dns, that you can add to your .bashrc file, will do the translation for you. In order to use the function you will:
- ensure you have the aws CLI installed and functional;
ensure you have jq command available;
- ensure you have valid AWS credentials set, so that your aws CLI will work.
Update 2020-08-14: jq not needed any more
This is mostly a note to self. When I need an EC2 instance to run a quick test, it may be overly annoying to provision one through the web console, or it may feel a bit overkill to do that using large frameworks like terraform. Using the AWS command line is just fine, if you know what command to run with which parameters, and it pays off quickly if, to run your tests, you use the settings often (AMI, subnet, security groups…) or if during the same test session you need to scrap and rebuild test instances a few times. Here is an example on how to do so with the AWS command line client.
Say you have access to two separate AWS accounts, and say you have EC2 instances running in a certain region and availability zone, e.g eu-west-1a, in both accounts. Today I learned to my greatest surprise that, despite the same name, they may actually be two totally different locations. Intrigued? Read on!
I have published a small update to cf-keycrypt, so that it’s now easier to compile the tool on Debian systems and it’s compatible with CFEngine 3.15. You can find it here.
For those who don’t know the tool, I’ll try to explain what it is in a few words. The communication between CFEngine agents on clients and the CFEngine server process on a policy hub is encrypted. The key pairs used to encrypt/decrypt the communication are created on each node, usually at installation time or manually with a specific command. cf-keycrypt is a tool that takes advantage of those keys to encrypt and decrypt files, so that they are readable only on the nodes that are supposed to use them. The fact that the keys are created on the nodes themselves eliminates the need to distribute the keys securely.
cf-keycrypt was created years ago by Jon Henrik Bjørnstad, one of the founders of CFEngine (the company). The code has finally landed the CFEngine core sources as cf-secret, but it’s not part of the current stable releases. I had an hard time trying to compile it, but I made it with good help from the CFEngine help mailing list. I decided to give the help back to the community, publishing my updates and opening a pull request to the original code. Until it’s merged, if it ever will, you can find my fork on my github.
After five years after the release of cf-deploy v3, I have just released cf-deploy v4. This version of cf-deploy fixes a number of shortcomings that made their way up to this point and that I wasn’t able to see until recently. It is now more flexible and easier to configure than it ever was. In particular, the documentation is way more comprehensive, covering installation, configuration and usage. The documentation also covers some of the internals, that will allow the hardcore user to fine tune the tool to better suit their needs.
You will find cf-deploy on github, as always. Enjoy!
This article is about using configuration management to install software on your own computers (e.g. your laptops, or the computers used by your family and relatives) and how the complexity of this task is easy to overlook, no matter if you are a newbie or an expert.
If you already know about configuration management and how it makes sense to use it at a small scale like, again, your own computers or your family’s, you can just skip at the section “New job, new setup”.
If you already know about configuration management and you are asking yourself why it should make sense to use it at a small scale, I suggest that you start a section earlier, at Personal configuration management”.
If you are new to configuration management, or you wonder what could be difficult in installing software on a set of systems, I suggest that you read the whole article.
In any case, happy reading!
I have attended the Config Management Camp in Gent this year, where I also presented the talk “Promise theory: from configuration management to team leadership“. A thrilling experience, considering that I was talking about promise theory at the same conference and in the same track where Mark Burgess, the inventor of promise theory, was holding one of the keynotes!
The quality of the conference was as good as always, but my experience at the conference was completely different from the past. Last time I attended, in 2016, I was actively using CFEngine and that shaped in both the talks I attended and the people that I hanged on with the most. This year I was coming from a different work environment and a different job: I jumped a lot through the different tracks and devrooms, and talked with many people with a very different experience than mine. And that was truly enriching. I’ll focus on one experience in particular, that led me to see what the future of configuration management could be.
On March 10th I was in Bologna for Incontro DevOps Italia 2017, the Italian DevOps meeting organized by the great people at BioDec. The three tracks featured several talks in both Italian and English, and first-class international speakers. And, being a conference in Bologna, it also featured first-class local food that no other conference around the world will ever be able to match.