Terraform modules release

I am publishing four Terraform modules today. This code has lived into a private repository of mine for two years and I decided that it was time to dig it out, put it in dedicated, public repositories and release it. Two of them were my first experiments in making Terraform modules and probably too simple for general use. The other two are related to CFEngine: one helps making CFEngine test clients, the other making CFEngine servers.

As always, this code is released with a GPL license in the hope that it will be useful to more people than just myself.

Continue reading
Advertisement

Reading one-line lists with the Bash shell

Commands like the AWS CLI may return a list of values all in one line, where each item in the list is separated by the nearby items with spaces. Using a plain read command doesn’t really work: read will read all the values in one go into the variable. You need to change the delimiter that read uses to split the input. No need to pipe the output through Perl or other tools, read got you covered with the -d option.

In this example I get the list of the ARNs of all target groups in an AWS account, and then iterate over those ARNs to list all the instances in each target group. The ouput will also be saved into a file through the tee command:

aws elbv2 describe-target-groups \
  --query 'TargetGroups[].TargetGroupArn' \
  --output text | \
  while read -d ' ' ARN ; do \
    echo -n "$ARN: " ; \
    aws elbv2 describe-target-health \
      --target-group-arn "$ARN" \
      --query 'TargetHealthDescriptions[].Target.Id' \
      --output text ; sleep 1 ; \
  done | \
  tee tg-instances.txt

The ouput of this one liner will be in the format:

ARN: instance_ID [instance_ID...]

Things to notice:

  • the AWS CLI’s describe-target-groups command will list all target groups’ ARNs thanks to the --query option and list as many as possible on single lines, according to the shell’s output buffer capacity; the ouput is piped through a while loop;
  • the while loop uses read -d ' ' to split each line at spaces and save each item in the $ARN variable, one per cycle;
  • the echo command prints the value of $ARN followed by a colon, a space, but will not output a newline sequence due to the -n option;
  • the AWS CLI’s describe-target-health command will list all target IDs thanks to the --query option and print them out in a single line; it will also provide a newline sequence, so that the next loop will start on a new line;
  • the sleep 1 command slows down the loop, so that we don’t hammer the API to the point that they will rate limit us;
  • finally, the tee command will duplicate the output of the while loop to both the standard output and the file tg-instances.txt.

A public inbox S3 bucket

A few weeks ago I found myself in need of a a place where I could share public encryption keys with others for a side project of mine. As the adjective public implies, there is nothing secret about public keys: they can be shared in the open safely, so that was not a concern. The problem was to find a convenient way to do that. More precisely, I needed a place where I could share certain public keys with everyone, and where anyone could put their public keys to share them with me, and with me only.

In the end, I turned to AWS S3 as it is a natural place to look at when it comes to file storage and sharing. But it took a lot of trial and error before I was actually able to find an appropriate configuration for the bucket. I also put some automation with terraform into the mix, both because I prefer to automate things that I may have to do several times, and because it turned out that I’ll have to bring this inbox of mine up and down at need. The outcome is a terraform module that I have just published on github.

Sounds interesting? Read on!

Continue reading

Automating installation/updates of the AWS CLI on Linux

Are you annoyed that there are no native Linux packages for the AWS CLI (deb, rpm…)? And, thus, no repositories? I am, a bit.

But it’s also true that the installation is not difficult at all, right? Well, yes, if you want to install it in locations different than the defaults (e.g. your user’s home directory) and on more than one machine you still have to do some work, but it’s not terrible, is it?

Then, one day, you find that one of the AWS CLI commands you need to use was added in a newer version than the one you are running, so you have to update the AWS CLI on all machines, and possibly rediscover the parameters you used during the initial installation. Are you happy with that?

I am not, and I decided to do something to automate the process: a Makefile, the simplest form of automation you can have on UNIX systems. Here you go: aws-cli-manager on github.

If you find it useful, I am happy. And if you want to support more Linux distributions or more operating systems (MacOS should be fairly easy, I expect), just go ahead and throw me a pull request. Enjoy!

From AWS instance IDs to private DNS names

github-logoJust a small bash snippet for those cases where, for example, a command returns AWS instance IDs but not the matching DNS names or an IP addresses. The function id2dns, that you can add to your .bashrc file, will do the translation for you. In order to use the function you will:

  • ensure you have the aws CLI installed and functional;
  • ensure you have jq command available;
  • ensure you have valid AWS credentials set, so that your aws CLI will work.

Enjoy!

Update 2020-08-14: jq not needed any more

 

Creating and terminating test instances in AWS quickly

This is mostly a note to self. When I need an EC2 instance to run a quick test, it may be overly annoying to provision one through the web console, or it may feel a bit overkill to do that using large frameworks like terraform. Using the AWS command line is just fine, if you know what command to run with which parameters, and it pays off quickly if, to run your tests, you use the settings often (AMI, subnet, security groups…) or if during the same test session  you need to scrap and rebuild test instances a few times. Here is an example on how to do so with the AWS command line client.

Continue reading

How cheap is cheap?

Last month I wrote a new post, namely: Building a simple, resilient, cheap network service with AWS. Now that AWS’ billing cycle for November is complete we can ask ourselves how cheap was that “cheap” in the article, and give an answer. The cost is not exact to the cent, due to the presence of a a stopped instance I am keeping around, but will still be good enough.

To summarise, the article described how to set up a service on spot instances, thus saving on the EC2 cost. We want to understand if we really saved money, and if there are opportunities to save even more. If you haven’t read the previous post, it’s probably a good time to do it now. If you did, let’s go and check the bill.

Continue reading

Building a simple, resilient, cheap network service with AWS

In this post I’ll describe how I put together a number of pieces of information about AWS features to experiment with an idea. It’s nothing advanced, rather: it’s what happens when you are studying on something and you start seeing the possibilities. Don’t expect rocket science then, it’s more like a handful of notes I made in the hope they may be useful to more people than just myself.

Being an experiment where I was supposed to learn how to do things, it’s a manual set-up. Automation will follow, and in my case , it will be Terraform, but not in this post.

Continue reading