bash

Bash scripting: using ‘read’ without a loop

/ bronto / Leave a comment

Another post in the “note to myself” style.

For anyone who does bash scripting, the command read is a well known tool. A usual task that we use read for it is to process the output of another command in a while loop, line by line, picking up a few fields and doing something with them. A stupid example:

sysctl -a 2> /dev/null | grep = | while read PARM DUMMY VALUE
do
  echo "Value for $PARM is $VALUE"
done

That is: we read the output of sysctl, line by line, selecting only the lines that contain a = sign, then read the name of the setting and its value in PARM and VALUE respectively, and do something with those values. So far so good.

Based on what we have just seen, it’s easy to expect that this:

echo foobar 42 | read PARM VALUE
echo "Value for $PARM is $VALUE"

would print “Value for foobar is 42“. But it doesn’t:

Value for  is

So, where did those values go? Did read work at all? In hindsight I can tell you: yes, it worked, but those values have disappeared as soon as read was done with them. To both parse them and use them you have to run both read and the commands using the variables in the same subshell. This works:

echo foobar 42 | ( read PARM VALUE ; echo "Value for $PARM is $VALUE" )

Or even

echo foobar 42 | (
    read PARM VALUE
    echo "Value for $PARM is $VALUE"
)

This will print “Value for foobar is 42”, as expected.

Advertisement

Reading one-line lists with the Bash shell

/ bronto / Leave a comment

Commands like the AWS CLI may return a list of values all in one line, where each item in the list is separated by the nearby items with spaces. Using a plain read command doesn’t really work: read will read all the values in one go into the variable. You need to change the delimiter that read uses to split the input. No need to pipe the output through Perl or other tools, read got you covered with the -d option.

In this example I get the list of the ARNs of all target groups in an AWS account, and then iterate over those ARNs to list all the instances in each target group. The ouput will also be saved into a file through the tee command:

aws elbv2 describe-target-groups \
  --query 'TargetGroups[].TargetGroupArn' \
  --output text | \
  while read -d ' ' ARN ; do \
    echo -n "$ARN: " ; \
    aws elbv2 describe-target-health \
      --target-group-arn "$ARN" \
      --query 'TargetHealthDescriptions[].Target.Id' \
      --output text ; sleep 1 ; \
  done | \
  tee tg-instances.txt

The ouput of this one liner will be in the format:

ARN: instance_ID [instance_ID...]

Things to notice:

  • the AWS CLI’s describe-target-groups command will list all target groups’ ARNs thanks to the --query option and list as many as possible on single lines, according to the shell’s output buffer capacity; the ouput is piped through a while loop;
  • the while loop uses read -d ' ' to split each line at spaces and save each item in the $ARN variable, one per cycle;
  • the echo command prints the value of $ARN followed by a colon, a space, but will not output a newline sequence due to the -n option;
  • the AWS CLI’s describe-target-health command will list all target IDs thanks to the --query option and print them out in a single line; it will also provide a newline sequence, so that the next loop will start on a new line;
  • the sleep 1 command slows down the loop, so that we don’t hammer the API to the point that they will rate limit us;
  • finally, the tee command will duplicate the output of the while loop to both the standard output and the file tg-instances.txt.

From AWS instance IDs to private DNS names

/ bronto / Leave a comment

github-logoJust a small bash snippet for those cases where, for example, a command returns AWS instance IDs but not the matching DNS names or an IP addresses. The function id2dns, that you can add to your .bashrc file, will do the translation for you. In order to use the function you will:

  • ensure you have the aws CLI installed and functional;
  • ensure you have jq command available;
  • ensure you have valid AWS credentials set, so that your aws CLI will work.

Enjoy!

Update 2020-08-14: jq not needed any more

 

cf-deploy: easier deployment of CFEngine policies

/ bronto / 2 Comments

Update: this article refers to the very first version of cf-deploy. For the latest release, check the github repository.

GitRepoStructureIn my latest post “git repository and deployment procedures for CFEngine policies” I explained how we structured our git repository for CFEngine policies, and how we built a deployment procedure, based on GNU make, to easily deploy different projects and branches from the same repository to the policy hubs. Please read that post if you haven’t yet, as this one is not going to make much sense without it.

The make-based deployment procedure worked pretty well and was functional, but still had annoyances. Let’s name a few:

  • the make command line was a bit long and ugly; usually it was something like:
    make -C /var/cfengine/git/common/tools/deploy deploy PROJECT=projX BRANCH=dev-projX-foo SERVER=projX-testhub
  • the Makefile was not optimized to deploy on more than one server at a time. To deploy the same files on several hubs, the only solution was to run make in a cycle several times, as in
    for SERVER in projX-hub{1..10} ; do make -C /var/cfengine/git/common/tools/deploy deploy PROJECT=projX BRANCH=dev-projX-foo SERVER=$SERVER ; done
  • deploying a project on all the policy hubs related to that project required one to remember all of the addresses/hostnames; forget one or more of them, and they would simply, hopelessly left behind.

At the same time, there were a few more people that were interested in making tiny changes to the configurations via ENC and deploy, and that long command line was a bit discouraging. All this taken together meant: I needed to add a multi-hub deployment target to the Makefile, and I needed a wrapper for the deployment process to hide that ugly command line.

For first, I added to the Makefile the functionality needed to deploy on more than one hub without having to re-create the temporary directory at every run: it would prepare the files once, deploy them as many times as needed, and then wipe the temporary directory. That was nice and, indeed, needed. But the wrapper couldn’t wait any longer, and I started working on it immediately after. That’s where cf-deploy was born.

Continue reading →