make

Automating installation/updates of the AWS CLI on Linux

/ bronto / Leave a comment

Are you annoyed that there are no native Linux packages for the AWS CLI (deb, rpm…)? And, thus, no repositories? I am, a bit.

But it’s also true that the installation is not difficult at all, right? Well, yes, if you want to install it in locations different than the defaults (e.g. your user’s home directory) and on more than one machine you still have to do some work, but it’s not terrible, is it?

Then, one day, you find that one of the AWS CLI commands you need to use was added in a newer version than the one you are running, so you have to update the AWS CLI on all machines, and possibly rediscover the parameters you used during the initial installation. Are you happy with that?

I am not, and I decided to do something to automate the process: a Makefile, the simplest form of automation you can have on UNIX systems. Here you go: aws-cli-manager on github.

If you find it useful, I am happy. And if you want to support more Linux distributions or more operating systems (MacOS should be fairly easy, I expect), just go ahead and throw me a pull request. Enjoy!

Advertisement

cf-deploy v4 released

/ bronto / Leave a comment

After five years after the release of cf-deploy v3, I have just released cf-deploy v4. This version of cf-deploy fixes a number of shortcomings that made their way up to this point and that I wasn’t able to see until recently. It is now more flexible and easier to configure than it ever was. In particular, the documentation is way more comprehensive, covering installation, configuration and usage. The documentation also covers some of the internals, that will allow the hardcore user to fine tune the tool to better suit their needs.

You will find cf-deploy on github, as always. Enjoy!

Continue reading →

cf-deploy v2 released

/ bronto / Leave a comment

Update: this article refers to the third version of cf-deploy. For the latest release, check the github repository.

Errata corrige: it’s actually v3! This is what happens when you don’t publish updates for your software for too long…

github-logo I took some time this weekend to release an update for cf-deploy. You have now the option to override the configuration hardcoded in the script by means of environment variables. Check the README for the details.

If you don’t know what cf-deploy is, that’s fair 😉 In two words, it’s a Makefile and a Perl front-end to it that makes it easier to pack together a set of files for a configuration management tools and send them to a distribution server. Designed with git and CFEngine in mind, it’s general enough that you can easily adapt it to any version control system and any configuration management tool by simply modifying the Makefile. If it sounds interesting, you are welcome to read Git repository and deployment procedures for CFEngine policies on this same blog. Enjoy!

 

Report from Config Management Camp 2015

/ bronto / 1 Comment

cfgmgmtcamp-logoThe Config Management Camp 2015 is gone leaving its trail of inspiring presentations, interesting discussions, pleasant meetings with great people and, hopefully, satisfaction for how each of us has played his/her part to make this edition a success.

A big thank for the brave people that attended my seminar and to those who asked questions. The questions gave me a couple of ideas to further expand the seminar, and more may come if you’re so kind to let me know your opinion on the talk: what you liked, what you didn’t, what could be improved in both the talk and the speaker’s style. Thanks in advance. As for the code of the tools, I promise to publish it on GitHub as soon as I get “clearance” (this week, possibly!).

For those who weren’t at the seminar, I presented how we evolved our git repository structure to support more than one project, each one with its own needs, but at the same time being able to share the relevant common libraries and tools and to make the deployment of the policies easy, manageable and maintainable, whatever the number of hubs and projects involved. The questions dove nose down to how we manage access to the hubs so that a person working on project A can’t accidentally deploy his policies on the hubs supporting project B, how we manage access rights to files in separate projects and to branches, and how easy or hard is to extend the deployment tool with new functionality.

The slides of the presentation are on SpeakerDeck (or further down the post if you don’t bother go to SpeakerDeck 😉 The good guys at Normation also filmed the seminar, so it’s just a matter of time that a video of the seminar will be available. Then you’ll also be able to hear my appeal to support cancer research, talking of which you can check another blog post of mine.

Regarding the talks I attended and the “hallway track”, Jez Humble’s keynote was definitely a mind blowing experience. Leaving aside the things that I am doing wrong, that we are doing wrong in my work environment, and that a broad set of people in our profession is f***ing up completely, I understood that there is a category that definitely needs to be more present at events like this: bosses. Because we can do a good job as professionals, follow the best practices, use the brightest and shiniest tools of today and some of the tools of tomorrow, but that’s definitely not enough to establish a culture of cross-area collaboration. That’s not going to happen without the direct involvement of the bosses and their mandate.

Continue reading →

Test dummies on sale!

/ bronto / Leave a comment

DummySaleSomeone in the CFEngine community said that configuration management is a big hammer: you can manage a zillion of systems with ease. Or wreck them, with the same ease. Deploying configuration changes is a risky business. It takes responsibility. And testing, lots of testing.

The biggest shops in town™ have wonderful continuous integration systems, where every commit spawns a number of virtual machines where the changes are tested and won’t make it into the master branch unless they work correctly. In smaller shops most of the testing phase is done by hand, and it’s absolutely key to have the ability to destroy or create VMs in a snap, well configured and with all the necessary software already on board. We are that kind of small shop, we made that kind of system and we did it on the cheap. If you want to buy yourself one, just read on.

Continue reading →

cfe: agent runs made easier

/ bronto / Leave a comment

This is the third and final installment of a three-post series. In the first post, “git repository and deployment procedures for CFEngine policies”, I explained how we have structured our repository to efficiently manage many projects in one branch and share common policies across projects. In the second post, “cf-deploy: easier deployment of CFEngine policies“, I illustrated the script that we use to deploy the policies on our policy hubs and to preview the changes that would be applied to the policies themselves. In this final post I’ll illustrate yet another tool that helped us simplify the way we run the agent by hand on a node. Although one doesn’t need very often to run the agent by hand on production nodes, that is much more needed on test nodes and when debugging policies, and we do that often enough that it made sense to optimize the process. The principle is the same as the other script, cf-deploy: a makefile does the hard work, and we provide a convenient front-end to run make.

Continue reading →

cf-deploy: easier deployment of CFEngine policies

/ bronto / 2 Comments

Update: this article refers to the very first version of cf-deploy. For the latest release, check the github repository.

GitRepoStructureIn my latest post “git repository and deployment procedures for CFEngine policies” I explained how we structured our git repository for CFEngine policies, and how we built a deployment procedure, based on GNU make, to easily deploy different projects and branches from the same repository to the policy hubs. Please read that post if you haven’t yet, as this one is not going to make much sense without it.

The make-based deployment procedure worked pretty well and was functional, but still had annoyances. Let’s name a few:

  • the make command line was a bit long and ugly; usually it was something like:
    make -C /var/cfengine/git/common/tools/deploy deploy PROJECT=projX BRANCH=dev-projX-foo SERVER=projX-testhub
  • the Makefile was not optimized to deploy on more than one server at a time. To deploy the same files on several hubs, the only solution was to run make in a cycle several times, as in
    for SERVER in projX-hub{1..10} ; do make -C /var/cfengine/git/common/tools/deploy deploy PROJECT=projX BRANCH=dev-projX-foo SERVER=$SERVER ; done
  • deploying a project on all the policy hubs related to that project required one to remember all of the addresses/hostnames; forget one or more of them, and they would simply, hopelessly left behind.

At the same time, there were a few more people that were interested in making tiny changes to the configurations via ENC and deploy, and that long command line was a bit discouraging. All this taken together meant: I needed to add a multi-hub deployment target to the Makefile, and I needed a wrapper for the deployment process to hide that ugly command line.

For first, I added to the Makefile the functionality needed to deploy on more than one hub without having to re-create the temporary directory at every run: it would prepare the files once, deploy them as many times as needed, and then wipe the temporary directory. That was nice and, indeed, needed. But the wrapper couldn’t wait any longer, and I started working on it immediately after. That’s where cf-deploy was born.

Continue reading →

Git repository and deployment procedures for CFEngine policies

/ bronto / 4 Comments

This is the first installment of three, where I’ll talk about how we structured the git repository for our CFEngine policies together with the deployment policies. This first episode will be about how the repository was (badly) structured before and how we redid it (better), and it will introduce our deployment procedures based on GNU Make. The second installment will talk about how we built upon the deployment procedure and we made it easier. The third installment will be about how we greatly simplified how we manage agent runs by hand on our nodes, so that even the non-CFEngine-savvy can do the right thing with little to no knowledge of CFEngine.

Continue reading →

…and promise_summary.log met Solarized

/ bronto / Leave a comment

Inspired by the blog post at nanard.org, I’ve spent a few days hacking on promise_summary.log and rrdtool, and I have finally something to show.

Let’s recap quickly: a file promise_summary.log in cfengine’s workdir (usually /var/cfengine) contains a summary of every agent run: when the run started and finished, which version of the policy has run, and how many promises (in percentage) were kept, repaired, and not repaired. The first thing I wanted were graphs for these metrics; a medium-term goal is to bring these metrics into some well known tool like Munin or Cacti — that will come later.

I chose to use RRDtool for many reasons; among all:

  • it takes care for both saving the data and making the graphs;
  • it saves the data at different resolutions, automatically;
  • all aspects of a graph are customizable, and different type of graphs can be embedded in the same picture

I had previous experience with RRDtool, and I knew the downsides of course, mainly: the odd, cryptic syntax. What I had forgotten since such a long time was that it’s actually easier than it looks 🙂 … Continue reading →