Update: this article refers to the very first version of cf-deploy. For the latest release, check the github repository.
In my latest post “git repository and deployment procedures for CFEngine policies” I explained how we structured our git repository for CFEngine policies, and how we built a deployment procedure, based on GNU make, to easily deploy different projects and branches from the same repository to the policy hubs. Please read that post if you haven’t yet, as this one is not going to make much sense without it.
The make-based deployment procedure worked pretty well and was functional, but still had annoyances. Let’s name a few:
- the
make
command line was a bit long and ugly; usually it was something like:
make -C /var/cfengine/git/common/tools/deploy deploy PROJECT=projX BRANCH=dev-projX-foo SERVER=projX-testhub
- the Makefile was not optimized to deploy on more than one server at a time. To deploy the same files on several hubs, the only solution was to run
make
in a cycle several times, as in
for SERVER in projX-hub{1..10} ; do make -C /var/cfengine/git/common/tools/deploy deploy PROJECT=projX BRANCH=dev-projX-foo SERVER=$SERVER ; done
- deploying a project on all the policy hubs related to that project required one to remember all of the addresses/hostnames; forget one or more of them, and they would simply, hopelessly left behind.
At the same time, there were a few more people that were interested in making tiny changes to the configurations via ENC and deploy, and that long command line was a bit discouraging. All this taken together meant: I needed to add a multi-hub deployment target to the Makefile, and I needed a wrapper for the deployment process to hide that ugly command line.
For first, I added to the Makefile the functionality needed to deploy on more than one hub without having to re-create the temporary directory at every run: it would prepare the files once, deploy them as many times as needed, and then wipe the temporary directory. That was nice and, indeed, needed. But the wrapper couldn’t wait any longer, and I started working on it immediately after. That’s where cf-deploy
was born.
Meet cf-deploy
cf-deploy
is a bash script that reads the settings from the command line and configuration files and runs make
for you. Each project is associated to a configuration file (actually more, but let’s make believe it’s just one for now), for example projX.proj
, that specifies: 1) if the project is remote (must rsync to a remote server to deploy) or local (must rsync to a local filesystem); 2) the name of one or more files containing lists of hubs to deploy to; 3) which branch should be deployed by default; 4) which subdirectory should be deployed together with common/
.
An example of such a file is:
PROJECT_TYPE=remote HUB_LISTS=projectX.hubs BRANCH=master PROJECT=projX
To deploy this project in production one would just run:
cf-deploy deploy projX
However, since deploy
is probably the most common action one will use this script for, the command line above can be shortened to:
cf-deploy projX
The script reads the settings from the project file and runs make
with the appropriate parameters to deploy the master branch (set in BRANCH
) on all the hubs listed in the file projectX.hubs
(set in HUB_LISTS
). Notice that HUB_LISTS
could list more than one file, e.g.:
HUB_LISTS="projectX-location1.hubs projectX-location2.hubs projectX-location3.hubs"
If we want to preview the changes instead, that is: seeing which files would be changed on the policy hub when we deploy, we run
cf-deploy preview projX
Checking the diff between the current files on a hub and the ones we are about to deploy on an hub (e.g.: projX-hub1
) is as easy as running:
cf-deploy diff projX hub projX-hub1
All the command lines above will deploy the branch that was declared as default for that project with the BRANCH
setting. If we want to override that setting and deploy a different branch, we can do that specifying a branch
argument in the command line:
cf-deploy deploy projX branch dev-projX-foo
This approach of having all project details specified in external files comes handy in many situations.
Deploying in a test environment
Suppose, for example, that you want to deploy in a test environment instead of production: in the same way we had a file called projX.proj
we can also have a projX-test.proj
file that is a bit different:
PROJECT_TYPE=remote HUB_LISTS=projectX-test.hubs BRANCH=merge-projX PROJECT=projX
With this, you could use cf-deploy projX-test
to deploy your integration branch on projX’s test environment: it is no more complicated than deploying in production! (It could be trickier by using the makefile directly).
Deploying on pre-production hubs only
In the same way, you can have files listing (for example) all pre-production hubs for a project, and make a projX-preprod.proj
file with a HUB_LISTS setting that will help you deploy on those hubs only when you want to test a special feature on a small set of guinea pig nodes.
Is there more?
cf-deploy
also has a show
action and I’ll leave it to you to read through the code of cf-deploy
and understand what it does. Talking of which…
The script
Update, February 16th, 2015: the current code for cf-deploy is now on github.
#!/bin/bash ### SCRIPT CONFIGURATION ############################################### GITDIR=/var/cfengine/git TOOLDIR=common/tools/deploy ### YOU SHOULD NOT CHANGE ANYTHING BELOW THIS LINE ##################### MAKEDIR="${GITDIR}/${TOOLDIR}" MAKECMD="make -e -C $MAKEDIR" function usage() { echo "" echo "Usage:" echo " $0 PROJECT_NAME" echo " deploys project PROJECT_NAME on all hubs. It's a shortcut for" echo " $0 deploy PROJECT_NAME" echo "" echo " $0 deploy PROJECT_NAME [ branch BRANCH ] [ hub SERVER ]" echo " Deploys PROJECT_NAME with the specified options" echo "" echo " $0 preview PROJECT_NAME [ branch BRANCH ] [ hub SERVER ]" echo " preview the changes that would be applied" echo "" echo " $0 diff PROJECT_NAME [ branch BRANCH ] [ hub SERVER ]" echo " runs a diff for a project (hub is mandatory)" echo "" echo " $0 show PROJECT_NAME" echo " Describes the project PROJECT_NAME, other options are ignored" echo "" echo " $0 show list" echo " Lists all defined projects, other options are ignored" echo "" echo " You can override the default branch of a project by using the" echo " keyword branch. You can also override the project's hub list" echo " by specifying an hub with the hub keyword -- notice that" echo " specifying a hub for a local project is pointless" echo "" exit $1 } function source_project() { PROJECT_NAME=$1 # Source the project file, bail out if we can't find it source "$MAKEDIR/$PROJECT_NAME.proj" if [ $? -ne 0 ] then echo "Cannot load project $PROJECT_NAME, bailing out" exit 32 fi } # No arguments: show usage info and get out if [ $# -eq 0 ] then usage 1 ; fi # One argument, we assume we must deploy this project with default # settings if [ $# -eq 1 ] then ACTION="deploy" PROJECT_NAME=$1 shift else # More than one argument. The first is an action: deploy preview diff # The second argument is the project name # More arguments can follow ACTION=$1 case "$ACTION" in deploy|preview|diff|show) shift ;; *) echo "What do you want to do?" usage 1 esac PROJECT_NAME=$1 shift ; # Parse the remaining arguments, check that they are known and # act accordingly while (( "$#" )) do CMD=$1 shift case "$CMD" in branch) # We override the branch set in the project file with # the one specified after the keyword "branch" BRANCH_OVERRIDE=$1 shift ;; hub) # We override the hub list in the project file with the # policy hub specified after the keyword "hub" SERVER=$1 shift ;; *) # Watcha talkin'bout, Willis? usage 2 ;; esac done fi # Bail out if the project name is null if [ -z "$PROJECT_NAME" ] then echo "Which project do you want to deploy?" usage 4 fi # The action project needs to be handled here so that: # * it doesn't get polluted by BRANCH_OVERRIDE # * it doesn't try to source list.proj when a project list is requested if [ "$ACTION" == "show" ] then if [ "$PROJECT_NAME" == "list" ] then echo "PROJECTS DEFINED IN $MAKEDIR:" ( cd "$MAKEDIR" && ls -1 *.proj ) | sed -e 's/\.proj$//' else source_project "$PROJECT_NAME" echo "Description for project $PROJECT_NAME" echo "Project type: ${PROJECT_TYPE}" echo "Default branch: ${BRANCH}" echo "Git project ID: ${PROJECT}" echo "Hub lists: ${HUB_LISTS}" echo "Defined in: $MAKEDIR/$PROJECT_NAME.proj" fi exit 0 fi source_project "$PROJECT_NAME" # Override the default branch if so asked if [ -n "$BRANCH_OVERRIDE" ] then BRANCH="$BRANCH_OVERRIDE" fi # Define the action to execute depending on the project being a local # or a remote one case "${PROJECT_TYPE}" in remote) DEPLOY_ACTION=deploy PREVIEW_ACTION=preview DIFF_ACTION=diff DEPLOY_MULTI_ACTION=deploy_multi PREVIEW_MULTI_ACTION=preview_multi ;; local) DEPLOY_ACTION=deploy_local PREVIEW_ACTION=preview_local DIFF_ACTION=diff_local DEPLOY_MULTI_ACTION=$DEPLOY_ACTION PREVIEW_MULTI_ACTION=$PREVIEW_ACTION ;; *) echo "project type ${PROJECT_TYPE} not supported" exit 5 ;; esac # Make these variables visible to "make", so that they override the # makefile defaults and the makefile actually does what we want export PROJECT BRANCH SERVER LOCALDIR MASTERDIR HUB_LISTS # If the action requested is deploy, deploy on all the hubs defined in # the project, unless we specified a specific hub on the command line if [ "$ACTION" == "deploy" ] then if [ -n "$SERVER" ] then $MAKECMD $DEPLOY_ACTION else $MAKECMD $DEPLOY_MULTI_ACTION fi # If the action requested is preview, preview on all the hubs defined in # the project, unless we specified a specific hub on the command line elif [ "$ACTION" == "preview" ] then if [ -n "$SERVER" ] then $MAKECMD $PREVIEW_ACTION else $MAKECMD $PREVIEW_MULTI_ACTION fi # See the comment above the "preview" action for the logic of this. elif [ "$ACTION" == "diff" ] then case "${PROJECT_TYPE}" in local) $MAKECMD $DIFF_ACTION 2>&1 ;; remote) if [ -z "$SERVER" ] then echo "diff in remote projects requires a hub" exit 4 fi $MAKECMD $DIFF_ACTION 2>&1 ;; *) echo "What the fuck do you do here?!" exit 64 ;; esac fi
Disclaimer: bash is not my favorite language and the script may well need improvements. If you have suggestions, please speak!
Where do we go from here?
As you can see, cf-deploy
gives you plenty of possibilities to make your life easier and, once it was released, it took me a very little time to become addicted to it! But yet another area was calling for improvement: agent runs. To get the latest version of the policies and run them immediately on a node, one would usually run something like:
cf-agent -KIf update.cf && cf-agent -KI
Quite OK for any CFEngineer, but kind of weird for anyone else. That’s when a new tool was born, and it will be the subject of my next post. Watch out!
Pingback: cfe: agent runs made easier | A sysadmin's logbook
Pingback: My round of conferences in February | A sysadmin's logbook