I had a problem to solve today. I have a bunch of files in a remote rsyncd repository, which I'll call "PuppetConf", and a number of them that I want to synchronize more often than the others, which I'll call "volatile files" or simply "volatile". Now, the problem is that volatile files, which used to reside in /volatile, now need to be located in other paths as well, and I needed a clever way to synchronize them alltogether without involving complex, long command line expressions.

After some experimenting with export, import, and filters, I finally found a way to do that. This is my filter file:

# This should be run with:
# rsync -zav --delete --filter="merge /path/to/this/file" rsyncdserver::PuppetConf /destination/path
#
# To understand what the patterns in this file means, see the rsync man
# page, in particular the section FILTER RULES.
# If you don't want to go through all that, please read at least the
# following excerpt.
#
#        o      if  the pattern starts with a / then it is anchored to a
#               particular spot in the hierarchy of files, otherwise  it
#               is  matched  against  the  end of the pathname.  This is
#               similar to a leading ^  in  regular  expressions.
# [...]
#        o      a '*' matches any non-empty path component (it stops  at
#               slashes).
# 
#        o      use '**' to match anything, including slashes.
# 
# [...]
#        o      if the pattern contains a / (not counting a trailing  /)
#               or a "**", then it is matched against the full pathname,
#               including  any  leading  directories.
# [...]
#        o      a  trailing "dir_name/***" will match both the directory
#               (as if "dir_name/" had been specified) and everything in
#               the  directory (as if "dir_name/**" had been specified).
# [...]
#        Note that, when using the --recursive  (-r)  option  (which  is
#        implied  by  -a),  every  subcomponent of every path is visited
#        from the top down,  so  include/exclude  patterns  get  applied
#        recursively  to  each subcomponent's full name (e.g. to include
#        "/foo/bar/baz" the subcomponents "/foo" and "/foo/bar" must not
#        be  excluded).  [...] One solution is to ask for
#        all directories in the hierarchy to be included by using a sin-
#        gle  rule: "+ */" (put it somewhere before the "- *" rule)


# Scan every directory, top down
+ */

# Transfer the top-level volatile directory, and everything under it
+ /volatile/***

# Transfer the top-level release file
+ /release

# Find all those paths which end with nodes, and transfer everything below it
+ nodes/***

# Don't copy anything else
- *

So, if I run this command I update just the volatile files. Good enough, but I still have a long command line here.

The next step was to slightly change the rsyncd configuration that Claudia made server side. Having this stanza in rsyncd.conf:

[Volatile]
 path = /puppet
 comment = Volatile files
 read only = true
 transfer logging = true
 filter = merge /etc/rsyncd-filter.conf
 log format = %a %h %o %f %l %b
 log file = /var/log/rsyncd.log

where /etc/rsyncd-filter.conf is the same filter file you see above, shortens the rsync line to this:

rsync -zav rsyncdserver::Volatile /destination/path

and I like it a lot better 🙂

Note that I can't use the –delete option anymore, or I'd wipe everything but the volatile files off. This is not a big deal anyway, since I have to occasionally do a full sync, like:

rsync -zav --delete rsyncdserver::PuppetConf /destination/path

It's so nice to learn new things on the go 🙂