Virtualization

Test dummies on sale!

/ bronto / Leave a comment

DummySaleSomeone in the CFEngine community said that configuration management is a big hammer: you can manage a zillion of systems with ease. Or wreck them, with the same ease. Deploying configuration changes is a risky business. It takes responsibility. And testing, lots of testing.

The biggest shops in town™ have wonderful continuous integration systems, where every commit spawns a number of virtual machines where the changes are tested and won’t make it into the master branch unless they work correctly. In smaller shops most of the testing phase is done by hand, and it’s absolutely key to have the ability to destroy or create VMs in a snap, well configured and with all the necessary software already on board. We are that kind of small shop, we made that kind of system and we did it on the cheap. If you want to buy yourself one, just read on.

Continue reading →

Advertisement

Building a Xen firewall with Firewall Builder

/ bronto / 7 Comments

Original post date: March 30th, 2011
Updated: April 4th, 2011 (missing rule in prolog)

It's a kind of a problem to manage a firewall for a Xen dom0 with firewall builder. Xen itself adds forwarding rules when starting a virtual machines, and these rules are wiped away when fwbuilder scripts install theirs, which is unfortunate.

Summing up everything, the final plan was to install a firewall on dom0, which should a) forward to the VM the packets originating outside and directed to them (and back), and b) protect the dom0 itself.

It took me some time, experiments and advice to get it right, and here's how. … Continue reading →

Restoring Xen’s iptables rules

/ bronto / Leave a comment

Those of you that use Xen may have noticed that, by default, Xen adds some iptables rules when a VM starts, so to ensure that some specific packets are actually forwarded to the virtual machines. If, for any reason, those rules are wiped away, it would be nice to recover them, wouldn't it?

I found out it's quite easy. The following script will just echo the iptables commands so you can safely test it on a running dom0. If it does something that you actually need, just wipe those echo's away!

#!/bin/bash

xm list | perl -alne 'next if not $F[1] > 0 ; print "@F[0,1]"' | while read VM ID 
do
  xm network-list $ID | perl -alne 'next if not $F[0] =~ m{^d+$} ; print $F[0]' | while read IFID
  do
    VIF="vif$ID.$IFID"
    echo iptables -A FORWARD -m physdev --physdev-in $VIF -s $VM -j ACCEPT
    echo iptables -A FORWARD -m physdev --physdev-in $VIF -p udp --sport bootpc --dport bootps -j ACCEPT
  done
done

I am using Perl here because I know it better than awk, but I am sure that awk can accomplish the same task as well as perl does.

Debugging NTP again (part 4 and last)

/ bronto / Leave a comment

I finally got the Xen servers' clock to stabilize! The clocks kept converging overnight, and are now sporting a very little offset -so little that the graph for today have visible hiccups.

The problem appears to be caused by a kernel bug. If it was fixed in more recent kernels and/or patches, I just don't know. Anyway, the kernel bug is worked around if you set disable kernel in your ntp.conf. Besides, you should set xen.independent_wallclock to zero, and set your clocksource to xen.

Sure, it would not have been possible with the help of other people, so let's tell the full story and provide acknowledgements for those who deserve it. … Continue reading →

Debugging NTP again (part 2)

/ bronto / Leave a comment

If you wonder what I tried yesterday, well here you are. I tried to make one of the two crazy servers a unicast client. I hoped that having the chance to decide when to poll the stratum 1's would help them to adjust better. Although a bit "irregular", it seemed to work better for a while. Unfortunately, when I came back to the office today and looked at the graphs, it eventually started to oscillate more and more during the night, until it had a reset today. The other one is still going bad, of course.

One more day has passed, and still looking for a solution… … Continue reading →

Debugging NTP again (part 1)

/ bronto / Leave a comment

Xen servers are well known for having time synchronization problems. I have a few here, too. Two datacenters, with three multicast NTP servers and two Xen servers in each. And in each datacenter, one of the Xen servers is working like a charm, while the other's offset keeps oscillating more and more, until ntpd resets, and the cycle starts again. Using known workarounds just made things worse.

Having a Xen servers's clock suddenly going backwards from time to time isn't any good. This really calls for a deep debug. … Continue reading →

VirtualBox: properly renaming virtual applications (updated)

/ bronto / Leave a comment

This is one thing that I am doing quite often: create a "gold image" system, export it as a Virtual Application, and then create a number of clones from that. Unfortunately, VirtualBox doesn't change the MAC address of the network interfaces, so if you clone N systems, they will end all with the same MAC… Ouch!!! And what about host names?

So, let's see how to properly rename the system (both hostname and MAC). Note that there may be other stuff you need to change (like, for example, server SSH keys). We'll make a concrete example here:

  1. the "golden image" isn't a running system (we can safely start a clone of it without screwing the network or anything else)
  2. the "golden image" is a Debian system (Lenny, in my specific case)
  3. the "golden image" hostname is golden, the name of the clone will be newname

First thing: start the system and log in as root. Then:

vi /etc/hosts /etc/hostname

In vi, do:

:%s/golden/newname/g
:w
:n
:s
:wq

These instructions will:

  • change all occurences of golden to newname
  • write out the file
  • switch to the next file
  • repeat the substitution
  • save and exit

Then move away some annoying udev rules:

mv /etc/udev/rules.d/70-persistent-net-rules{,.save}

Now shut down the system:

shutdown -h now

Once the system is down, run the VirtualBox gui and select the settings for the new system; get to the advanced network settings and push the "recycle" button to generate a new MAC. Do this for each interface the machine has. Then save the settings and boot again.

…and you are done 😉 Have fun with your new clone!

Update: if your golden image can't be switched off, you can use the following procedure.

Before booting the new machine, set both interfaces as they had the cable unplugged, and generate a new MAC address for each interface straight away. Then boot it, and do all the file changes as described above, including moving away the 70-persistent-net-rules file. Now plug the virtual interfaces by right clicking on the network icon at the bottom of the console, selecting the Network Interface item and ticking the corresponding checkbox.

You are done, and once you reboot you'll see your system come up with its new name and with the network interfaces properly configured.

Proof-of-concept template for Convirt released

/ bronto / Leave a comment

As I wrote in an earlier post, at $WORK we are evaluating Convirt 2.0. While not optimal, we like it in that it works well enough, has a web interface, it's not intrusive, and looks quite configurable. I haven't been digging a lot into it recently, due to higher priority tasks, but something good happened anyway.

To see if convirt allowed us to provision VMs "our way" (that is: Xen Tools + Debootstrap) I tried to create a simple, down-to-earth provisioning template, but I had a lot of issues with custom variables that disappeared from the template in some circumstances.

We had some information exchange on the forums, and even a conversation with a guy at Convirt, and we finally spotted where the problem was. I applied the workaround I was suggested by him, and I finally got it working! So, now this very first version is available from the convirt forums, and the whole forum thread is linked from the HOWTOs page.

Being a so basic, alpha quality template, you may find that there is still a lot to do, and that this stuff just doesn't fit your needs. Well, I agree 🙂 and I encourage you to read the "to-do list" and contribute to make this template better and better.

For now, have the right amount of fun 😉

OpenQRM and Convirt 2.0

/ bronto / Leave a comment

At work we have a fairly big Xen-based Virtual Infrastructure. I was asked to test OpenQRM to see if it could be easily integrated into the existing environment and used as a monitoring and provisioning tool.

OpenQRM is a very nice and powerful tool, and provides means for VM provisioning and life cycle management, it integrates with Nagios and has other interesting features like the Visual Cloud Designer that we'd liked to use.

Unfortunately, a transition of the existing virtual infrastructure to OpenQRM would not be exactly seamless, and we had to give up on this.

On the other end, the beta for Convirt 2.0 was out, with a Web 2.0 interface. My experiments with it were encouraging. Integration of existing Xen servers and VMs was seamless (almost 😉 this time. Things improved with the stable release, but there is still some work to do, and I am trying to contribute.

I am investing some time in trying to do useful things with it, and reporting bugs and impressions in the forums. In a few days I should contribute a (somehow pre-alpha, proof-of-concept, incomplete… yeah, yeah, it's a disclaimer!) template for everybody to test and improve.