To the point & Out of the box

The Zen of Xen

Ever wanted to install XEN on a dedicated root server and use DomUs for public servers?

I’m using virtualization more and more.

I started to use it on my non-server machines. I have one VM for all the MS Office stuff and related tools. One VM with all my development environment and one VM to test all kind of programs. This works pretty well, especially as I have switched to Apple for my development machine. I just took the VMs with me and are ready to start in less then 5 minutes.

Since I had to upgrade my dedicated server system (located somewhere in a datacenter I have never seen) I moved to virtualization as well. For the server side I use XEN, as it seems to be the de-facto standard.

Well, what I want to do is quite simple:

  1. My server has one static public IP address.
  2. Public servers like web-server, mail, and so one should be running in a VM (named DomU in XEN)
  3. The XEN Dom0 (the host) should not run anything beside XEN to keep it as small and fast as possible

I thought this is a pretty simple and standard setup. Hence, I expected this to a be a no-brainer to setup. That’s what I thought...

Things went differently. Very differently...

My first idea was to use a NAT setup. Since what I want to do is pretty like what you do at home when using a router to connect to the internet.

Said, configured, started... no success. Everything worked as expected the only thing that was missing was: I couldn’t access my WWW server from the internet.

Ok, maybe a problem with some IPTABLES setup. I configured it, forward, backward, using Shorewall, etc. No success! In the meantime I became quite an expert with IPTABLES. Nice side-effect.

Than I read about possible MAC address problems, because ISPs filter out traffic from non-expected MAC addresses. Ok, let’s try this. I read about ARP (address resolution protocol), routing, bridging, layer 2 and layer 3 stuff. I tried all kind of XEN setup. While doing all this (up to here it took me already 6-7 days experimenting) I became quite good in deep network know-how. Nice side-effect.

Normally it doesn’t take me that long to fix a IT problem but this one seems to be a hard one.

Slowly I became really nuts. No one could help me, there are some discussions in different mailing lists etc. but nothing solved the problem. But I couldn’t believe that I’m the only person having this problem.

Luckily, I came across this web site here:

Wow, what a good explanation and setup. Very sophisticated but really sounds great. So I worked through everything, downloaded the scripts and tried my luck. Well, I made it quite far but not to the end where everything is working like expected.

I sent an comment to the author Fulup (which is the Breton version of Philippe) Le Foll with some questions and comments. I got an answer very fast and we had a short mail conversation about all this. Fulup than offered to me, how about setting up a call and debug it together on my machine.

I must say, I was very surprised by his offer. So we did. And we got it working. Otherwise you wouldn’t be able to read this Blog post.

Overall, what was the problem?

XEN built-in network stuff is designed to either operate on a local LAN where you have full control or in a setup where you have a bunch of public IP addresses available.

The provided network scripts are buggy! And that was the main problem at all. It looks OK for the non-network guru but it isn’t. The distribution I use is EisXen. It’s a very good distribution, but it does some things in the background you can’t get rid of via the setup GUI.

And how to fix it?

  1. Don’t use the XEN provided network scripts.
  2. Use Fulup’s scripts to setup a bridge that will work as you expect. For example: Routing is only possible if the bridge has an IP address because routing is a layer 3 action. Whereas bridging is a layer 2 action. Hence, you can read the term Brouter for XEN bridges.
  3. Specify just a static internal IP address for your DomU in the VM config file.
  4. Change the network-script entry in xend-config.sxp to use Fulup’s network script.
  5. Use Fulup’s firewall script to setup IPTABLES rules that will work with this setup.
  6. Done.