It is possible to integrate the configuration and installation of Xen guest virtual machines (VM) quite closely with an existing QWG setup. Using pypxeboot and suitable Xen packages (such as those available from CERN Linux support here, it is possible to perform a fully-automated install of Xen guest VMs, with all relevant configuration automatically extracted from the guest's own templates.

The aim of our procedure is to integrate VM configuration as closely as possible with the configuration of "normal" machines. This means:

  • Guest VMs are normally installed using the automated PXE/Kickstart installation configured by Quattor's AII software.
  • Guest VMs are installed on virtual "hardware" described in its own hardware template. For example, MAC addresses, disk sizes and names, CPU architecture, etc. can all be specified in the VMs hardware template, and will then be used to generate appropriate configuration files for guest VMs.

The current checked-in templates support 64-bit SL 4.5. However, the following combinations of Xen hosts and guests can be made to work:

Host OS Host Xen Guest OS Comments
slc450-i386 3.0.3 sl308-i386, slc450-i386 Scientific Linux Cern used here rather than plain SL as profiles already available in QWG.
sl450-x86_64 3.0.3 sl450-x86_64 Xen guest disk is now xvda not sda (see here)

Example templates are available at source:templates/trunk/clusters/example-xen


The templates include both platform-specific and non-platform specific files.


This is the basic Xen configuration and is found at standard/xen/host.


This contains configuration specific to a particular platform (RPM lists and guest AII configuration).

Install server configuration

You will need to install Xen kernels for use during the initial PXE boot of the Xen guest VM. The following table shows where these should be installed and provides links to the relevant files.


I have encountered some problems with the "new" partitioning scheme used by AII where fdisk is invoked repeatedly to configure the partitions. You may want to use the traditional partitioning scheme where Anaconda does the work itself. This KS template attachment:sl_ks_old_partitioning.conf works for me.

Host configuration

There are two components to the configuration of a Xen host: the configuration of the host's own software and the creation of Xen configuration files for the guests. The template xen/host/config includes the template xen/configure_guests which will configure a set of guests in a pre-defined way according to the contents of their hardware templates. This means:

  • The disk size defined in the hardware template will be use to create backing storage for the VM filesystem.
  • Make sure the OS defined for your VM is correct. Check the Quattor template os_version_db.tpl and edit it if necessary to ensure that your host has the correct OS selected.
  • The assignment of guests to host is most easily done by creating a "database" in the variable XEN_DB. If you do this, it has the advantage that the hardware/location field on the guest can be set to the name of the host. See source:templates/trunk/sites/example/site/xen/db.tpl for an example.
  • Alternatively, you can define the variable XEN_GUESTS on the host to be a list of the guests you want to install on the host, using the names of their templates. (N.B. there is currently a restriction that the host and guest templates must be in the same cluster.) For example:
    variable XEN_GUESTS = list("","");
  • Include the template xen/host/config: this will set up some basic configuration and also pull in the correct RPMs.
  • The configuration of the host is controlled by a set of variables with default values. These can be customised if required.
Variable Default value Comments
XEN_BOOT_DEVICE "/dev/sda2" Root filesystem of guests (not currently used as we use bootloaders for guests)
XEN_DOM0_ROOT_DEVICE /dev/root Root partition for DOM0. May be used to specify a label-based root device
XEN_DOM0_MEM "400M" Memory assigned to Xen Domain 0 (added to grub entry, must be a string)
XEN_VG "vg01" Base volume group to create guest VM FS on
XEN_GUESTS None A list of FQDNs for guest VMs (e.g. list(""))
XEN_BOOTLOADER_DEFAULT "/usr/bin/pypxeboot" Default bootloader for VMs. Change this to e.g. pygrub if no DHCP server is available
XEN_BOOTLOADER nlist mapping VM names to bootloader to override default, e.g. nlist("","/usr/bin/pypxeboot")
XEN_BOOTARG_DEFAULT "" "vif[0]" argument needed by pypxeboot will be added automatically if using it, set to "" for pygrub
XEN_BOOTARGS nlist specifying special bootargs for individual VMs
XEN_CREATE_FILESYSTEMS true Whether ncm-xen should try and create filesystems defined for VMs
XEN_CREATE_DOMAINS true Whether ncm-xen should try and create (i.e. start up) domains when it runs
XEN_PROFILE_PREFIX "" Only needed if using old "profile_" naming scheme for nodes
XEN_INDEPENDENT_WALLCLOCK false Set to true if you want your guests' clocks be independent of the host

Guest configuration

  • As with the host, make sure the OS for the guest is correctly configured in os_version_db. Check the table above to ensure that the guest you're installing is compatible with the host OS.
  • Create a hardware template for the machine based on cfg/grid-ireland/hardware/machines/xen. Note that the disk and RAM sizes defined in this template will be used to generate the Xen configuration for the VM. Also, from SL 4.5 the base disk should be xvda not sda.
  • Set up the mapping between the node name and hardware template in the site databases template (site/databases.tpl or pro_site_databases.tpl.
  • Include the template xen/guest/config in the node's template.


Hardware issues

  • On the later Dell PE1950 machines there is an incompatibility between the network card and the Xen network setup. This will cause you to lose network connectivity when Xen starts up. There is a workaround documented here.


  • For some reason, when the Xen kernel is installed on the host, the initrd that is created does not have the necessary modules for LVM included. If your root filesystem is hosted on LVM, then your machine won't be able to find it at boot time. To ensure that the LVM modules are included in your initrd, do this before booting into Xen for the first time:
    mkinitrd --preload dm-mod <initrd file name> <kernel version>
  • For example on an sl450-x86_64 machine the command would be as follows:
    mv /boot/initrd-2.6.18-1.2835.slc4xen.img /boot/initrd-2.6.18-1.2835.slc4xen.img.orig
    mkinitrd --preload dm-mod /boot/initrd-2.6.18-1.2835.slc4xen.img 2.6.18-1.2835.slc4xen
  • Guest VMs shouldn't use LVM for their filesystems as it just adds needless complexity.
  • You may encounter problems with DHCP when guests reboot with the default Xen packages. I have a patched version that works better -- I will try and rebuild for x86_64.
Last modified 8 years ago Last modified on Jan 23, 2009, 5:03:28 PM

Attachments (1)

Download all attachments as: .zip