Introduction
Table of Contents
It is possible to integrate the configuration and installation of Xen guest virtual machines (VM) quite closely with an existing QWG setup. Using pypxeboot and suitable Xen packages (such as those available from CERN Linux support here, it is possible to perform a fully-automated install of Xen guest VMs, with all relevant configuration automatically extracted from the guest's own templates.
The aim of our procedure is to integrate VM configuration as closely as possible with the configuration of "normal" machines. This means:
- Guest VMs are normally installed using the automated PXE/Kickstart installation configured by Quattor's AII software.
- Guest VMs are installed on virtual "hardware" described in its own hardware template. For example, MAC addresses, disk sizes and names, CPU architecture, etc. can all be specified in the VMs hardware template, and will then be used to generate appropriate configuration files for guest VMs.
The current checked-in templates support 64-bit SL 4.5. However, the following combinations of Xen hosts and guests can be made to work:
Host OS | Host Xen | Guest OS | Comments |
slc450-i386 | 3.0.3 | sl308-i386, slc450-i386 | Scientific Linux Cern used here rather than plain SL as profiles already available in QWG. |
sl450-x86_64 | 3.0.3 | sl450-x86_64 | Xen guest disk is now xvda not sda (see here) |
Example templates are available at source:templates/trunk/clusters/example-xen
Templates
The templates include both platform-specific and non-platform specific files.
Non-platform-specific
This is the basic Xen configuration and is found at standard/xen/host
.
Platform-specific
This contains configuration specific to a particular platform (RPM lists and guest AII configuration).
Install server configuration
You will need to install Xen kernels for use during the initial PXE boot of the Xen guest VM. The following table shows where these should be installed and provides links to the relevant files.
/osinstall/nbp/slc308_i386_xen | http://linuxsoft.cern.ch/cern/slc308/i386/images/xen/ |
/osinstall/nbp/sl450-x86_64_xen | http://ftp.scientificlinux.org/linux/scientific/45/x86_64/images/xen/ |
I have encountered some problems with the "new" partitioning scheme used by AII where fdisk is invoked repeatedly to configure the partitions. You may want to use the traditional partitioning scheme where Anaconda does the work itself. This KS template attachment:sl_ks_old_partitioning.conf works for me.
Host configuration
There are two components to the configuration of a Xen host: the configuration of the host's own software and the creation of Xen configuration files for the guests. The template xen/host/config
includes the template xen/configure_guests
which will configure a set of guests in a pre-defined way according to the contents of their hardware templates. This means:
- The disk size defined in the hardware template will be use to create backing storage for the VM filesystem.
- Make sure the OS defined for your VM is correct. Check the Quattor template
os_version_db.tpl
and edit it if necessary to ensure that your host has the correct OS selected. - The assignment of guests to host is most easily done by creating a "database" in the variable XEN_DB. If you do this, it has the advantage that the
hardware/location
field on the guest can be set to the name of the host. See source:templates/trunk/sites/example/site/xen/db.tpl for an example. - Alternatively, you can define the variable XEN_GUESTS on the host to be a list of the guests you want to install on the host, using the names of their templates. (N.B. there is currently a restriction that the host and guest templates must be in the same cluster.) For example:
variable XEN_GUESTS = list("testui.example.org","testwn.example.org");
- Include the template
xen/host/config
: this will set up some basic configuration and also pull in the correct RPMs. - The configuration of the host is controlled by a set of variables with default values. These can be customised if required.
Variable | Default value | Comments |
XEN_BOOT_DEVICE | "/dev/sda2" | Root filesystem of guests (not currently used as we use bootloaders for guests) |
XEN_DOM0_ROOT_DEVICE | /dev/root | Root partition for DOM0. May be used to specify a label-based root device |
XEN_DOM0_MEM | "400M" | Memory assigned to Xen Domain 0 (added to grub entry, must be a string) |
XEN_VG | "vg01" | Base volume group to create guest VM FS on |
XEN_GUESTS | None | A list of FQDNs for guest VMs (e.g. list("xen-guest.example.org")) |
XEN_BOOTLOADER_DEFAULT | "/usr/bin/pypxeboot" | Default bootloader for VMs. Change this to e.g. pygrub if no DHCP server is available |
XEN_BOOTLOADER | nlist mapping VM names to bootloader to override default, e.g. nlist("xen-guest.example.org","/usr/bin/pypxeboot") | |
XEN_BOOTARG_DEFAULT | "" | "vif[0]" argument needed by pypxeboot will be added automatically if using it, set to "" for pygrub |
XEN_BOOTARGS | nlist specifying special bootargs for individual VMs | |
XEN_CREATE_FILESYSTEMS | true | Whether ncm-xen should try and create filesystems defined for VMs |
XEN_CREATE_DOMAINS | true | Whether ncm-xen should try and create (i.e. start up) domains when it runs |
XEN_PROFILE_PREFIX | "" | Only needed if using old "profile_" naming scheme for nodes |
XEN_INDEPENDENT_WALLCLOCK | false | Set to true if you want your guests' clocks be independent of the host |
Guest configuration
- As with the host, make sure the OS for the guest is correctly configured in
os_version_db
. Check the table above to ensure that the guest you're installing is compatible with the host OS. - Create a hardware template for the machine based on
cfg/grid-ireland/hardware/machines/xen
. Note that the disk and RAM sizes defined in this template will be used to generate the Xen configuration for the VM. Also, from SL 4.5 the base disk should bexvda
notsda
. - Set up the mapping between the node name and hardware template in the site databases template (
site/databases.tpl
orpro_site_databases.tpl
. - Include the template
xen/guest/config
in the node's template.
Issues
Hardware issues
- On the later Dell PE1950 machines there is an incompatibility between the network card and the Xen network setup. This will cause you to lose network connectivity when Xen starts up. There is a workaround documented here.
LVM
- For some reason, when the Xen kernel is installed on the host, the initrd that is created does not have the necessary modules for LVM included. If your root filesystem is hosted on LVM, then your machine won't be able to find it at boot time. To ensure that the LVM modules are included in your initrd, do this before booting into Xen for the first time:
mkinitrd --preload dm-mod <initrd file name> <kernel version>
- For example on an sl450-x86_64 machine the command would be as follows:
mv /boot/initrd-2.6.18-1.2835.slc4xen.img /boot/initrd-2.6.18-1.2835.slc4xen.img.orig mkinitrd --preload dm-mod /boot/initrd-2.6.18-1.2835.slc4xen.img 2.6.18-1.2835.slc4xen
- Guest VMs shouldn't use LVM for their filesystems as it just adds needless complexity.
- You may encounter problems with DHCP when guests reboot with the default Xen packages. I have a patched version that works better -- I will try and rebuild for x86_64.
Attachments (1)
- sl_ks_old_partitioning.conf (12.6 KB) - added by 16 years ago.
Download all attachments as: .zip