= Introduction = [[TracNav]] [[TOC(inline)]] It is possible to integrate the configuration and installation of Xen guest virtual machines (VM) quite closely with an existing QWG setup. Using [http://www.cs.tcd.ie/Stephen.Childs/pypxeboot pypxeboot] and suitable Xen packages (such as those available from CERN Linux support [https://twiki.cern.ch/twiki/bin/view/LinuxSupport/XenHowTo here], it is possible to perform a fully-automated install of Xen guest VMs, with all relevant configuration automatically extracted from the guest's own templates. The aim of our procedure is to integrate VM configuration as closely as possible with the configuration of "normal" machines. This means: * Guest VMs are normally installed using the automated PXE/Kickstart installation configured by Quattor's AII software. * Guest VMs are installed on virtual "hardware" described in its own hardware template. For example, MAC addresses, disk sizes and names, CPU architecture, etc. can all be specified in the VMs hardware template, and will then be used to generate appropriate configuration files for guest VMs. The current checked-in templates support 64-bit SL 4.5. However, the following combinations of Xen hosts and guests can be made to work: || '''Host OS''' || '''Host Xen''' || '''Guest OS''' || '''Comments''' || || slc450-i386 || 3.0.3 || sl308-i386, slc450-i386 || Scientific Linux Cern used here rather than plain SL as profiles already available in QWG. || || sl450-x86_64 || 3.0.3 || sl450-x86_64 || Xen guest disk is now xvda not sda (see [https://twiki.cern.ch/twiki/bin/view/LinuxSupport/UpdatingFromSLC44toSLC45 here]) || Example templates are available at source:templates/trunk/clusters/example-xen = Templates = The templates include both platform-specific and non-platform specific files. == Non-platform-specific == This is the basic Xen configuration and is found at {{{standard/xen/host}}}. == Platform-specific == This contains configuration specific to a particular platform (RPM lists and guest AII configuration). = Install server configuration = You will need to install Xen kernels for use during the initial PXE boot of the Xen guest VM. The following table shows where these should be installed and provides links to the relevant files. || /osinstall/nbp/slc308_i386_xen || http://linuxsoft.cern.ch/cern/slc308/i386/images/xen/ || || /osinstall/nbp/sl450-x86_64_xen || http://ftp.scientificlinux.org/linux/scientific/45/x86_64/images/xen/ || I have encountered some problems with the "new" partitioning scheme used by AII where fdisk is invoked repeatedly to configure the partitions. You may want to use the traditional partitioning scheme where Anaconda does the work itself. This KS template attachment:sl_ks_old_partitioning.conf works for me. = Host configuration = There are two components to the configuration of a Xen host: the configuration of the host's own software and the creation of Xen configuration files for the guests. The template {{{xen/host/config}}} includes the template {{{xen/configure_guests}}} which will configure a set of guests in a pre-defined way according to the contents of their hardware templates. This means: * The disk size defined in the hardware template will be use to create backing storage for the VM filesystem. * Make sure the OS defined for your VM is correct. Check the Quattor template {{{os_version_db.tpl}}} and edit it if necessary to ensure that your host has the correct OS selected. * The assignment of guests to host is most easily done by creating a "database" in the variable XEN_DB. If you do this, it has the advantage that the {{{hardware/location}}} field on the guest can be set to the name of the host. See source:templates/trunk/sites/example/site/xen/db.tpl for an example. * Alternatively, you can define the variable XEN_GUESTS on the host to be a list of the guests you want to install on the host, using the names of their templates. (N.B. there is currently a restriction that the host and guest templates must be in the same cluster.) For example: {{{ variable XEN_GUESTS = list("testui.example.org","testwn.example.org"); }}} * Include the template {{{xen/host/config}}}: this will set up some basic configuration and also pull in the correct RPMs. * The configuration of the host is controlled by a set of variables with default values. These can be customised if required. || '''Variable''' || '''Default value''' || '''Comments''' || || XEN_BOOT_DEVICE || "/dev/sda2" || Root filesystem of guests (not currently used as we use bootloaders for guests) || || XEN_DOM0_ROOT_DEVICE || /dev/root || Root partition for DOM0. May be used to specify a label-based root device || || XEN_DOM0_MEM || "400M" || Memory assigned to Xen Domain 0 (added to grub entry, must be a string) || || XEN_VG || "vg01" || Base volume group to create guest VM FS on || || XEN_GUESTS || None || A list of FQDNs for guest VMs (e.g. list("xen-guest.example.org")) || || XEN_BOOTLOADER_DEFAULT || "/usr/bin/pypxeboot" || Default bootloader for VMs. Change this to e.g. pygrub if no DHCP server is available || || XEN_BOOTLOADER || || nlist mapping VM names to bootloader to override default, e.g. nlist("xen-guest.example.org","/usr/bin/pypxeboot") || || XEN_BOOTARG_DEFAULT || "" || "vif[0]" argument needed by pypxeboot will be added automatically if using it, set to "" for pygrub || || XEN_BOOTARGS || || nlist specifying special bootargs for individual VMs || || XEN_CREATE_FILESYSTEMS || true || Whether ncm-xen should try and create filesystems defined for VMs || || XEN_CREATE_DOMAINS || true || Whether ncm-xen should try and create (i.e. start up) domains when it runs || || XEN_PROFILE_PREFIX || "" || Only needed if using old "profile_" naming scheme for nodes|| || XEN_INDEPENDENT_WALLCLOCK || false || Set to true if you want your guests' clocks be independent of the host || = Guest configuration = * As with the host, make sure the OS for the guest is correctly configured in {{{os_version_db}}}. '''Check the table above to ensure that the guest you're installing is compatible with the host OS.''' * Create a hardware template for the machine based on {{{cfg/grid-ireland/hardware/machines/xen}}}. Note that the disk and RAM sizes defined in this template will be used to generate the Xen configuration for the VM. Also, from SL 4.5 the base disk should be {{{xvda}}} not {{{sda}}}. * Set up the mapping between the node name and hardware template in the site databases template ({{{site/databases.tpl}}} or {{{pro_site_databases.tpl}}}. * Include the template {{{xen/guest/config}}} in the node's template. = Issues = == Hardware issues == * On the later Dell PE1950 machines there is an incompatibility between the network card and the Xen network setup. '''This will cause you to lose network connectivity when Xen starts up.''' There is a workaround documented [http://mywiki.ncsa.uiuc.edu/wiki/Dell_PE1950_NIC_Firmware_Workaround here]. == LVM == * For some reason, when the Xen kernel is installed on the host, the initrd that is created does not have the necessary modules for LVM included. If your root filesystem is hosted on LVM, then your machine won't be able to find it at boot time. To ensure that the LVM modules are included in your initrd, do this '''before''' booting into Xen for the first time: {{{ mkinitrd --preload dm-mod }}} * For example on an sl450-x86_64 machine the command would be as follows: {{{ mv /boot/initrd-2.6.18-1.2835.slc4xen.img /boot/initrd-2.6.18-1.2835.slc4xen.img.orig mkinitrd --preload dm-mod /boot/initrd-2.6.18-1.2835.slc4xen.img 2.6.18-1.2835.slc4xen }}} * Guest VMs shouldn't use LVM for their filesystems as it just adds needless complexity. * You may encounter problems with DHCP when guests reboot with the default Xen packages. I have a patched version that works better -- I will try and rebuild for x86_64.