Xen Tips and Tricks

• Xen can host Linux, Solaris, and OpenBSD domU's with a modified kernel in each. If the CPU's have built-in hardware virtualization technology (Intel's VT chips, and AMD Pacific), then you may run unmodified OS'es, including Windows.
• When you assign a MAC address to a domU, be sure the second digit is an even number – 00 or AA or some such. This is because 16th bit of the MAC address as a "1" (which makes that an odd number) designates a "multicast" MAC address; your stuff won't work, you won't be able to figure out why, and it will generally be very frustrating. Also, as PIX ARP entries don't expire for 30 minutes, you'll probably save yourself some grief if the MAC mimics the ip address, e.g., IP of 172.31.17.136 = MAC of 00:23:11:71:36 . That way, each domU has a unique MAC, and a re-IP of a system gives it a new MAC, which the PIX will immediately recognize.
• The easiest to install Xen (for me) has been the from-source install. This has the following prerequisites, which may be installed with "yum install" followed by the names below:
  • ncurses-devel
  • ncurses
  • bridge-utils
  • gcc
  • zlib-devel
  • python-devel
  • iproute
  • bison
  • flex
• It is possible to present specific PCI devices directly to domU's, and to mask specific PCI devices from the dom0. Read the manual for details.
• For a source install of Xen, follow the README at the top level of the source. It is short, easy, and works. Don't forget to run the depmod and initrd as instructed at the end of the file.
• /etc/modprobe.conf: The name of the kernel module that drives the LSI SCSI controllers in some servers (notably Sun v20z and v40z) was changed. you need lines something like the following in your /etc/modprobe.conf before you run the mkinitrd; remove the other "mpt" lines in /etc/modprobe.conf and add these:
  {{alias scsi_hostadapter mptbase
  alias scsi_hostadapter1 mptspi}}
• Xen is very slow if it uses the native tls libraries. To make it not use those, rename/move the tls libraries in the dom0 and the domU:
  mv /lib/tls /lib/tls.disabled
  ...and on a 64-bit dom0 also run:
  mv /lib64/tls /lib64/tls.disabled
  The other "gotcha" is that updating the OS may cause new versions of those libraries to be installed. Check often and rename them, or, better yet, make a cron script to notify you if they appear again.
• VMware cannot be installed on a Xen domU or dom0. This is because there is at least one kernel module for VMware that expects to run at CPU ring 0; Xen dom0's and domU's do not run at ring0 – only Xen itself runs at ring0, and the VMware module is not compatible with Xen. It may work with virtualization-enabled CPU's, though.
• i386 (32-bit) and x86_64 (64-bit) domU's can be mixed on an x86_64 dom0, but a x86_64 domU cannot run on an i386 dom0.
• Live migration of domU's is only supported if the domU's filesystems are either:
  • present as the same local device (e.g. /dev/sdc1) on both the source and destination dom0, with for example, drbd or dom0 iSCSI initiators and/or a clustered filesystem.
  • accessible over the network at the same address from both dom0's, as with an iSCSI, GFS, or NFS root.
• Always leave the non-Xen and the last-used Xen kernel on the system and in grub.conf; this way, if there's a problem, your system is still bootable from the hard disk.
• Change the line in your dom0 /etc/sysconfig/kernel to UPDATEDEFAULT=no so that your Xen kernel won't be replaced in the grub.conf each time yum updates the kernel.
• Dom0 security – The dom0 is a critical host, as it offers access to console, filesystems, and other resources of all of the domU's it hosts.
  • selinux – selinux appears to work fine in dom0 without hindering domU's. Go ahead and set it in "active" mode.
  • iptables – You should probably enable iptables on the dom0, and only allow ssh inbound. Add specific ports only if you want to allow, for example, live migration; even those should only accept connections from certain ip addresses.
  • root login over ssh – probably better to create an external access account and disable root logins in /etc/ssh/sshd_config.
    *kernel download/build time – the Xen source install will download a version of the Linux kernel (for Xen 3.0.2-2, it downloads kernel 2.6.16), and compile it completely. The kernel download is about 40-50 MB, so it will take at least 10 minutes probably; building the kernels – Xen and xen-modified Linux kernel – will take another 20-50 minutes. Just know that it will take a little while.
    *python-twisted – The Xen management tools use the Python "twisted" libraries for XML communication. If you use python-twisted-1.3.0-1.2.el4.rf.i386.rpm , which is on rootstore under software\Xen, you can get away without having to install additional dependencies. If you go for a more recent python twisted, you'll find yourself in dependency hell.
• DomU template – It can be handy to create a domU template, stripped of non-essential packages, with your admin scripts and configuration info already entered. You can then make an archive of the template, and each time you need to add a new server, just copy the template to the domU's filesystems.
• In order to use 2 or more virtual network interfaces (with Xen bridge) on your domU, add a script "my-network-script" such as the following to /etc/xen/scripts, and be sure to run chmod +x /etc/xen/my-network-script to make it executable:
  #!/bin/sh
  # start bridges on both eth0 and eth1
  XENDIR="/etc/xen/scripts"
  $XENDIR/network-bridge "$@" netdev=eth0 bridge=xenbr0 vifnum=0
  $XENDIR/network-bridge "$@" netdev=eth1 bridge=xenbr1 vifnum=1
  #$XENDIR/network-bridge "$@" netdev=eth2 bridge=xenbr2 vifnum=2
  ...and change this line in /etc/xen/xend-config.sxp:
  (network-script my-network-script)
• After Xen is installed, here is a summary of commands that may prove useful; each must be run as root:

  Command	Use / Notes
  xm list	see running domains (vm's), their status, memory usage, domain ID, and what ports to telnet to (e.g. 9607) to get a console on each.
  xm console	attach to the console on a domU.
  xm create -c	start a domain and view its console, then type "^]" to disconnect. Note: Be sure to log off root before disconnecting!!!
  xm top	list realtime auto-updating stats on each domain's cpu, memory, and network usage, as well as other info.
  xm create	start a domU. Note that this should actually be a file name (e.g., /etc/xen/www3), not a domU name.
  xm shutdown	shut a domU down.

• It doesn't work well to lvm-snapshot a domU filesystem from a dom0. This is because the lvm snapshot mechanism uses the dm-snapshot module, which upon snapshot creation, sends a "sync" to the filesystem driver to cause the filesystem to flush out its cache. The filesystem of a domU is not accessible by the dom0 kernel, where the snapshot would be created. You can, however, make snapshots within a domU of that domU's filesystems. However, it seems to be easier and just as clean to do a rsync --archive --stats --delete --hard-links --numeric-ids --exclude=/tmp/** --exclude=/proc/** --exclude=/sys/** root@domU_name:/ Local_path/domU_name/ to back up the domU filesystem from the dom0 or another host.
• Certain enabled-by-default services don't make much sense in a Xen domU, and some probably hurt. Here are some services you probably want to disable (or uninstall, if possible) in your domU's:
  • acpid, auditd, autofs, cpuspeed, diskdump, gpm, haldaemon, ipmi, irqbalance, kudzu, mdmonitor, mdmpd, messagebus, microcode_ctl, netfs, netplugd, ntpd, openibd, portmap, readahead, readahead_early, xinetd
• Kernel modules in domU's – some processes will log an error when they can't find the proper kernel module. An example might be httpd trying to load an IPv6 module by default. Most domU's seem to do okay without having the kernel modules available, but if you want to keep the logs cleaner or have an additional capability that requires the kernel module, you may have to rsync /lib/modules/ from dom0 to domU (rsync and preserve hardlinks, to keep the kernel modules tree from growing a lot).
• NTP / Time – domU's use the same hwclock as the dom0, and therefore have the potential to screw things up if they access it directly.
  • Don't run ntpd or ntpdate on domU's. Instead, run it in the dom0 and that will keep all your domU's up to date automatically.
  • In each domU, replace /sbin/hwclock with an executable script that has only the line /bin/true. You'll have to watch this, as some OS updates may replace it with a new binary, and you'll need to make it the script again. Try making it read-only(?).
• On CentOS 5 beta, you have to mkinitrd and include and preload certain xen devices (networking and block device, respectively), else you won't be able to mount your root partition if you're exporting a partition / LV directly from dom0 to domU.
•