Creating a Xen virtual machine

So that we can keep our build system totally isolated from anything else we may install on this physical system and to provide enhanced control over how much it is allowed to utilise the CPU and disks we shall be creating a paravirtualised virtual machine using Xen to host our build server. As this section requires a degree of familiarity with the Xen virtualisation system and that the Xen hypervisor be installed and configured you may wish to examine the Virtualisation with the Xen hypervisor guide before continuing with this section.

If you would rather install the build server on a separate physical server, or even combine it with other installed applications, then feel free to skip the Xen related parts of this section and proceed with Creating a 64bit build-space or Creating a 32bit build-space as desired once the relevant storage has been allocated.

Creating volumes and filesystems

The first task is to create a logical volume for our build server virtual-machine and format it with an appropriate filesystem. In the example below we have used ext3 as it is widely tested however feel free to use the filesystem du jour if desired.

As we shall be using the ext2 and ext3 filesystems for all our volumes we need to create a filesystem type for portage using a smaller block size and a more suitable inode ratio so that disk space can be used as efficiently as possible. The example below shows the configuration required to create such a filesystem type.

[fs_types]

portage = {
    blocksize = 1024
    inode_ratio = 2048
}

Now that we have defined a new filesystem type more suited to storing portage snapshots we can create a volume to store the "live" portage snapshot. As the data stored here will be pulled from the portage snapshot mirrors reliability of storage is of less concern than CPU and disk utilisation so we shall use the non-journaled ext2 filesystem. We shall also use the portage type we defined earlier to ensure sufficient inodes are allocated.

The next step is to create a volume to store distribution files downloaded during package building. As before these files will be downloaded from mirrors so reliability of data storage is not at a premium and a non-journaled filesystem can be used. We can also specify the predefined largefile filesystem type to ensure that space is not wasted by thousands of pointless inodes.

With storage allocated for the virtual-machine, "live" portage snapshot and the distribution files we can now allocate more storage for the build-spaces. As you can see we are intending to create a build-space for server and workstation configurations for both 64bit and 32bit systems bringing the total number of build-spaces to four. If you wish to build additional configurations, or have no desire to build for a particular configuration, you will obviously need to modify the examples below accordingly.

To reduce the storage requirements all the build-spaces will share /var/tmp so we will need to create and format a volume for this purpose as shown below. As it is a temporary file store we have used the ext2 filesystem to reduce CPU and disk utilisation.

So that we will be able utilise the packages which we have built in each of the build-spaces while building and testing the next generation of packages to deploy each build-space will also require storage to be allocated for a copy of the portage tree, any overlays which are being used and the configuration files associated with that build. As all of these files are similar in size and nature and will be copied from the "live" repository or the build-space we shall be storing them together in small volumes formatted with the ext2 filesystem using the portage filesystem type we created earlier.

In a similar vein we shall now allocate storage for kernel sources. In the example below we have used the ext3 filesystem for the testing locations and the ext3 filesystem for the stable locations as the kernels will be built in the former and copied to the latter.

Pay particular attention to the type of filesystem used in the above examples. Whilst it does not really matter if you use the ext3 filesystem for all of the above volumes remember that they will need matching entries in fstab later on or they will fail to mount correctly.

 

The final preparatory step is to allocate storage for the packages which will be built in the above build-spaces. As you can see in the example below we have created a testing and a stable store for binary packages for each configuration so that we can keep tested packages available while testing newly built packages before deployment. As before we have used the ext2 filesystem as storage consistency during writes is of little importance as a system crash during package building would render the package useless anyway.

Install a Linux virtual machine

Now that we have allocated sufficient storage for the various build spaces, portage snapshots, kernel images and package repositories we can begin the actual installation process for the build server. To accomplish this we first need to download a stage3 archive from the Gentoo Distribution Site◳ and unpack it into the volume we created above. As shown in the example below we shall also be creating some 64bit build spaces once the new virtual machine is running so we move the archive we downloaded into the newly created root directory of our build server so it is available there.

Configuring a Xen virtual machine

Before we can complete the configuration of the Linux virtual-machine we began creating in the previous section we need to create a Xen configuration file specifying, amongst other things, how the volumes we created in the previous sections should be presented, the network address to be assigned as well as how much memory and how many virtual CPUs should be allocated.

# general
name   = "portage";
memory = 2048;
vcpus  = 3;

# booting
kernel = "/usr/xen/kernels/stock";
root   = "/dev/xvda1 rw";
extra  = "vdso32=0";

# virtual harddisk
disk = [ "phy:volumes/portage-vm,xvda1,w",

         "phy:volumes/portage-vx-distfiles,xvdb1,w",
         "phy:volumes/portage-vx-repo-live,xvdc1,w",

         "phy:volumes/portage-vx-repo-testing-x86-64bit-server,xvdd1,w",
         "phy:volumes/portage-vx-repo-testing-x86-64bit-workstation,xvdd2,w",
         "phy:volumes/portage-vx-repo-testing-x86-32bit-server,xvdd3,w",
         "phy:volumes/portage-vx-repo-testing-x86-32bit-workstation,xvdd4,w",

         "phy:volumes/portage-vx-repo-stable-x86-64bit-server,xvde1,w",
         "phy:volumes/portage-vx-repo-stable-x86-64bit-workstation,xvde2,w",
         "phy:volumes/portage-vx-repo-stable-x86-32bit-server,xvde3,w",
         "phy:volumes/portage-vx-repo-stable-x86-32bit-workstation,xvde4,w",

         "phy:volumes/portage-vx-packages-testing-x86-64bit-server,xvdf1,w",
         "phy:volumes/portage-vx-packages-testing-x86-64bit-workstation,xvdf2,w",
         "phy:volumes/portage-vx-packages-testing-x86-32bit-server,xvdf3,w",
         "phy:volumes/portage-vx-packages-testing-x86-32bit-workstation,xvdf4,w",

         "phy:volumes/portage-vx-kernels-testing-x86-64bit-server,xvdg1,w",
         "phy:volumes/portage-vx-kernels-testing-x86-64bit-workstation,xvdg2,w",
         "phy:volumes/portage-vx-kernels-testing-x86-32bit-server,xvdg3,w",
         "phy:volumes/portage-vx-kernels-testing-x86-32bit-workstation,xvdg4,w",

         "phy:volumes/portage-vx-packages-stable-x86-64bit-server,xvdh1,w",
         "phy:volumes/portage-vx-packages-stable-x86-64bit-workstation,xvdh2,w",
         "phy:volumes/portage-vx-packages-stable-x86-32bit-server,xvdh3,w",
         "phy:volumes/portage-vx-packages-stable-x86-32bit-workstation,xvdh4,w",

         "phy:volumes/portage-vx-kernels-stable-x86-64bit-server,xvdi1,w",
         "phy:volumes/portage-vx-kernels-stable-x86-64bit-workstation,xvdi2,w",
         "phy:volumes/portage-vx-kernels-stable-x86-32bit-server,xvdi3,w",
         "phy:volumes/portage-vx-kernels-stable-x86-32bit-workstation,xvdi4,w",

         "phy:volumes/portage-vx-buildspace-x86-64bit-server,xvdj1,w",
         "phy:volumes/portage-vx-buildspace-x86-64bit-workstation,xvdj2,w",
         "phy:volumes/portage-vx-buildspace-x86-32bit-server,xvdj3,w",
         "phy:volumes/portage-vx-buildspace-x86-32bit-workstation,xvdj4,w",

         "phy:volumes/portage-vx-buildspace-tmp,xvdk1,w" ];

# virtual network
vif = [ "ip=10.0.1.1, vifname=vif.portage" ];

If you intend to run 32bit applications and you have installed Xen on a 64bit platform then you will need to ensure that the vdso32=0 entry above is present or segmentation faults will occur when any 32bit application issues a system call. For more information see Virtual Dynamically-linked Shared Object◳ and What is linux-gate.so.1?◳

 

Configuring the Linux virtual machine

Once we have created our Xen configuration, and decided on the presentation of the virtual devices for the domain, we have the required information to complete the configuration of the operating system on the virtual machine. Before we can apply this configuration however we first need to create some mount points for the volumes we created as shown in the example below.

We also need to create some directories inside each repository volume so that we can mount the packages and kernels volumes there as well as directories for the conf and portage files. In the example below we have created directories for the testing repository for the 64bit server configuration.

Assuming that you are building packages for the four configurations we described earlier then the procedure above will need to be repeated seven more times as shown below. If you are using a different set of configurations you will obviously need to modify the examples presented here accordingly.

Once we have created all the mount-points described above we will need to re-mount the root filesystem of our new virtual machine to complete the configuration, as shown below.

With the mount points created and the virtual device presentation information from when we created the Xen configuration above we can create an fstab file for the virtual machine to automatically mount the volumes during the boot process. As you can see in the example below we have mounted most of the volumes using the noatime option to improve efficiency however the distfiles store needs the atime information as we shall be using it later to implement an LRU (Least Recently Used) deletion algorithm allowing us to share distribution files between buildspaces more efficiently.

/dev/xvda1      /                                                               ext3            noatime         0 1

/dev/xvdb1      /mnt/repositories/distfiles                                     ext2            atime           0 2
/dev/xvdc1      /mnt/repositories/live                                          ext2            noatime         0 2

/dev/xvdd1      /mnt/repositories/testing/x86-64bit-server                      ext2            noatime         0 3
/dev/xvdd2      /mnt/repositories/testing/x86-64bit-workstation                 ext2            noatime         0 3
/dev/xvdd3      /mnt/repositories/testing/x86-32bit-server                      ext2            noatime         0 3
/dev/xvdd4      /mnt/repositories/testing/x86-32bit-workstation                 ext2            noatime         0 3

/dev/xvde1      /mnt/repositories/stable/x86-64bit-server                       ext2            noatime         0 4
/dev/xvde2      /mnt/repositories/stable/x86-64bit-workstation                  ext2            noatime         0 4
/dev/xvde3      /mnt/repositories/stable/x86-32bit-server                       ext2            noatime         0 4
/dev/xvde4      /mnt/repositories/stable/x86-32bit-workstation                  ext2            noatime         0 4

/dev/xvdf1      /mnt/repositories/testing/x86-64bit-server/packages             ext2            noatime         0 5
/dev/xvdf2      /mnt/repositories/testing/x86-64bit-workstation/packages        ext2            noatime         0 5
/dev/xvdf3      /mnt/repositories/testing/x86-32bit-server/packages             ext2            noatime         0 5
/dev/xvdf4      /mnt/repositories/testing/x86-32bit-workstation/packages        ext2            noatime         0 5

/dev/xvdg1      /mnt/repositories/testing/x86-64bit-server/kernels              ext3            noatime         0 6
/dev/xvdg2      /mnt/repositories/testing/x86-64bit-workstation/kernels         ext3            noatime         0 6
/dev/xvdg3      /mnt/repositories/testing/x86-32bit-server/kernels              ext3            noatime         0 6
/dev/xvdg4      /mnt/repositories/testing/x86-32bit-workstation/kernels         ext3            noatime         0 6

/dev/xvdh1      /mnt/repositories/stable/x86-64bit-server/packages              ext2            noatime         0 7
/dev/xvdh2      /mnt/repositories/stable/x86-64bit-workstation/packages         ext2            noatime         0 7
/dev/xvdh3      /mnt/repositories/stable/x86-32bit-server/packages              ext2            noatime         0 7
/dev/xvdh4      /mnt/repositories/stable/x86-32bit-workstation/packages         ext2            noatime         0 7

/dev/xvdi1      /mnt/repositories/stable/x86-64bit-server/kernels               ext2            noatime         0 8
/dev/xvdi2      /mnt/repositories/stable/x86-64bit-workstation/kernels          ext2            noatime         0 8
/dev/xvdi3      /mnt/repositories/stable/x86-32bit-server/kernels               ext2            noatime         0 8
/dev/xvdi4      /mnt/repositories/stable/x86-32bit-workstation/kernels          ext2            noatime         0 8

/dev/xvdj1      /mnt/buildspaces/x86-64bit-server                               ext3            noatime         0 9
/dev/xvdj2      /mnt/buildspaces/x86-64bit-workstation                          ext3            noatime         0 9
/dev/xvdj3      /mnt/buildspaces/x86-32bit-server                               ext3            noatime         0 9
/dev/xvdj4      /mnt/buildspaces/x86-32bit-workstation                          ext3            noatime         0 9

/dev/xvdk1      /var/tmp/portage                                                ext2            noatime         0 10

shm             /dev/shm                                                        tmpfs           nodev,nosuid,noexec

Please pay special attention to the mount options specified in the example above. Some filesystems are ext2 while some are ext3, some are mounted with the noatime option whilst others are mounted with the atime option. File system types will need to match those chosen earlier while the atime options should be left exactly as shown for proper functioning of later sections.

 

Next we need to configure networking settings for the build server to match those we specified when configuring the Xen virtual-machine. As you can see from the examples below there are four configuration files which will need modifying to reflect the network topology and naming convention of your network.

config_eth0=( "10.0.1.2/24" )

routes_eth0=( "10.0.0.0/24"
              "default via 10.0.0.1" )

domain internal.hacking.co.uk
nameserver 10.0.1.3

10.0.1.2        portage.local.hacking.co.uk       portage

# Set to the hostname of this machine
HOSTNAME="localhost"
HOSTNAME="portage"

Finally we need to configure the timezone which shall be used by this guest before completing the configuration from inside our new environment.

TIMEZONE="Europe/Amsterdam"

With our new virtual machine configuration almost complete we can finally enter the new environment using the chroot application and perform the remaining configuration tasks. As you can see from the example below these consist of creating a symbolic link for the timezone we set earlier, adding the ssh daemon to the default run-level so we can connect remotely and setting a password for the root user.

>>> Regenerating /etc/ld.so.cache... 

 * sshd added to runlevel default 

New password: 

Retype new password: 

passwd: password updated successfully 

exit 

Now that the configuration of the virtual-machine and guest operating system is complete we can unmount the filesystems we were using and start the Xen guest domain using the commands shown in the example below.

Assuming that the virtual-machine started correctly we can connect to it to continue with configuring the build-spaces.