Sharing and using the repositories

The importance of shared configuration files

Sharing configuration files and kernel sources

So that the configurations of the build-spaces can be easily made available to other machines we first need to bring them all together in a single location in our testing repository. This is most easily accomplished using bind mounts as in the previous section when we bound the portage files into each build-space. In the example below we create a directory to contain each of our configurations and a file in each of those directories which will be used as the mount point.

With all the mount points created we can add entries to /etc/fstab to bind the /etc/make.conf file and the /etc/portage directory from each of the build-spaces into the testing repository as shown below.

/mnt/buildspaces/x86-64bit-server/etc/make.conf       /mnt/repositories/testing/x86-64bit-server/conf/make.conf        none bind,ro
/mnt/buildspaces/x86-64bit-server/etc/portage         /mnt/repositories/testing/x86-64bit-server/conf/portage          none bind,ro

/mnt/buildspaces/x86-64bit-workstation/etc/make.conf  /mnt/repositories/testing/x86-64bit-workstation/conf/make.conf   none bind,ro
/mnt/buildspaces/x86-64bit-workstation/etc/portage    /mnt/repositories/testing/x86-64bit-workstation/conf/portage     none bind,ro

/mnt/buildspaces/x86-32bit-server/etc/make.conf       /mnt/repositories/testing/x86-32bit-server/conf/make.conf        none bind,ro
/mnt/buildspaces/x86-32bit-server/etc/portage         /mnt/repositories/testing/x86-32bit-server/conf/portage          none bind,ro

/mnt/buildspaces/x86-32bit-workstation/etc/make.conf  /mnt/repositories/testing/x86-32bit-workstation/conf/make.conf   none bind,ro
/mnt/buildspaces/x86-32bit-workstation/etc/portage    /mnt/repositories/testing/x86-32bit-workstation/conf/portage     none bind,ro

We have used read-only bind mounts when binding the configuration files into the repositories so that they cannot be accidentally edited from outside of the build-space they relate to.

 

So that we can proceed as if the server had just booted normally we will need to mount the locations we have just added to /etc/fstab manually as shown in the following example.

As these configurations files are going to be used by both the build-space during the build process and the client during the installation of binary packages we need some mechanism to specify the different options required by each phase while keeping the remainder of the configuration file identical across systems.

We shall be achieving this end by editing the /etc/make.conf file in all the build-spaces to no longer build binary-packages but instead to consume them. We shall also be dynamically including an extra configuration file, called /etc/make.extra, which will contain our local overrides to be used by the build-spaces during package building. The modifications required to the 64bit server build-space are shown in the example below.

CHOST="x86_64-pc-linux-gnu"
CFLAGS="-O2 -pipe"
CXXFLAGS="-O2 -pipe"

PORTDIR=/mnt/portage/portage
DISTDIR=/mnt/portage/distfiles
PKGDIR=/mnt/portage/packages

FEATURES="buildpkg sandbox collision-protect -preserve-libs"
FEATURES="sandbox collision-protect -preserve-libs"
EMERGE_DEFAULT_OPTS="-K"

PORTAGE_NICENESS=0
MAKEOPTS="-j4"

LINGUAS="en en_GB"

source /etc/make.extra

Now we can create a file with the local overrides for the build-space, namely turning the buildpkg feature back on so that binary packages are built and removing the -K switch from the default emerge options so that binary packages will not be used when running emerge inside the build-space. The example below again shows the modifications required to the 64bit server build-space.

FEATURES="${FEATURES} buildpkg"
EMERGE_DEFAULT_OPTS=""

We have only shown the modifications required for the 64bit server build-space in the above examples. If you have installed additional build-spaces remember to make similar modifications for all additional build-spaces as required.

 

Synchronising the testing repository to the stable repository

Before any clients can use the just-built binary packages and the portage tree we need to copy everything from the testing repository to the stable repository as shown here.

Using the repository on the build-server

The build-server can now be configured to use the configuration files and packages from the relevant build-space however before the build-server can access the portage tree and packages using the same paths as every other client we need to add some more bind mount entries to /etc/fstab.

/mnt/repositories/distfiles                               /mnt/portage/distfiles        none ro,bind
/mnt/repositories/stable/x86-64bit-server/conf            /mnt/portage/conf             none ro,bind
/mnt/repositories/stable/x86-64bit-server/kernels         /mnt/portage/kernels          none ro,bind
/mnt/repositories/stable/x86-64bit-server/overlays        /mnt/portage/overlays         none ro,bind
/mnt/repositories/stable/x86-64bit-server/packages        /mnt/portage/packages         none ro,bind
/mnt/repositories/stable/x86-64bit-server/portage         /mnt/portage/portage          none ro,bind

We can then create the required directories to bind mount the repository locations we configured above and then mount them as shown in the example below.

We can then remove our /etc/make.conf file and the contents of /etc/portage/ so we can link to the exported ones and set up those links as shown below.

Finally we can remove the /usr/src directory from the build-server and link to the one exported by the relevant build-space as shown in the example below.

Sharing the testing and stable repositories using NFS

Before we can share the configurations, portage trees, overlays, binary packages and kernel sources from our repositories using NFS we need to install the nfs-utils package. We cannot install this package directly on the build-server however as we have configured it to use the binary packages produced by a build-space and we have not built the nfs-utils package in the build-space yet.

The first step therefore is to build the net-fs/nfs-utils package in the relevant build-space as shown below.

As we have not yet configured a test framework we are unable to test our new binary package before using it so we shall just have to trust that it will work and synchronise the testing repository to the stable repository as shown in the following example.

We can now install the binary nfs-utils package we just built in the 64bit server build-space on the build-host.

Now that we have installed the nfs-utils package we can configure the directories which will be exported. The example below shows the additions required to the /etc/exports file to export the distfiles, testing and stable repositories.

/mnt/repositories/distfiles                    *(ro,no_subtree_check)
/mnt/repositories/testing                      *(ro,no_subtree_check,crossmnt)
/mnt/repositories/stable                       *(ro,no_subtree_check,crossmnt)

The crossmnt export option is critical to the correct functioning of the testing and stable repository exports as without it sub-mounts of the exported directory are not available. As the distfiles repository is entirely contained on a single filesystem this option is not required.

 

We can now start the NFS daemon and add it to the default run-level so that it will be automatically started during the boot sequence.

Using the stable and testing repositories over NFS

Before we can utilise the configurations, binary packages and kernel sources from our repository over NFS we need to install the nfs-utils package on the client. We cannot use the binary package of the nfs-utils package we built in the build-space as we have not yet configured our client to use the packages. Clearly therefore the first step is to install the nfs-utils package built locally on the client using the normal emerge command shown below.

Once we have installed the nfs-utils package we can add entries to /etc/fstab to mount the exported directories from the required repository. The example below shows the entries needed to access the stable repository for the 64bit server configuration.

portage:/mnt/repositories/distfiles                           /mnt/portage/distfiles   nfs   nfsvers=3,ro,noatime,nodev,nosuid,noexec     0 0
portage:/mnt/repositories/stable/x86-64bit-server/conf        /mnt/portage/conf        nfs   nfsvers=3,ro,noatime,nodev,nosuid,noexec     0 0
portage:/mnt/repositories/stable/x86-64bit-server/portage     /mnt/portage/portage     nfs   nfsvers=3,ro,noatime,nodev,nosuid,noexec     0 0
portage:/mnt/repositories/stable/x86-64bit-server/overlays    /mnt/portage/overlays    nfs   nfsvers=3,ro,noatime,nodev,nosuid,noexec     0 0
portage:/mnt/repositories/stable/x86-64bit-server/packages    /mnt/portage/packages    nfs   nfsvers=3,ro,noatime,nodev,nosuid,noexec     0 0

portage:/mnt/repositories/stable/x86-64bit-server/kernels     /usr/src                 nfs   nfsvers=3,ro,noatime,nodev,nosuid,noexec     0 0

We can now create the corresponding mount-points and mount the NFS shares as shown below.

Now that we have the configuration exported by the repository available we can remove the local /etc/make.conf file and any /etc/portage/package/* files and replace them with links to those supplied by the exported configurations as shown below. We also create an empty file at /etc/make.extra so that the source command in the imported make.conf succeeds.

The next step is to remove any kernel sources from your system as we shall be using the kernel sources provided by the appropriate repository. Assuming that you only use the standard sys-kernel/gentoo-sources package for your kernel sources the example below will suffice.

Once all installed kernel sources have been removed we can empty the /usr/src directory and mount the shared one in its place instead as shown below.

To prevent any packages trying to install the sys-kernel/gentoo-sources package, or indeed any other kernel sources package, we can inform portage that this package is provided by other means using the commands shown below.

With the configuration complete we are now ready to update the local portage metadata cache and then re-merge every package on our system. The commands shown in the example below will ensure that the system is using the binary packages provided by the repository for every installed package, that python and perl modules have been correctly updated, that any packages which are no longer required as dependencies have been removed and that dynamic linking consistency has been tested and, if necessary, packages have been reinstalled as required.