Gentoo Wiki


This article is part of the HOWTO series.
Installation Kernel & Hardware Networks Portage Software System X Server Gaming Non-x86 Emulators Misc



This document will outline one approach toward building an infrastructure using gentoo linux. In our office, we have approximately 15 clients, networked behind a NAT. The goal was simple: to install gentoo in such a way that the clients would always remain in sync with each other, maintain themselves with as little oversight as possible, and be easy to roll out changes to the whole infrastructure at once. This turned out to be a rather non-trivial pursuit, so I'm sharing my experiences in the form of a Howto with the rest of the gentoo community.

This HOWTO assumes that the reader is adept at the basics of Gentoo Linux. Where applicable, links are provided for more reference, but a lot of this document will make no sense to a beginning reader. This HOWTO was written from This Document. The modifications for the gentoo wiki were intended to generalize some of our organization-specific configurations. This code is not provided as something that will "just work" for you. You'll have to take some time to think through the process and modify the scripts to your needs. When you've finished, you will have a client infrastructure that mostly takes care of itself.

Creating a Build Server

In our setup, all of our clients contain identical hardware. Although hardware could vary, there will be points along the way that you'll have to adjust for such differences (ie, xorg.conf). The first step is to take one of your clients, and call it the "Build Server." My unoriginal hostname for our buildserver is gentoo.

The system build

The buildserver will be responsible for storing binary packages of all installed software, so you may want to give it a slightly larger harddrive (My buildserver has 40GB, whereas the clients each have 20GB). Pop in your 2005.0 (or newer) gentoo LiveCD and build up the system. When partitioning, be sure to create a partition for your packages. I chose to allocate 20GB to /packages.

In /etc/make.conf, add:

File: /etc/make.conf

Now, whenever you install a package from portage, make sure to use emerge -b in order to build a binary copy of the package. Also, make sure that you use CFLAGS that will be compatible with all of the machines in your infrastructure. While the system is building, sit back and relax.

Tweaks to make it a buildserver

This working gentoo box now needs to offer a few services before we can consider it a build server.

Local Portage Mirror

There's no point in using gentoo's limited bandwidth to have each of our clients sync every day. Instead, we'll make use of a local copy on the buildserver. First, make sure that net-misc/rsync is installed (it should be part of the gentoo base).

Create the following entry for the portage mirror in /etc/rsyncd.conf:

File: /etc/rsyncd.conf
#modern versions of portage use this entry
path = /usr/portage
comment = Gentoo Linux Portage tree mirror
exclude = distfiles/ packages/

Don't forget to create a motd for rsync:

File: /etc/rsync/rsyncd.motd

Welcome to the Rescomp Portage mirror!


You'll also have to add rsyncd to the default runlevel. rc-update add rsyncd default
and on the client-box:

File: /etc/make.conf
SYNC="rsync://<local rsync-server ip>/gentoo-portage"

Creating a Portage Overlay

Depending on the needs of your organization, you may need to have a portage overlay to contain some custom ebuilds. I placed my portage overlay on the build server in /usr/local/rescomp/portage. The exact path is up to you, just be sure to place it in the correct spots in the following configs. More details on overlays and third party ebuilds can be found at wiki HOWTO Installing 3rd Party Ebuilds.

We want to allow the clients to sync their overlay trees with ours, so we need to edit /etc/rsyncd.conf to include:

File: /etc/rsyncd.conf
path = /usr/local/rescomp/portage
comment = Rescomp Overlay for Gentoo Portage

Lastly, make sure to emerge -b app-portage/gentoolkit-dev . We'll need the gensync tool from it when we configure the clients.

Binary Packages File Server

This section is really straightforward. We have to export the /packages directory to the clients so that they can install using the pre-built binaries. To configure the buildserver, we look at /etc/exports:

File: /etc/exports
#We add this line to export packages
/packages     ,sync,no_root_squash)

This should look pretty simple. We'll discuss root_squash when we examine the ideals for client configuration. At the moment, we know that the /packages directory is being exported to all the office clients. They can now mount that share to their /packages directory. We'll figure that out when we detail client installations and configurations.

Also, you'll want to do the usual when enabling nfs:

Code: NFS Boot Services
rc-update add nfs default
rc-update add portmap default

Ideal Configurations Overlay

The next issue is that we need a single place in which we can make changes to configurations that will take effect on all of the clients. My solution to this problem was to create a directory on the buildserver with a filesystem overlay.

Anything that is not default, with the exception of host keys and hostname, is overlayed to the filesystem every night. So we have a standard tree underneath /packages/overlay/ with directories such as etc and sbin.

Because of the way this works, you should not put any host-unique files into this directory. For example, the /etc/conf.d/domainname file is shared, but /etc/conf.d/hostname is not in the tree. Any host-specifics will be taken care of by the installation script. If they need to occur after the client has been built, a little screen scripting can go a long way.

The ideal configurations are located in /packages/overlay. This directory has a couple of aliases via symlink, but it's all the same. Every night, as a cron job, the clients will rsync from the nfs mount to the root filesystem. Thus, any configuration changes made to the ideal tree will be pulled to the clients during the nightly script (which will be detailed in the maintenance section.

The build server functionality for this service has already been provided by exporting /packages, so we're all set. The reason we do no_root_squash is so that root on the client can see all the files. The shares are read-only, so we aren't concerned with people changing things -- just seeing sensitive configuration information.

One important file is portage's world file. The way I chose to share this was:

Code: Exporting /var/lib/portage/world
mkdir /packages/overlay/var{,/lib{,/portage}}
mv /var/lib/portage/world /packages/overlay/var/lib/portage/world
ln -s /packages/overlay/var/lib/portage/world /var/lib/portage/world

Now all of the necessary services are running on the build server. We have binary packages to share. We just need the install scripts to be in place, but we'll take care of that in the client discussion.

Creating the Clients

The easy answer to creating a client is that you just pop in your LiveCD and grab a drink. Problem is, you're going to have to create that LiveCD first.

Scripting the Installation

You have likely realized that gentoo does not make use of a friendly user interface for installations. That means you've also realized that performing an installation is nothing more than typing many commands in. The thing is, these commands are predictable, so we're going to make the computer do them for us. The installation process is divided into 6 scripts (you could do this differently, this was how I chose to deal with it).

I will link the source of each of my scripts and give an explanation for what it does. Keep in mind all code is released under the GPL, (c) 2005 Residential Computing, UC Berkeley. A copy of the License is available with the source files at LICENSE.

Install all of the following scripts to buildserver:/packages/overlay/sbin.


This script is launched as part of the system login. It will automagically appear on tty1 immediately after booting from the RC-LiveCD. Of concern is the fact that the script expects the network to be established on the subnet. It will prompt for an IP address before going into a non-interactive frenzy of installation madness. It plays around with ifconfig to make sure that the ethernet is up -- for some reason, the Live CD likes to just drop the interface once or twice. The script continues by partitioning the harddrive (it better be at least 20GB) and creating mount points. It also mounts the local and remote filesystems. The script ends by handing control off to phase1. Feel free to tweak to your heart's content (for example, you may have dhcp - you're life is easier).


As part of this script, we extract the stage 2 tarball, take care of some logistics, and then control the rest of the phases. emerge has an interesting tendancy of causing a script to exit. As such, we have to call emerge with the chroot command from phase1. In fact, phase1 is basically a series of chroot commands, calling each of phase2, phase3, phase4, emerge system, and emerge world.

The emerge commands are likely the most noteworthy. We're using the -K flag to indicate that we are only to use binary packages from the PKGDIR. When we emerge world, we have to remember the the -u flag does not update packages that aren't installed, even if they're found in the world file. Also, we don't want to use -e, because we waste time re-installing all of the system packages. The solution is to cat</var> the world file to <tt>xargs as seen in the script. -n tells portage not to emerge already-installed files.


This phase executes a couple of simple steps for our installation, including syncing the configuration against the ideal with rsync and updating the environment. Lastly, it syncs the portage tree.


Phase3 performs two very important functions. It provides a work-around to a problem with upgrading gcc, and it installs some tools we need to continue our escapade. The work around is caused by the fact that when gcc updates, it edits the environment, but then we leave the chroot and lose the changes (at least, this is my theory. Could just be a bug.). Ironically, the gentoo tools that would normally fix this problem automagically are written in python -- which is linked against a missing library. Doh.

The solution comes in a bit of scripting magic. We have to determine the current path to gcc, update /etc/, and update the environment. More easily said than done. It's all in the script.

We have to get the gentoolkit-dev package so that gensync will work properly. Then we do an rc-sync to get the overlay. We're now ready to continue the installation by intalling all of the world packages.


Phase4 picks up after all of the packages are installed. It's basically a post-install script. Most Rescomp customizations that can't be synced out should be executed here. We also set the hostname and post-install network information. Grub gets installed to the boot device. Links are made. All that jazz. Phase4 brings us back to an interactive environment by prompting for the root password to be set on the sytem. After that's done, it will drop you to a shell inside the chroot. Read the banner.


[1] is run the first time the system is booted. We run some commands that take care of things that can't be rsynced out via files (for example, copying over ssl certs or enabling the office printer). This file will likely be very site-specific.

Custom LiveCD

Creating the Image

After we've created installation scripts, we want a really easy way of running them. This can be done by editing the LiveCD on the buildserver:

  1. Make a directory to contain the CD image (I used /packages/live-cd/livecd)
  2. Mount a current livecd (either iso via loopback or an actual CD)
  3. cp -a /mnt/cdrom/* /packages/live-cd/livecd/
  4. Unmount the CD
  5. Mount the livecd.squashfs file to some mountpoint (you will likely have to download the driver, patch the kernel, and build the module Squashfs Homepage). Example:
mount -t squashfs -o ro /packages/live-cd/livecd/livecd.squashfs \
  1. cp -a /mnt/squash/* /packages/live-cd/livecd.squashfs/
  2. Unmount the squashfs image
  3. Edit the contents of /packages/live-cd/livecd/livecd.squashfs to your needs (We'll explain what these are in just a bit). This is the root filesystem after the initrd does the pivot_root. Our site-specific changes will be outlined below.
  4. After making changes, rebuild the squash image:
rm -f /packages/live-cd/livecd.squashfs.img
mksquashfs /packages/live-cd/livecd.squashfs/ \
  1. And copy it into place
cp /packages/live-cd/livecd.squashfs.img \
  1. cd /packages/live-cd/livecd/
  2. And rebuild the iso:
mkisofs -no-emul-boot -boot-load-size 4 -boot-info-table -r \
    -b isolinux/isolinux.bin -c isolinux/boot.catalog \
    -o /packages/live-cd/gentoo-install-livecd.iso .

Note: Watch the escaped line breaks here. In fact, don't use them. mkisofs gets confused by them -- doesn't treat them as normal whitespace. They're just here for readability.

LiveCD Customizations

Assuming you've begun the process from above, and you're wondering how you should customize you're CD, read on:

The first step is copying gentoo-install-init into /sbin of the squashfs. Easy.

The file /bin/bashlogin is started on each virtual console by init. You can't directly edit /etc/inittab, because some other startup routine will literally comment out your changes and return them to the default.

So, I edited /bin/bashlogin to read:

File: /bin/bashlogin
# This is only installed by baselayout when USE=livecd to enable
# serial consoles for some architectures

# We still want to do the defaults on every terminal
cat /etc/motd 2>/dev/null
cd /root
[ -e .bashrc ] && source .bashrc
[ -e .bash_profile ] && source .bash_profile

# This returns which console we're working on.
OUR_CONSOLE=`ps x | awk "/$$/ { print \\$2 }" | head -n 1`

# We only want to start the script on tty1 so the user doesn't have to hunt
# through the terminals for it. We also only want to start the script once.
# This lock file will prevent the script from being launched again after it
# exits. You could still do it by hand if you chose...
if [ $OUR_CONSOLE = "tty1" -a ! -f /var/lock/gentoo-install ]
        touch /var/lock/gentoo-install
        exec /bin/bash -c /sbin/gentoo-install-init
        exec -l /bin/bash -l

That's it for custom edits. With those two changes, you can rebuild the livecd.

Custom Configurations

There are a couple configurations that must be placed in the /packages/overlay hierarchy before the installation can continue. Take a look at these:

You'll need /packages/overlay/etc/fstab. Make sure to mount the /packages export from your buildserver to </packages> on the client. During the installation, the gentoo-install-init script takes care of this detail. Make sure it's right for your situation.

Make sure to have /packages/overlay/etc/hosts. The script uses this for hostname setting. You can tweak away the need for hosts if it's appropriate for your environment.

File: /packages/overlay/etc/make.conf
#Be sure to include the following:
PORTDIR_OVERLAY="/usr/local/rescomp/overlay"  #If you're using one

As you build up the infrastructure, you'll have more files that need to be synced out to the clients, but this should provide a solid starting point for the installation process.


Place the CD in the drive and watch it go.

Tip: Once you have a couple clients with a minimal installation, consider setting up distcc to share the load once you start compiling Xorg and desktop environments.

Package Management

Gentoo has an amazing package management system that keeps track of updates and dependencies. There are a few ways that we've modified and improved this system in order to keep things running smoothly for the office clients.

Installing New Packages

In order to install a new package onto the workstations, you must run the commands from the build server. The package has to be installed on the build server. At the same time, a binary will be built, and placed in /packages.

Code: When building packages, use:
emerge -bva package
# The -a signifies that it should prompt you before actually building
# The -v is for verbose
# The -b tells it to build binaries

If you've already installed a package, but need to roll out a binary for it, use:

quickpkg package

Once a package has been installed onto the build server, it will be rolled out to the clients that night during the cron runs. No further attention is necessary. If you must install a package on a client immediately, you can either run the cron-script manually, or just run:

emerge -Kva package
# The -K tells it to use ONLY binary packages. The install will fail out if
# they are not available. If you use -k, it will build from ebuilds in the
# event that the packages aren't available -- we don't want this.

Maintenance and Configuration

I've made a few references to magical cron scripts. These run commands on a daily basis to keep everything in sync. You should take a look at these scripts and edit them to your own needs.

Kernel Management

Directly from our wiki: So, in an attempt to complicate matters, the kernel is a very special beast. The way we deal with kernels is a bit tricky, so read carefully. The directory /lib/modules is actually a symlink that points to /packages/overlay/lib/modules/ on gentoo. In this way, the modules directory on gentoo is the same as the modules directory on the clients (given a 2am sync). This works transparently -- as you'll see.

Now you're getting really antsy and want to install that brand new kernel release. Cool. On gentoo,

$ cd /usr/src
$ sudo rm linux
$ sudo ln -s linux-new-version linux
$ cd linux
$ sudo cp /boot/config-`uname -r` ./.config
$ sudo make oldconfig
$ sudo make xconfig/gconfig/menuconfig/your favorite config to fine tune the beast
$ sudo make
$ sudo make modules_install  # These go to /lib/modules which is in fact a symlink.
                             # This means they're now installed on all the clients (or will be at 2am)

Next, you can install the kernel in your favorite way. I personally prefer to manually copy bzImage, .config, and to /boot, always appending the kernel version. Then I edit /boot/grub/grub.conf by hand. You probably can do make install too, but I don't trust it. I guess I'm paranoid. Or just stupid and like to make things harder on myself.

If you have everything in /boot as it should be, you have to execute the following on gentoo: $ sudo cp -a /boot /packages/rc-office-conf/ This ensures that the contents of the boot partition are the same on all the clients (after 2am).

Last, but not least, you must re-emerge nvidia-kernel, AFTER /usr/src/linux points to the new kernel and you've made modules_install!

##Only if you use this kernel module. Insert code for your modules here (eg, ipw2200)
gentoo$ sudo emerge -vb nvidia-kernel
##Assuming you're using runonce's, outlined in script comments. See notes at end of this HOWTO
gentoo$ sudo cat <<EOF > /packages/rc-office-conf/sbin/rc-office-runonce
#! /bin/bash

emerge -kv nvidia-kernel

gentoo$ sudo chmod 744 !$

A note on, umm... good sysAdmin skills. You should probably test the procedure in a controlled environment before running it on gentoo. So, pick your favorite (or perhaps least favorite) office client. You'll notice /usr/src is one of those magic directories mirrored from gentoo. Build the kernel as above, except for copying /boot over (you can't, because /packages is a ro nfs mount). Reboot the client. If all is well, you can scp -r the appropriate directories to gentoo (/boot, /lib/modules, and /usr/src/linux -- be sure to change the linux symlink on gentoo first). You could also just move the config over and build again. You can leave the files as they are on the client you abused, because they'll all be verified during the sync.

Final Remarks

I hope you can make use of these findings in maintaining large groups of gentoo clients. I'm going to end with a list of known issues with this approach:

Lastly, good luck! I'd love to hear your comments regarding how you made this stuff useful in your own set up. Let me know if you have!

Retrieved from ""

Last modified: Fri, 05 Sep 2008 06:38:00 +0000 Hits: 3,001