Gentoo Wiki



Diskless in a Heterogeneous Environment

How does this differ from Regular Diskless

This HOWTO describes how to set up diskless in a heterogeneous environment. This means an environment with many different kinds of hardware, and different configurations. The Regular Diskless HOWTO is aimed at a single client configuration, and does not attempt hardware detection or dynamic configurations. (though this HOWTO is based off the Regular Diskless HOWTO, and as such will contain some very similar information).

To do this, we can set up an image with hardware detection, and dynamic configuration. The configuration is based on the hostname or IP address.


Server environment


In a diskless configuration, the percieved peformance of your client machines will depends entirely on your server configuration. You should select a server with a decent disk configuration and lots of RAM. Software or hard RAID is generally a good idea. You will also definitely want to have a gigabit network connection to the server.

My current server is a Dual Xeon 2.4Ghz with 2GB of RAM. It has 2 7200RPM IDE disks mirrored (Raid 1) using Linux software RAID (see HOWTO Gentoo Install on Software RAID for more information on setting up software RAID). This server serves around 200 client machines with very reasonable peformance, and a fairly low loadavg.


You will need the following packages installed on the server:


Kernel Configuration

You will need to make sure to have NFS support in your kernel. Client support is not necessary, but is recommended to prevent some errors starting the NFS service at boot.

Linux Kernel Configuration: NFS
File systems  --->
 Network File Systems  --->
  <*> NFS file system support
  [*]   Provide NFSv3 client support
  <*> NFS server support
  [*]   Provide NFSv3 server support


Set up the TFTP server

File: /etc/conf.d/in.tftpd
# /etc/init.d/in.tftpd

# Path to serve files from
# (this can be located anywhere, this is just an example)

# For more options, see tftpd(8)

Next we need to set up pxelinux. This comes as part of sys-boot/syslinux, you can install this package on the server

  1. emerge --pretend --verbose syslinux
  2. cp /usr/lib/syslinux/pxelinux.0 /diskless/tftproot/gentoo/
    • Put the pxelinux.0 in the TFTP directory for your image, for this example that directory would be /diskless/tftproot/gentoo/.

Now we need to make a configuration file for pxelinux, the default configuration file for pxelinux is located at pxelinux.cfg/default relative to pxelinux.0, so we need to create the default configuration file:

File: /diskless/tftproot/gentoo/pxelinux.cfg/default
DEFAULT vmlinuz
APPEND root=/dev/nfs nfsroot=,ro,rsize=8192,posix

We will need to copy the kernel over to the TFTP root after the client image is completed.

Warning: This section may be incomplete.

To set up yaboot, we first need to install sys-boot/yaboot on the PPC image/machine. Next copy /usr/lib/yaboot/yaboot binary to the TFTP server, something like scp /usr/lib/yaboot/yaboot root@ should copy the file to the appropiate place. Next you need to create a yaboot.conf file in the same directory as the yaboot binary. There is an example yaboot.conf that should work below.

File: yaboot.conf
timeout = 50
defaultos = linux

image = enet:0,maclinux
label = gentoo-mac
append = "root=/dev/nfs nfsroot=,ro,rsize=8192,posix ip=dhcp"

Note: The OpenFirmware installed on Apple machines has a bug where it can't boot from a subdirectory, so the yaboot and yaboot.conf need to be in the root of the TFTP server.


Next you need to set up your exports, your /etc/exports should look something like this:

File: /etc/exports
# /etc/exports: NFS file systems being exported.  See exports(5)
Note: This setup assumes that the location of your Gentoo image is at /diskless/gentoo you can put the image anywhere you like. This also exports to read-write to allow that machine to do updates on the image (so we are not doing all our compilation on the server.


Now we need to configure the DHCP server, this example configuration hasn't been tested.

File: /etc/dhcp/dhcpd.conf
default-lease-time 604800;
max-lease-time 604800;
ddns-update-style none;
get-lease-hostnames true;

shared-network mynetwork
  subnet netmask
    option broadcast-address;
    option routers      ;
    option domain-name-servers,;

    # uncomment this to give out addresses to any host
    # that requests one

      # this is the address of the TFTP server

      # the name of the PXE file to load
      filename "gentoo/pxelinux.0";

      # these are individual host definitions
      host diskless-1
        hardware ethernet 00:11:22:33:44:55;

      host diskless-1
        hardware ethernet 00:11:22:33:44:55;


      # this is the address of the TFTP server

      # the name of the PXE file to load, you could load an alternat
      # image or kernel for this group of hosts
      #filename "gentoo2/pxelinux.0";

DNS Server

You will also need to set up a DNS server, if your network does not already have one. Every host that will be booting diskless needs to have an IP that can be reverse-DNS'ed so it can figure out what it's hostname is. You can look at this howto for more information on setting up a DNS server: HOWTO Setup a DNS Server with BIND

Tip: If you have a private network behind a NAT server, you can use net-dns/dnsmasq to cover both DNS and DHCP. It is much more simple to set up than either BIND or DHCP.

Image setup

This chapter assumes that you already have an image ready to be used as a diskless image.

Tip: Compiling with -Os can help speed up the diskless setup, as the clients will need to load less data from the server to run applications.

Warning: When you are setting up your image, remember to set your CFLAGS to the work on the least capable machine you mighe be using this image on. Newer machines are usually compatible with older machines, but old machines are rarely 100% compatible with newer machines

Note: If you are going to have "user" mountable filesystems on the diskless image, you will need the "symlinkmtab" patch that allows mount to write to mtab even if it is a symlink. You can get the latest version of this patch from Bug 98403 in the Gentoo Bugzilla

Kernel Configuration

The absolute minimum kernel setup needs kernel-level network autoconfiguration and NFS booting:

Linux Kernel Configuration: Kernel Network Autoconfiguration
Networking  --->
 Networking options  --->
  [*]   IP: kernel level autoconfiguration
  [*]     IP: DHCP support

You will need NFS and NFS root support. I also recommend NFS server so the nfs service starts cleanly on boot. NFS over TCP is also a good idea, so if a some brain-dead firewall on your network decides UDP is a bad idea, you will still be able to boot your diskless systems.

Linux Kernel Configuration: NFS
File systems  --->
 Network File Systems  --->
  <*> NFS file system support
  [*]   Provide NFSv3 client support
  <*> NFS server support
  [*]   Provide NFSv3 server support
  [*]   Provide NFS server over TCP support
  [*] Root file system on NFS

You will also need tmpfs support in the kernel for mounting directories that need to be read-write on the client machines. Procfs is also needed so the scripts can access various information on the running kernel.

Linux Kernel Configuration: tmpfs, procfs
File systems  --->
 Pseudo filesystems  --->
  [*] /proc file system support
  [*] Virtual memory file system support (former shm fs)

Warning: Your kernel setup also absolutely MUST have the drivers for every network card that any client machine might be using. These drivers have to be built right in to the kernel, module is not good enough, since there will be no way to load the module before the network driver is loaded and the NFS root filesystem is mounted.

For the rest of the kernel configuration, you should make modules for every piece of hardware you might possibly have or ever have anywhere on your network. You can get the kernel configuration I use for gentoo-sources-2.6.19-r2 at "", it should give you a fairly good starting point.

Note: The "skge" and "sk98lin" drivers included with the kernel do not work with NFS booting - you will have to patch your kernel with the drivers Marvell provides on their website You will need to select "Install package for Linux kernel 2.4 and 2.6").


Now that you have the server set up, it's time to get the image ready. This assumes that you have already made an image in a chroot, and installed all the software you will need on it, and set up all the servieces. You do not need to configure X or anything hardware specific on the image, that will be done automatically at boot.

Image Cleanup

First we have to a little cleanup on the image, we will get rid of any unnecessary files, and other junk.

  1. Delete the contents of /tmp and /var/tmp/, a simple rm -rf /tmp/* /tmp/.X11-unix/* /tmp/.ICE-unix/* /var/tmp/* should get rid of most of the temporary files.
  2. Clean out any logs that are on the image, cd to /var/log and delete any files except lastlog.

Copying Image to Server

Next we need to copy the image to the server, the easiest way to do this is to the NFS share set up earlier. Assuming you are on the machine that you specified as rw in /etc/exports on the server, do something like this:

Code: Copying Image Over
# mkdir /mnt/diskless
# mount /mnt/diskless
# cp -a /path/to/image/* /mnt/diskless

This will probably take a long time as you are copying a huge amount of data, and creating a lot of small files (which is quite slow over NFS).

Copying Kernel to TFTP

Once you have the image copied over to the server, you can copy the kernel to the TFTP directory on the server. That will look something like this (depending on the particular version of the kernel you are copying.

Code: Copying Kernel
# cp /diskless/gentoo/boot/vmlinuz-2.6.14-gentoo-r3 /diskless/tftproot/gentoo/
# ln -sfn vmlinuz-2.6.14-gentoo-r3 /diskless/tftproot/gentoo/vmlinuz

Having the actual kernel that the pxelinux configuration points to be a symlink allows you to keep several kernels around and easily switch between them simply by changing the file that the link points to.

Setting up the Diskless

Compile Machine

Now that the image is on the server, we probably want to set up a machine to do emerging, and maintenance on. We could do this directly from the server, but it's generally best to keep as much load off the server as we can, so I recommend that you set up a machine with the diskless image mounted read-write so you can chroot into the diskless.

Since you will be emerging on this chroot, you will need some bind mounts. I also suggest bind mounting directorys of non-critical cache files so you won't be storing all kinds of unnecessary data on the server.

Code: Script to Mount Images

# this is the directory you want to store temporary files
# for the images in

if [ "$IMAGE" == "" ]; then
    if ! [ -d /diskless/$IMAGE ]; then
        echo "ERROR: directory \"/diskless/$IMAGE\" does not exist!"
        exit 255

case "$OPERATION" in
        mount -o bind /dev /diskless/$IMAGE/dev
        mount -o bind /proc /diskless/$IMAGE/proc
        mount -o bind /sys /diskless/$IMAGE/sys

        [ -d $DS/tmp/ ] || mkdir -p $DS/tmp/

        [ -d $DS/tmp/$IMAGE ] || mkdir $DS/tmp/$IMAGE
        mount -o bind $DS/tmp/$IMAGE /diskless/$IMAGE/tmp

        [ -d $DS/vtmp/$IMAGE ] || mkdir $DS/vtmp/$IMAGE
        mount -o bind $DS/vtmp/$IMAGE /diskless/$IMAGE/var/tmp

        [ -d $DS/distfiles/$IMAGE ] || mkdir $DS/distfiles/$IMAGE
        mount -o bind $DS/distfiles/$IMAGE /diskless/$IMAGE/usr/portage/distfiles
        umount /diskless/$IMAGE/dev
        umount /diskless/$IMAGE/proc
        umount /diskless/$IMAGE/sys
        umount /diskless/$IMAGE/tmp
        umount /diskless/$IMAGE/var/tmp
        umount /diskless/$IMAGE/usr/portage/distfiles
        echo "Syntax: $0 <mount|umount> <image>"

This script takes two parameters, the action (either mount or umount) and the name of the image. This script assumes that your images are mounted at /diskless/<image name>. So to setup the bind mounts for an image names "gentoo" you would run the script with the parameters "mount gentoo".

Diskless Scripts

Installing Diskless Scripts

There is an overlay available for the diskless, to get it, install subversion on your image and do this (you can also do it with subversion on the server, just provide the path to the image before the /usr/local/diskless):

emerge --pretend --verbose subversion

Code: Checking out the overlay

# svn co /usr/local/diskless

Note: Does anybody have copy of these scripts? --Stootch

Now edit /etc/make.conf and add /usr/local/diskless to the PORTDIR_OVERLAY on your image. Once you have that done, you can install the diskless scripts with emerge diskless.

"Layer" Introduction

These diskless scripts use a "layering" system to allow for separate configurations for different systems. Any files that need to change in different systesm, or change in the live system are replaced with symlinks to the "local" (the name of which is configurable) directory for the system directory they are in. For example /etc/fstab is actually a symlink to /etc/local/fstab, /etc/X11/xorg.conf is a symlink to /etc/local/X11/xorg.conf. The "local" directories are mounted read-write via either tmpfs or bind mounts to a locally mounted "scratch" drive (explained further down).

When the system boots, the initscripts consecutively copy each layer into the "local" directories on the system. By default the layers are stored in the /diskless directory on the image. This is configurable.

Configuration "Layers"

The order the layers are loaded are as follows:

  1. The "base" layer, this layer contains every file that is considered a "local" file, every system copies this layer first. This layer is stored at /diskless/base.
  2. The "network" layer, this layer is based on the machine's subnet (192.168.X.255), it often contains a resolv.conf and any other files that vary on a per-network basis. This layer is stored at /diskless/networks/*.
  3. The "lab" layer. This layer is based on the hostname, it is determined by taking the part of the hostname before the first "-" character. For example, a machine named lab3-14 would be in the "lab3" layer. This layer is stored at /diskless/labs/*.
  4. The "host" layer, this layer is simply the hostname of the machine, without the domain part. For example would be just lab3-12. This layer is stored at /diskless/hosts/*.
  5. Finally, the IP layer, this layer is simply the full IP address of the machine. This layer is stored at /diskless/hosts/*.

The diskless scripts need to be configured, there is a basic configuration file located at /etc/diskless/diskless.conf. This file should be fairly self-explanatory, the most important parts are the DIRS variable and the LOCALDIR variable. The DIRS variable is a space separated list of the directories that have a locally mounted subdirectory, and the LOCALDIR is the name of the local subdirectory.

There also should be a file in /etc/diskless/local.d/ for each directory listed in DIRS. These files contain the relative paths of all files in that directory that need to be stored locally. When the script sets up the diskless system, these files will be replaced with relative symlinks to the same path in the "local" directory. The "local" directory on the image will be populated with symlinks to the "base" layer. On booting, the client machines will mount a tmpfs filesystem over the "local" directory very early in boot, so they won't see these symlinks, only the files that are in the "local" directory after the layer copying finishes.

Finally, there is an informational file at /etc/diskless/about, this file contains the "name" for the current image and a short description. The "name" of the image has to be the entire first line, and cannot contain spaces or special characters. The name is an informational name, it is mainly used to name the prompt after chrooting into the diskless image. The contents of this file is displayed when the "about" command is typed from the image.

The "diskless" Script

Included with the diskless package is a script called diskless, by default this script is installed at /usr/sbin/diskless. This script provides several functions to help managing layers. The script has different options depending whether it is run on the master (with a rw root) or one of the client machines.

On the client machines it provides a "relayering" function. This function allows you to re-execute the "layering" procedure that was done at early boot, allowing you to apply a configuration change on the client machine. To relayer a client machine simply use the command diskless relayer from the client you want to relayer.

Another useful command is the diskless setup command. This command will perform all the initial steps to set up the layering on an image. It creates the /diskless/ tree and sets up all the symlinks for the local directories.

Tip: The full documentation for the "diskless" script is a available by executing diskless with no options.

Warning: The "diskless" command is a work in progress, that means that some functionality may be unimplemented or only partially working. It is also possible that some functionality might not work and/or break something. Please test any functionality in a non-critical system before using it in a production environment. If you do find a bug in the script, please report it (the "discussion" for this page will be monitored by the maintainer of these scripts) and it will be resolved as soon as possible.

The "diskless" initscript is the first service that starts at boot. This script sets up the hostname and domainname. Then mounts the "local" directories with the tmpfs filesystem and copies the "layers" into them. Finally this script initializes the "mtab" with the currently mounted filesystems.

All the configuration options that this script accepts are in /etc/diskless/diskless.conf.


Autoconfig is the script that detects the hardware in the system and loads the modules for that hardware. This script uses sys-apps/hwsetup and sys-apps/hwdata-gentoo to detect the hardware in the system and determine the appropriate module to load. This is the same method that the Gentoo LiveCDs use to detect hardware. This script is a modified version of the script with the same name that the LiveCD's use.

The configuration for this script is located at /etc/conf.d/autoconfig. In the configuration file, you can turn off the sound detection, and set the mixer devices that the script automatically unmutes. It also has switches for SCSI, hotplug and GPM support.

This script generally runs very early in the bootup procedure, after the "diskless", "checkroot", "modules" and "hostname" services.

findlocal, uselocal

The "findlocal" and "uselocal" scripts work together to make use of the local disks in a system. "findlocal" finds local partitions that can be mounted and adds them to /etc/fstab. Between "findlocal" and "uselocal" the system runs the standard Gentoo initscript "localmount" that mounts local filesystems. Once the filesystems found by "findlocal" are mounted by "localmount", "uselocal" will make use of them.

The "findlocal" works by scanning all disks in the system looking for partitions formatted with filesystems listed in /etc/conf.d/findlocal. This configuration file is fairly well documented, and contains a switch as to whether to use any partitions found that are marked as Linux swap partitions. It also contains a list of possible filesystems to use as "scratch" drives. A "scratch" drive is a partition that is mounted at /mnt/scratch on the machine, and has subdirectories bind mounted to the "local" directories on the system, as well as /tmp and /local. The "scratch" drive's main purpose is to provide local disk storage for the local files so they do not need to be stored in tmpfs using up system memory. Anything found in the paths on the scratch drive where it bind mounts the "local" directories can be optionally wiped before being used.

"findlocal" uses sys-apps/parted to find partitions by formatted filesystem, rather than the marker on partition. This means that the marker is ignored, if a filesystem is marked "FAT32" and formatted ext2, it will be detected as an ext2 partition.


"xorgconf"configures the X server, it will generate /etc/X11/xorg.conf. This script generated an initial configuration file by using the X -configure, this executes X, telling it to output a new configuration file in /root/ The script then edits this configuration file, fixing the mouse setup to a working one (X -configure never actually makes a working mouse configuration). It also sets up the default resolution as specified in /etc/conf.d/xorgconf.

The configuration for xorgconf contains a setting to turn off X, turning this setting on will make the script modify the inittab so the desktop manager won't run when /etc/init.d/xdm start is executed, then the script will exit.

Various files

Now that we have configured the image how we want it, and initialized it as a diskless setup, there are a few finishing touches that we need to do.

First we should set up the fstab for the image, here is an example fstab that sould work for a diskless setup. The first line is critical, otherwise you will get odd errors from the initscripts as the system is booting.

File: /etc/fstab
# <fs>          <mountpoint>    <type>     <opts>                           <dump/pass>

/dev/root       /              nfs      ro,rsize=8192,posix,hard,nfsvers=2  0 0
/dev/cdrom      /media/dvd     iso9660  ro,user,noauto,owner                0 0
/dev/cdrom1     /media/cdrom   iso9660  ro,user,noauto,owner                0 0
/dev/floppy/0   /media/floppy  auto     rw,user,noauto,owner                0 0
/dev/sda1       /media/usb     auto     rw,user,noauto,owner                0 0

none            /dev/shm       tmpfs    defaults                            0 0
none            /tmp           tmpfs    defaults                            0 0

Image Maintenance


To update the image (update or install packages), simply chroot into the image and emerge as you would on a normal machine. If there are new files that need to be stored locally, use the diskless command to add them to the list of locally stored files. There is no need to do source /etc/profile once you chroot, as the /root/.bashrc installed by the diskless ebuild will do this for you, as well as change your prompt to match the first line in /etc/diskless/about.

From time to time, an update will silently overwrite or move files that the diskless scripts depends on for boot. The diskless script has a function to clean up the more commonly broken parts of a diskless setup. To run this cleanup, use the diskless fix command.

Note: It is a good idea to run the "diskless fix" command after every update, as sometimes an update can change/overwrite a critical file, this often happens with /etc/

Client Machine Configuration

Hardware Requirements

There aren't a whole lot of requirements for the client machines. If you are going to be running something like GNOME or KDE, I would recommend at least 512MB of RAM.

If you want the machine to not seem absurdly slow, you probably want to have a 100 megabit network connection between all the clients and the server. Anything less (10 megabit, for example) results in the client machine generally being very slow.

You will also need to have a network card and BIOS capable of booting off the network, most modern systems can do this, but with older systems you may encounter some that can't. If you do encounter one that can't, you can purchase a PCI network card with a network boot ROM.

Getting Machines to Boot Diskless


If your motherboard has an onboard network card that supports PXE, you will have to enable PXE, then set the LAN as the first boot device in the system. It is a good idea to remove all other boot devices from the list. You may also want to set a setup password for the BIOS so no one decides to boot the machine off a CD or floppy.

If your network card doesn't support PXE, you will have to buy one that does, and install it in the machine. These cards often have an onboard BIOS that initializes the PXE and boots off the network. You may need to tell the BIOS to initialize the boot ROM.

Mac OpenFirmware Setup

First you need to get the OpenFirmware prompt, to do this boot up the machine holding down Apple-Option-O-F on the keyboard. At the OpenFirmware prompt, type these commands to change the default boot device to the network.

Code: Commands to set OpenFirmware to Boot from Network

This sets OpenFirmware to boot from the network

 > setenv boot-device enet:0,yaboot

This will set a password and lock the OpenFirmware so people can't make changes without the passwords.

 > password
 > setenv security-mode command

This will save all settings and reboot the machine.

 > reset-all
Note: The OpenFirmware installed on Apple machines has a bug where it can't boot from a subdirectory, so the yaboot and all required configuration files need to be in the root of the TFTP server.

Local Partition Setup (Swap and Scratch)

The diskless setup can taka advantage of the local disk to cache files and have a swap partition. by default, findlocal will use any reiserfs partitions it finds as scratch disks. This is configurable in /etc/conf.d/findlocal. findlocal will also use any swap partitions it finds on any disks, this is also configurable in the configuration file.

To prepare the local disk on a machine for running diskless, simply create a swap and Linux partiton and format the swap partition with mkswap and use mkreiserfs to format the Linux partition with reiserfs. The default findlocal configuration will ignore any other partitions it finds.

There are plans to implement an automatic partitioning system in findlocal, but as of yet it is not implemented.

Useful Extras

Automounting Home Directories

Distributed Logging

Useful Scripts


kill-procs is a script for managing usage of machines. It was primarily developed for usage on machines in a university computer lab. This script has two main functions, first it kills X sessions of users who have left the computer unattended for a certain period of time (the default is 3.5 hours). Second the script kills various background processes that desktop environments have been known to leave behind when a user logs off. This script will also report on other background processes that it finds running when the owner is not logged into the machine.

The script is currently available in the diskless overlay, you should be able to install it with emerge kill-procs. There are a few configuration settings at the top of the file.


I have a custom bashrc that does a few things automatically. Such as running "diskless fix" and making sure that dispatch-conf is patched properly to handle symlinks in /etc (the default version doesn't handle this properly). This is a little dirty, but it does cut down on maintenance a bit. Here is the current version:

File: /etc/portage/bashrc
patch_dispatchconf() {
        local dry="--dry-run --quiet" patch=1
        while [[ "${patch}" ]] &&  patch -p0 -t ${dry} <<EOF
--- /usr/lib/portage/bin/dispatch-conf.orig    2006-09-28 11:10:33.000000000 -0400
+++ /usr/lib/portage/bin/dispatch-conf  2006-09-28 11:11:01.000000000 -0400
@@ -20,7 +20,7 @@
 import portage, dispatch_conf
 from portage_exec import find_binary
-FIND_EXTANT_CONFIGS  = "find '%s' %s -iname '._cfg????_%s'"
+FIND_EXTANT_CONFIGS  = "find -L '%s' %s -iname '._cfg????_%s' -exec readlink -q -f '{}' ';' | sort | uniq | sed -e 's://:/:g'"
 DIFF_CONTENTS        = 'diff -Nu %s %s'
 DIFF_CVS_INTERP      = 'diff -Nu %s %s | grep "^[+-][^+-]" | grep -v "# .Header:.*"'
 DIFF_WSCOMMENTS      = 'diff -Nu %s %s | grep "^[+-][^+-]" | grep -v "^[-+]#" | grep -v "^[-+][:space:]*$"'
                if [[ "${dry}" ]]; then
                        unset dry
                        unset patch

post_pkg_postinst() {
        [[ "${PN}" == "portage" ]] && patch_dispatchconf
        /usr/sbin/diskless autofix



Miscellaneous Extras

Trouble shooting

Retrieved from ""

Last modified: Tue, 23 Sep 2008 04:07:00 +0000 Hits: 13,718