Search:  
Gentoo Wiki

TIP_Providing_binary_packages

This article is part of the Tips & Tricks series.
Terminals / Shells Network X Window System Portage System Filesystems Kernel Other

Contents

Introduction

If you are running a couple of Gentoo boxes in your LAN or WAN, it might come handy to use a "master" Gentoo box, which provides the portage tree and (with it) pre-compiled binary packages. The "client" Gentoo boxes then only need to fetch and use these binary packages. Luckily, the binary packages can be distributed by rsync when running "emerge --sync" on the client boxes.

The upside of using binary packages is that merging a binary package only takes a couple of seconds compared to compiling the ebuild from source, which can take a long time for complex ebuilds.

The downside is that you are restricted to the binary packages provided by the master box, which might provide only parts of the needed ebuilds as binary packages. Another thing is that you are restricted to the USE flags that were used when the packages were compiled.

What needs to be done

Things to do on the master box

Building packages

On the master box, you should use the following setting in /etc/make.conf:

FEATURES="buildpkg <other_features>"

This means that each and every ebuild compiled and merged on the master box will create a corresponding package in /usr/portage/packages or the value of the PKGDIR variable in /etc/make.conf. This hardly takes longer than just compiling and merging the ebuild. Alternatively, you can use emerge's -b option every time you compile and merge an ebuild.

You can also only build a package without merging it into the system by using the -B option, but this implies that all dependencies have been merged already.

You might also consider using "quickpkg" (see its manpage) to create a binary package from an already installed ebuild on your system. This is quick, but has the drawback that it includes all locally made file modifications after the ebuild has been merged, which may not be what you want.

The following script can be used to create binary packages from all installed ebuilds on the system:

 #!/bin/bash
 packages=`/usr/bin/find /var/db/pkg -mindepth 2 -maxdepth 2 | /bin/sed 's/\/var\/db\/pkg\///' | sort`
 for p in ${packages}; do
   "/usr/sbin/quickpkg" "--include-config=y" "=${p}"
 done
Warning: Using --include-config=y for publicly distributed packages is a bad idea. Configuration files frequently contain information such as passwords which you do not want others to know. Using this option includes the configuration files as they exist on your installation. ("--include-unmodified-config=y" might be a better option.)

Providing your portage tree and binary packages via rsyncd

In order to provide your portage tree and your compiled binary packages, you need rsync (should be installed; if not, simply run "emerge rsync").

Then edit your /etc/rsyncd.conf as follows:

# This line is required by the /etc/init.d/rsyncd script
motd file = /etc/rsyncd.motd
pid file = /var/run/rsyncd.pid
use chroot = yes
read only = yes
max connections = 25
uid = nobody
gid = nobody
hosts allow = 192.168.100.0/24
hosts deny = *
 
# Simple example for enabling your own local rsync server
[gentoo-portage]
        path = /usr/portage
        comment = Gentoo Linux Portage tree
        exclude = /local /distfiles

Edit "hosts allow" to match the clients which may connect and create the MOTD file /etc/rsyncd.motd. Note that /packages is not excluded. This is important for fetching the binary packages via rsync.

Next, start rsyncd:

# /etc/init.d/rsyncd start

In order to start rsyncd automatically each time the master box boots, use

# rc-update add rsyncd default

Updating your portage tree once a day

Place the following lines into your /etc/crontab in order to automatically sync once a day on your master box:

# sync gentoo-portage
2 8 * * * root   /etc/init.d/rsyncd stop;emerge --sync;/etc/init.d/rsyncd start

Use whatever time you like instead of 8:02 a.m. Stopping rsyncd is a safety measure to prevent clients from syncing while the master box syncs, which could lead to unwanted results.

Cleaning up your packages directory

You should clean up your packages directory every once in a while. If you don't, /etc/portage/packages will grow endlessly on your master box and on your client boxes, too! You can use the Perl script find_pkg_dups.pl to clean up your packages. Of course, the script only needs to be run on your master box. rsync then handles the deletion on your client boxes automatically.

The eclean command found in the gentoolkit package is probably a better way to handle this.

Things to do on the client boxes

Using your master box as a sync source

The client boxes must use your master box for rsyncing. Just use the following setting in /etc/make.conf:

SYNC="rsync://<yourmasterbox>/gentoo-portage"
RSYNC_EXCLUDEFROM="/etc/portage/rsync_excludes"

For newer versions of Portage use the following in place of RSYNC_EXCLUDEFROM in make.conf or the /packages directory will not get synced to your clients. It does give a warning about it.

PORTAGE_RSYNC_EXTRA_OPTS="--exclude-from=/etc/portage/rsync_excludes"

Create /etc/portage/rsync_excludes with the following contents:

!
/local
/distfiles

Now execute "emerge --sync" on a client box. It should fetch the portage tree from your master box, including the directory /usr/portage/packages.

If you have set up your master box to sync once a day but want your client box to be able to sync more often than that (e.g., because you have new binary packages), emerge will normally complain like this:

>>>
>>> Timestamps on the server and in the local repository are the same.
>>> Cancelling all further sync action. You are already up to date.
>>>

In this case, simply do the following on your client box:

# rm -f /usr/portage/metadata/timestamp*
# emerge --sync

Using the binary packages

In order to install the pre-compiled binary packages on your client boxes, you must use the emerge option -k. This option will use binary packages, if available, and will pull and compile the sources otherwise. So instead of using "emerge -uD world" to update your client boxes, you must use "emerge -kuD world".

If you want to see the available binary packages for updating your client boxes, you can run "emerge -pkuD world". This will show you for each package whether an ebuild or a binary package will be used for upgrading.

Tips

Using the same proxy server

If your master box doesn't provide all needed packages in binary form and so some packages need to be compiled on your client boxes, it is a good idea to make all client boxes and your master box use the same local proxy server. This will save some bandwidth when source files are (re)downloaded. To make it work properly, all boxes should also use the same Gentoo mirrors in /etc/make.conf in the same order.

Simply put the following lines into /etc/make.conf on all your boxes:

GENTOO_MIRRORS="http://linux.rz.ruhr-uni-bochum.de/download/gentoo-mirror/ http://mirrors.sec.informatik.tu-darmstadt.de/gentoo/ ..."
http_proxy="http://yourproxy:yourproxyport"
ftp_proxy="http://yourproxy:yourproxyport"

Edit your list of Gentoo mirrors and the proxy server to use as desired.

Retrieved from "http://www.gentoo-wiki.info/TIP_Providing_binary_packages"

Last modified: Thu, 07 Aug 2008 17:24:00 +0000 Hits: 20,892