Search:  
Gentoo Wiki

HOWTO_Distcc_over_SSH_with_Portage

Image:OpenSSH-logo.png

SSH Basics

Tips & Tricks

Other Gentoo-wiki SSH

edit

What will we be doing, and why?

distcc allows you to distribute a compilation over several machines, significantly reducing build times. There is already an official Gentoo distcc guide, but it only covers how to set up distcc using its own daemon, distccd. Sometimes, you have to use distcc over SSH instead, perhaps because of restrictive firewalls, or because you just don't want another daemon listening to the network. This guide covers how to configure distcc and Portage to get your Emerge jobs distributed by distcc using SSH.

Requirements

Setting up distcc

Set up distcc user

The distcc ebuild creates a distcc user, but it cannot log in, as is standard secure practice for daemon accounts. We will now change that.

First, the distcc user needs a home directory. This should be done on all nodes.

# mkdir -p /etc/distcc/.ssh
# usermod -d /etc/distcc distcc

On the compilation nodes, it also needs a valid shell:

# usermod -s /bin/bash distcc

Then we set up our ssh keys, since distcc can't supply passwords at login. Do this on the front end machine.

# ssh-keygen -t dsa -f /etc/distcc/.ssh/id_dsa

When ssh-keygen asks for a passphrase, just hit enter so it sets none.

Now, we need to distribute distcc's public key to all compilation nodes.

Method 1

# scp /etc/distcc/.ssh/id_dsa.pub <compilation-node>:/etc/distcc/.ssh/authorized_keys

Method 2

You should append each new host to the old one...

# scp /etc/distcc/.ssh/id_dsa.pub <compilation-node>:/root/id_dsa.pub
# cat `/root/id_dsa.pub` >> /etc/distcc/.ssh/authorized_keys
# rm /root/id_dsa.pub

Maybe would be done in a more elegant way :)

Method 3

A more elegant way

# for i in <compilation-nodes>; do cat \
/etc/distcc/.ssh/id_dsa.pub | ssh $i \
"cat >> /etc/distcc/.ssh/authorized_keys"; done

Fix ssh permissions

Make sure ssh is happy with all permissions.

On all nodes,

# chown -R distcc:daemon /etc/distcc

On the front end node only,

# chown portage:portage /etc/distcc/.ssh/id_dsa
# chmod 600 /etc/distcc/.ssh/id_dsa
# chmod 644 /etc/distcc/.ssh/id_dsa.pub

On compilation nodes only,

# chmod 644 /etc/distcc/.ssh/authorized_keys

Test ssh

Try to log in as the distcc user from your front end machine to a compilation node using the just-created keys:

# ssh -i /etc/distcc/.ssh/id_dsa distcc@<compilation-node>

If you get a prompt for a password, check syslog on the compilation node to see why sshd didn't like the ssh key. One possible reason could be that the ssh server does not allow empty passwords. Make sure that you set "PermitEmptyPasswords yes" in /etc/ssh/sshd_config on the compilation nodes.

If that fails and shows something like this:

[sshd] User distcc not allowed because account is locked

then execute

# passwd -u distcc

to unlock this account.

If distcc was able to log in, then you need to collect the public ssh host keys of all compilation nodes so distcc doesn't get stuck waiting for you to confirm each host identity. A simple way to do this is to use ssh-keyscan:

# ssh-keyscan -t rsa <compilation-node-1> <compilation-node-2> [...] > /var/tmp/portage/.ssh/known_hosts
# chown portage:portage /var/tmp/portage/.ssh/known_hosts

You should then manually verify each host key, perhaps by logging in at the physical console of each machine and running ssh-keyscan locally. Your paranoia may vary.

Making a wrapper for ssh

Unfortunately, distcc can't supply arguments to ssh, so we need a wrapper script it can call that supplies the correct arguments. Point your favorite editor to /etc/distcc/distcc-ssh and enter:

#!/bin/bash
exec /usr/bin/ssh -i /etc/distcc/.ssh/id_dsa "$@"

and make sure the file is executable

# chmod a+x /etc/distcc/distcc-ssh

Setting up Portage

distcc is controlled by a couple of environment variables. One place to set such variables so that they will only be used for Portage is in /etc/make.conf. Add the following lines at the bottom:

DISTCC_SSH="/etc/distcc/distcc-ssh"
DISTCC_HOSTS="localhost/2 distcc@<compilation-node>/<num-parallel-jobs> [...]"

(see the distcc manual page for the syntax of the DISTCC_HOSTS line).

To take advantage of all your new compilation power, you need to run a lot of jobs in parallel. Find the MAKEOPTS line, still in /etc/make.conf, and change the number after "-j" to the sum of the allowed number of jobs on each node, e.g.

MAKEOPTS="-j6"

for three machines with two jobs each.

Lastly, we need to notify portage that we will be using this feature so also add the following line below:

FEATURES="distcc"

Test drive

That should be it. To watch it in action, start an emerge on the front end and watch top on a compilation node - you should see some gcc processes owned by the distcc user. If it doesn't seem to work, try setting

DISTCC_VERBOSE="1"

in /etc/make.conf on the front end and see if you get any informative messages.

TODO

This procedure might be improved by using keychain


Response: The security of this approach is based on your ability to control access to the keys and access to the computer which has permission to login. Anyone who has an account on the host computer could potentially gain access to the client computer(s). Other than that, this approach should be sufficiently secure. You can add another layer of security by using IPTABLES to limit access based on the computer's ip address or network. You can also disable the login account whenever it is not in use. You could also achieve a further layer of security by creating the client computer as a special purpose (untrusted) virtual machine, that way even if compromised the attacker wouldn't be able to do much. But such extreme measures aren't really needed; on the other hand, the isolation provided by a virtual computer could have other benefits.

A very different approach is to use ssh instead of NFS with the method shown here. HOWTO Emerge on very slow systems.

Sharing SSH keys

Command to run for sharing SSH keys on 2 linux servers to login without authentication

Retrieved from "http://www.gentoo-wiki.info/HOWTO_Distcc_over_SSH_with_Portage"

Last modified: Fri, 06 Jun 2008 08:23:00 +0000 Hits: 22,880