Search:  
Gentoo Wiki

Ccache

This article is part of the Tips & Tricks series.
Terminals / Shells Network X Window System Portage System Filesystems Kernel Other

Contents

Introduction

ccache is a compiler cache. It uses the GCC -E switch and a hash to detect when a compilation can be satisfied from cache. The effect is that packages frequently compile 5-10 times faster than they would otherwise.

The first time that you emerge a package after setting up ccache, there will be a very slight increase in its compilation time, but thereafter (when the compilation is cached), you will notice significant reductions. However, remember that a compiler cache can only be used from the same source to see speed benefits.

Installation

Usually after you emerged ccache and added it to your make.conf (see later in this article), ccache should work. You can emerge something and check 'CCACHE_DIR="/var/tmp/ccache" ccache -s' if the stats update. If not, you can try to install ccache manually:

  1. Make sure your portage version is higher than 2.0.46-r11: emerge --version
  2. Emerge ccache: emerge -av dev-util/ccache
  3. Find your CHOST (see below): emerge --info | grep CHOST
  4. Copy the CHOST value (in double quotes) into this command: ccache-config --install-links YOUR-CHOST-HERE
Code: Installing and configuring ccache (/bin/bash)
# emerge --version
Portage 2.1.1-r2 (default-linux/x86/2006.1, gcc-4.1.1, glib-2.4-r3, 2.6.18-gentoo-r5 i686)
# emerge -av dev-util/ccache
...
Would you like to merge these packages? [Yes/No] y
...
# emerge --info | grep CHOST
CHOST="i686-pc-linux-gnu"
# ccache-config --install-links i686-pc-linux-gnu

File Placement

When emerging ccache as root the following ccache files and folders are installed:

When emerging ccache as a user, another folder is additionally created:

Solid information on this structure has proven difficult to come by and anyone who can provide an authorative answer should edit this page to do so.

However it seems that /var/tmp/ccache is unique to Gentoo, and should be chosen over /root/.ccache to store the cache.

Note: I could not get ccache to work properly until I created a symlink, it seems to be getting confused by the existence of two separate directory locations.
      mv /root/.ccache  /root/snafu.ccache
      ln -s /var/tmp/ccache  /root/.ccache
  

After that everything worked. I have been using it for awhile now (including emerge world) with no problems. (codeslinger.comspalot)

 

Configuration

Note: The instructions in ccache's man page to create symlinks in the ccache directory are not applicable to gentoo users. This is because the ccache-config --install-links step above creates the links. The steps below are only a verification, you should find these links already created in: /usr/lib/ccache/bin


Code: /bin/bash
$ ls -l /usr/lib/ccache/bin
...
lrwxrwxrwx 1 root root 15 Dec 27 c++ -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Dec 27 cc -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Dec 27 g++ -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Dec 27 gcc -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Dec 27 i686-pc-linux-gnu-c++ -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Dec 27 i686-pc-linux-gnu-g++ -> /usr/bin/ccache
lrwxrwxrwx 1 root root 15 Dec 27 i686-pc-linux-gnu-gcc -> /usr/bin/ccache
...

All the files are links pointing to: /usr/bin/ccache

When portage seeks to compile a package, it of course looks for the relevant compiler. However we don't want portage to go direct to the compiler, rather we want the compiler accessed through ccache. Thus we have a number of symlinks in the ccache directory which bear names that correspond to the various c and c++ compilers that portage looks for. With ccache enabled, Portage follows the path to /usr/lib/ccache/bin, and thus one of these files is invoked instead of one of the actual compilers. As you will see from the box above, all of these files are merely links to the ccache executable, (/usr/bin/ccache), which itself will call the genuine compilers.


Configuration takes place within /etc/make.conf

File: /etc/make.conf
...
FEATURES="ccache"
CCACHE_DIR="/var/tmp/ccache"
CCACHE_SIZE="2G"
...
  1. Most importantly we need /etc/make.conf to inform portage to look for and use ccache. This is achieved by the first of the three lines above. If you already have some features set, naturally just add "ccache" to the list.
    • If you also use DistCC, be sure you add the "ccache" option first and "distcc" second in your features list to take full advantage of both.
  2. The CCACHE_DIR environment variable specifies where ccache will keep its cached compiler output.
  3. The 3rd line sets the maximum space that is allocated to caching compilations. At the time of writing the default maximum size is 976.6M. If you can afford the space you may well want to allow for more as in the example.
    • The ccache stats won't update to show your chosen CACHE_SIZE until after the emerge that follows your change in specification.
    • You can also set the cache size by employing the -M option described on ccache's man page. However, if you have a CCACHE_SIZE value set in /etc/make.conf, it will overwite previously set values from as soon as you next emerge.

Execution

Now, for the purpose of testing, emerge something small...

To ensure that ccache was running, pull up the ccache stats afterwards with: ccache -s

If you're using the CCACHE_DIR in /var/tmp/ccache you need to add that variable to the command string as shown below.

Code: /bin/bash
CCACHE_DIR="/var/tmp/ccache" ccache -s
/* The output will look something like this if it's working */
cache hit                            189
cache miss                          1983
called for link                      159
multiple source files                  4
compile failed                        52
preprocessor error                     4
bad compiler arguments                 3
not a C/C++ file                     141
autoconf compile/link                357
unsupported compiler option           30
no input file                        145
files in cache                      3966
cache size                          30.1 Mbytes
max cache size                       2.0 Gbytes

If no space shows as used, your cache directory is still empty and it might be the case that ccache (when run direct rather than by Portage) is looking for the cache store in the wrong place. You can overcome this by declaring the location of the cache store system-wide. Update your profile to include the second line that we added to /etc/make.conf as follows:

Code: /bin/bash
echo "CCACHE_DIR=\"/var/tmp/ccache\"" >> /etc/env.d/99local 
env-update && source /etc/profile

Troubleshooting

Note: It seems like using ccache to compile the kernel can cause some problems. At least the DVB drivers seem to not compile correct. My av7110 module didn't work cause of unresolved symbols. After compiling without ccache everything worked fine.

ccache and Genkernel

Genkernel doesn't use ccache by default, even when you've enabled it for root. You can easily fix this by appending the --kernel-cc option to your genkernel call:

Code: /bin/bash
$ genkernel --kernel-cc=/usr/lib/ccache/bin/gcc --menuconfig all

You can add following alias (see TIP_alias) to your profiles, and later use genkernel as usually:

Code: /bin/bash
$ alias genkernel='genkernel --kernel-cc=/usr/lib/ccache/bin/gcc'

ccache and non-portage compiling

(This section has been updated in the documentation. Old method of editing /etc/env.d/ does not work anymore, probably never has: you can rm /etc/env.d/02ccache)

To ensure all compilation uses ccache, the directory /usr/lib/ccache/bin has to precede /usr/bin in your PATH/ROOTPATH environment variable. This has to be done after /etc/profile.env gets called, so changing anything in /etc/env.d will not work.


Note that this does not affect genkernel; to force genkernel to use ccache it is still necessary to specify the --kernel-cc option to your genkernel call as above.

Method A

You can accomplish this for a given user by adding this line to his .bash_profile or .bashrc:

PATH="/usr/lib/ccache/bin:${PATH}"

or for all users including root by appending to the end of your /etc/profile:

PATH="/usr/lib/ccache/bin:${PATH}"
ROOTPATH="/usr/lib/ccache/bin:${PATH}"

As always after updating environment variables, run env-update if you changed /etc/env.d entries to refresh your /etc/profile.env, source /etc/profile to apply the changes to your current shell.

Method B

You can put a file called ccache.sh within /etc/profile.d, the content of the file should be

if [ -d /usr/lib/ccache/bin ] ; then
    PATH="/usr/lib/ccache/bin:${PATH}"
    export PATH
fi

because there is a line at the end of /etc/profile,

for sh in /etc/profile.d/*.sh ; do
    if [ -f "$sh" ] ; then
        . "$sh"
    fi
done
unset sh

so this should do the trick.

ccache adjusts directory permissions

Apparently ccache will sometimes try to set the permissions of the ccache_dir after and before each emerge. Type this to fix that:

Code: /bin/bash
# chmod g+X -R /usr/lib/ccache

Maybe you can automate this by setting the variable CCACHE_UMASK to 777 (be carefull!).

ccache & userpriv


Links

From Gentoo Linux Documentation:

Others:

Retrieved from "http://www.gentoo-wiki.info/Ccache"

Last modified: Thu, 14 Aug 2008 11:46:00 +0000 Hits: 79,976