Copyright Randall Munroe.
Firefox is a big project and takes quite some time to compile. If you’re working on such a large project, you make a change, recompile, accidentally touch a header, recompile then lose a lot of time waiting and resort to checking social media or bouts of wheelie-chair jousting.
Within my home office, I’ve set up icecc and ccache on Linux Mint (similar to Ubuntu) on amd64 using GCC. I haven’t yet tried clang but probably will soon, instructions should be similar for other OSs and compilers, but YMMV. I’m going to give instructions for setting up both tools at the same time, they can be used independently but if want to compile large C/C++ projects often, you probably want both of them.
icecc (pronounced ice cream) is a tool to distribute c-compiler jobs among a network of peers for parallel compilation. Think make -j but across multiple computers. You may have heard of distcc, it’s like that but smarter at scheduling jobs. Use icecc (or distcc if that’s your thing) if you have some spare (or used) computers to distribute compilation across. Some Mozillians (not sure I like that word yet, labels etc) setup icecc groups within the Mozilla offices. I’m told from someone who tried that it’s not worth connecting to these from home. I’m also avoiding WiFi for this reason and more. Also on connecting to other networks, be aware this could be a security issue, a peer could replace your code with something nasty and trick you into running it.
ccache is a c compiler cache. If you’re recompiling the same project often this ccache will remember the .o file generated previously and return it rather than running the compiler again.
sudo aptitude install ccache icecc
And on your workstation you can also install icecc-monitor
icecc uses two daemons, iceccd accepts jobs and runs them locally on each node, it connects to icecc-scheduler which manages the jobs for a group of machines and distributes them. I believe it’s supposed to work if you have multiple icecc-schedulers on your network, but I found that this would easily create two separate smaller clusters as my laptop came and went from the network. Instead it was simple to disable icecc-scheduler on all but one of the nodes.
sudo update-rc.d icecc-scheduler disable sudo service icecc-scheduler stop
icecc-monitor is a GUI application that will let me visualise the cluster and its jobs. It was useful at this point to confirm that things were working, start it as part of your usual desktop environment. You should see the nodes of your cluster sitting idle. In the image you can see my two nodes, "fluorine" and "oxygen", I will be adding "neon" soon and have 16 cores/threads at my disposal.
If you’re not seeing this then I’ve found that:
-
You may have to disable your firewall, I use a firewall on my laptop when I’m out-and-about but find I have to turn it off to use icecc in my home office.
-
Restart the iceccd daemons to get them to connect to the scheduler.
-
Close and reopen icemon any time you restart the scheduler.
We could use icecc on its own, it’s as simple as adding /usr/lib/icecc/bin to your path. Instead of doing that we’ll add ccache. ccache likes a lot of hard disk space, 15-20GB is suitable if you’re working on Firefox, which usually uses about 5GB per build (reported by shu, I didn’t measure myself). I use btrfs with which I like to use snapshots, but there’s no point snapshotting my ccache, instead I created a new logical volume, used ext4 (remember to use noatime or relatime in your mount options (for any FS)) and mounted that at /mnt/ccache, depending on how your system is configured these steps could be quite different, or you might not use a separate filesystem at all. (I wish installers would let me name the volume group.)
Make the filesystem:
sudo lvcreate -L 20G -n ccache mint-vg sudo mkfs.ext4 /dev/mint-vg/ccache
Put this in /etc/fstab:
/dev/mapper/mint--vg-ccache /mnt/ccache ext4 errors=remount-ro,noatime 0 2
Mount the filesystem a and make one directory per user in it, that’s probably just one directory:
sudo mkdir /mnt/ccache sudo mount /mnt/ccache sudo mkdir /mnt/ccache/paul sudo chown paul:paul /mnt/ccache/paul
And put this in your user’s ~/.ccache/ccache.conf, set the size here, and the filesystem size appropriately. Most filesystems run more smoothly with some free space, your SSD may be happier with some free space too.
max_size = 17G cache_dir = /mnt/ccache/paul
One more thing, tell ccache to use icecc to run the compiler. I put this in my ~/.bashrc.
export CCACHE_PREFIX=icecc
Time to test it out. I’m not sure how this works generally, but for SpiderMonkey (JS shell only) builds you simply add --with-ccache to your ./configure arguments, then build with make -j12 (since I have 12 cores/threads in my two machines). For Firefox itself a build is normally configured by placing a mozconfig file in the project root directory. Add to that file:
ac_add_options --with-ccache=/usr/bin/ccache mk_add_options MOZ_MAKE_FLAGS="-j12"
I haven’t measured the effects of using either ccache or icecc, but I’ve definitely noticed that ccache can speed up repeated builds. I also suspect that some parallel slackness (adding more tasks than there are cores) could help speed things up to cover some latency introduced by ccache.
The icecc-monitor program has a number of different views. The image above was "Star view" I think my favorite is "Gantt view" (below).
Update 2017-08-17
Most of ccache’s options can be controlled by either configuration options in ~/.ccache/ccache.conf or by environment variables. Therefore I have removed CCACHE_PREFIX from my ~/.bashrc file and instead added it to ccache.conf.
I have also learnt that when -g (or similar) is on the command line ccache will hash the directory name (I think the working directory) and incorporate that into its cache. This ensures that any path references in debugging symbols resolve correctly and don’t mislead you during a debugging session. Which is a good idea, but if most of your builds use -g and you use multiple workspaces for the same projects it can lead to more cache misses. If you don’t use a debugger often, and promise set to CCACHE_NOHASHDIR when you do (or be confused by references to different source files), then this can be disabled with the hash_dir option.
Now my ccache.conf looks like
max_size = 17G cache_dir = /mnt/ccache/paul prefix_command=icecc # Might get confusing for debugging hash_dir = false