Best way to speed up compile/build times.

[dsmithfvc]

I've taken on the roll as a Devops Engineer for the first time working with Android and the imx6.  Currently we run Jenkins and have builds for things like MFG-Tool, MATS, new dev changes.  Our Git repository is 5Gig and I'm told 300 million lines of code.   I'm just getting started trying to dig in and speed up compile times.  Our builds take a very longtime.  Can someone recommend where a beginner at Android should focus in this area?  Does it make sense to compile everything for a MFG tool or because of small code changes checked into Git?  Is there a best path to getting incremental builds, such as, Gradle or Jenkins incremental builds?  Sorry if this is too basic. I've only been digging a few hours so, I'm still kind of lost.  Just hoping someone may have insight from learned experiences and can point me in a direction to faster build times.

[patrickwood]

Only 5GB?  You must have the android sources installed yet.  They're a good 40GB or so.

I've been building android from the sources for a couple of years now.  My experience has been that you need the biggest, fastest server you can get your hands on with fast SSD or a SATA 3 RAID array.  Android build time (from scratch) on a 2.8GHz Core i7 with 6GB of RAM and SATA 3 rotating hard drive runs around 45 mins.  Build time on a 2GHz Core i7 with 16GB of RAM and a SATA 3 SSD drive runs around 30 minutes.

An incremental build of the entire android tree can take just a minute or two on a really fast system (1:08 on my Core i7 with the SSD drive, and the build system does allow you to build individual subsystems, so if most of the work in going on in customizing uboot or the kernel, you're in really good shape.  On the other hand, if most of the work is on the android components, you're looking at potentially long build times, depending on the amount of changes.

The entire android build tree runs under make; it's huge, and you won't want to try to swap it out for something else.  Even if you tried, the amount of regression testing needed to verify that it works properly would be prohibitive.

YMMV.

[dsmithfvc]

Well the reason it is only 5GB I believe is due to stripping out all the unneeded packages.  The builds were taking about an hour on an 8 core i7.  Last night I moved Jenkins to a Dual Xeon with 10 cores each (40 threads in total).   I changed make -j 8 to make -j 80 and now the builds take 17 minutes vs. 56 minutes before.  I'd still like to do incremental builds however, I read the old Android Build System (using make) is not good at that. 

I found out the reason we don't use Gradle is mostly because in order to do full builds with C++.  I'm working with someone now to build pieces with make files which will help out.  I have RAID 1+0 with 10k RPM drives so, not sure how big a difference SSD will really make since CPU seems to be the main way to speed up.  Next I'm adding in CCACHE for full builds and I'll try to report back here how much that helps.

Thanks for the reply.

[patrickwood]

If you're down to 5GB for the entire Android tree, you should be building a lot faster than an hour on an 8-core i7.  Perhaps not using CCACHE is affecting your times; I forgot to mention that I'm using that as well with a 50GB limit (17G current usage).  Having lots of RAM helps with the ccache (as well as everything else).

Incremental builds in android are hit-or-miss.  I do mostly kernel work, and incremental builds there work just fine.  Also, changes affecting just one package should be fine.  It's when you start mucking around with system APIs that are exposed and used by lots lots of packages (e.g., notifications) that you'll see lots of rebuilding activity.  The simplest thing to do is experiment a bit by touching various files in different locations to see how that affects what's recompiled.

You can also build subsets via make targets, e.g.,

make bootimage

make systemimage

make platform

which could be done on separate build servers depending on how you do your unit testing.