[Nix-dev] Re: How do parallel builds scale?

[Peter Simons]

Hi Ludovic,

 >   http://hubble.gforge.inria.fr/parallel-builds.html
 >   http://hubble.gforge.inria.fr/parallel-build-details.html

these results are very interesting -- thank you for sharing that data.


 > In short, package build scale very poorly, and more than half of the
 > packages didn’t scale beyond 2.

The majority of open source software is built with Automake, and Automake
is not particularly good at utilizing several cores. The most significant
problem is that the common setting

  SUBDIRS = foo bar

causes these two directories to be built *sequentially*. Automake won't
even try to build 'bar' unless 'foo' is complete. There are historical
reasons for that behavior, of course, but it's a really bad choice as far
as parallel building is concerned. 

A build system that is superior in that regard is Boost.Jam, and indeed
your numbers show that the Boost libraries benefit a *lot* from having
several cores available.

Take care,
Peter