[Nix-dev] make -jx within builds?

[Eelco Dolstra]

Hi,

On 05/23/2010 06:29 PM, Marc Weber wrote:

> I'd like to run make -j4 within a job. Some packages such as qt take
> very long. So you have four cores and 3 are idle.

How about the following solution, which is pure as long as makefiles are
concurrency-safe.  The builder should do:

  nrCores=$(cat /proc/cpuinfo | grep 'processor.*:' | wc -l)

  makeFlags="$makeFlags -j -l $nrCores"

This allows GNU make to start an unlimited number of parallel builds, as long as
the load is less than the number of cores.  It has the advantage of playing nice
with Nix's own parallel builds.  (If we used Nix's `-j <N>' and Make's `-j <M>',
we could end up with N * M parallel builds.  This would be particularly bad for
Hydra.)  If the GNU make manpage is correct, then Make will always run at least
one build in parallel regardless of load, so there should be no starvation.

We can try this out in the trunk for a few packages by putting it in a setup
hook, e.g.

  buildInParallel = makeSetupHook ...
    ''
      nrCores=$(cat /proc/cpuinfo | grep 'processor.*:' | wc -l)
      makeFlags="$makeFlags -j -l $nrCores"
    '';

and then adding it to the buildInputs of selected packages (e.g. the kernel or KDE):

  buildInputs = [ buildInParallel ... ];

Eventually we could merge it into stdenv and possibly make it the default behaviour.

-- 
Eelco Dolstra | http://www.st.ewi.tudelft.nl/~dolstra/