[Nix-dev] make -jx within builds?

[Marc Weber]

Excerpts from David Brown's message of Fri May 28 16:38:58 +0200 2010:
> On Fri, May 28, 2010 at 03:04:07PM +0200, Marc Weber wrote:
> >>   buildInParallel = makeSetupHook ...
> >>     ''
> >>       nrCores=$(cat /proc/cpuinfo | grep 'processor.*:' | wc -l)
> >>       makeFlags="$makeFlags -j -l $nrCores"
> >
> >-j -l ? Agreed. This is superior.
> 
> I recommend still limiting the '-j', perhaps with the number of cores,
> or twice the number of cores.  The load average lags behind, and this
> can easily fire off dozens of compilations before the load catches up.
> If you're short on memory, this can really hurt.

So if you have 4 cores you recommend this:
make -j 8 -l 4

where -j is an upper bound ?

I also thought today about other platforms? Is cygwin still active? Does
the /proc interface exist there?

We can't make the kernel take care of this, can we?
Eg ask it to only run 1 out of four threads if cpus are busy (eg more
than one build are running)

If neither the coreutils tool nor the /proc interface are portable
enough we can make Nix define NUM_CORES or such which could be a nice
idea because you may want to have Nix use only some of your cpu cause
you're watching video or doing whatsoever.. or maybe just to keep
cores sleeping or such (does this make sense?)
If Nix defined NUM_CORES it could always pass 1 to disable this feature.
Why do I prefere NUM_CORES ? You can control these feature without
changing the builder too much then.

So I vote for Nix defining NUM_CORES and builders using

make -j $NUM_CORES -l $NUM_CORES or similar

Marc Weber