[Nix-dev] make -j -l concurrency patch

[Marc Weber]

Hi list,

I implemented the proposal.

Details:

There is a new Nix internal global var called num_cores defaulting to
  1) num_cores (nix.conf)
  2) NUM_CORES (env var)
  3) 1

So if you set num_cores=1 in nix.conf nothing changes.

When the environment for the build job is built NUM_CORES is set to
1 when the derivation sets NUM_CORES=1. This way derivations can opt-out 
if they don't support parallel builds.

NUM_CORES is set to the Nix internal global var num_cores otherwise.

If NUM_CORES > 1 the -j and -l options are appended to makeFlags
automatically so that we can take advantage of the new feature in trunk
without changing builders [1].

-l is set to NUM_CORES + 1
-j is set to 2 * NUM_CORES

If load is 10 for a while make will stop starting jobs. Because it takes
some time until load drops below NUM_CORES during this period not all
cores will be used. That's why I target a load of NUM_CORES + 1

Now you can think that your browser has not enough CPU to run because
CPU is overloaded by 1. This should be addressed by niceness so that
your browser *always* gets cpu and builders only get CPU time assigned
if no other more important code wants to run.

Questions:

- Do we want to opt-out for make build and make install?

  INSTALL_IN_PARALLEL=false
  but
  BUILD_IN_PARALLEL=true

- should opt-out affect the hash?
  pro changing hash:
    If a build is not concurrency safe after adding NUM_CORES=1 or the
    like you will have a working packages again.

    This is less important if make all fails. But it is important if
    make install -jX does a bad job without failing.
    In this case we would have bad packages in store

  pro keeping hash:
    less rebuilds

  I think that having bad packages is a major annoyance. So I think hash
  should change if opt-out is used on packages.


Should I have been using string2int instead of ati? null is not passed,
if atoi returns 0 it num_cores will be set to minimum of 1 so it should
be safe.

I'm aware that this patch does not force perfect utilization and control
over CPU. However it should be much better than it is now.

Eg if you have different background jobs: nice -n 20
./my-long-calculation this can prevent make spawning more jobs if the
build is more important to you than your long-calculation.

TODO:
 - ensure NUM_CORES can be set to 1 only in derivations when calculating
   the hash rather than when building the env for the build job ?
 - write the documentation for this patch.

Enjoy!

Marc Weber

[1]: The kernel build script is using makeFlags=".." ignoring the passed
value. This must be fixed.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: nix-make-j-l.patch
Type: application/octet-stream
Size: 4243 bytes
Desc: not available
Url : http://lists.science.uu.nl/pipermail/nix-dev/attachments/20100531/9f09bb95/attachment.obj 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-pass-NUM_CORES-to-the-Nix-daemon.-You-can-override-t.patch
Type: application/octet-stream
Size: 1034 bytes
Desc: not available
Url : http://lists.science.uu.nl/pipermail/nix-dev/attachments/20100531/9f09bb95/attachment-0001.obj