[Nix-dev] Accepting -jX on builds

[Eelco Dolstra]

Hi,

Lluís Batlle wrote:

> I remember that someone here said that the make "-jX" parameter should
> not be used at all, because some build systems produce (relevantly)
> different results according to that parameter.
> 
> I argued on freenode#nixos that we could add an option to
> mkDerivation, to allow derivation authors to assert, whether the pkg
> build system accepts make's "-jX" or not. Defaulting to "not", of
> course. I think this can be done passing an environment variable from
> the nix-build calling user space.
> 
> As derivation results can also be affected by data outside the nix
> store, and the only way to assert anything about that is running the
> build in a chrooted environment (we have a nix parameter to force
> that), we can have a similar parameter which disallows any kind of -jX
> to make.
> 
> I don't think we should even think about getting *different hashes*
> for different -jX parameters, when the mkDerivation parameters assert
> that the build allows any -jX.
> 
> Do you like this idea comparing "-jX" to "data outside the store"?

I used to be against this because the purity risk, but if it's turned *on* on a
per-package basis, it's probably an acceptable risk.  (We also have a stack of
8-core machines here that we should put to good use ;-)

As a matter of policy I would say we should only turn it on for larger packages
(say, OpenOffice, KDE, Firefox, that sort of thing) - Nix itself already
provides enough parallism *between* package builds that small packages are
rarely a problem.

We could do it like this:

- We add an option to nix.conf specifying the number of cores a build may use,
say "build-max-cores" (not really a good name, but you get the idea).  The
default would be 1.

- Nix passes this to builders through some environment variable, say NIX_MAX_CORES.

- If stdenv sees that $NIX_MAX_CORES > 1, it adds "-j $NIX_MAX_CORES" to $makeFlags.

PS: I have this wild idea of making a front end for Nix that reads a Makefile
and basically converts it to a Nix expression - that is, every Make rule becomes
a derivation.  Then you get all the parallelism you want and it would still be
pure.  Or conversely, if your Makefile isn't pure (e.g. if build actions
interfere with each other), it would deterministically fail.  As an added bonus,
you would never have to write another "make clean" rule :-)

-- 
Eelco Dolstra | http://www.st.ewi.tudelft.nl/~dolstra/