[Nix-dev] Re: make -j<n> howto?

[Eelco Dolstra]

Hi,

Peter Simons wrote:

>  > Using -j x could lead to different builds (doesn't in most cases though).
> 
> Um, what exactly do you mean? Are you saying that nix doesn't utilize
> multi-processor machines because there is some inherent design problem that
> would lead to broken packages if multiple CPUs are being used during the
> build?

Indeed, but the design problem is with make and makefiles.

>  > The only way to do this is adding that option to the builder.sh file or
>  > such. (default buildPhase)
> 
> Does anyone agree with me when I say that this situation is unsatisfactory?

Yes, it's certainly unsatisfactory, but there is no obvious solution.  However, 
the situation isn't all that bad: Nix has its own "-j" option to perform 
multiple builds in parallel.  So if you're (say) rebuilding all of Nixpkgs from 
scratch, you do get a huge amount of parallelism.  And contrary to Make's "-j" 
this is guaranteed to be safe since builds don't interfere with each other.

Of course, this is parallelism between builds, not *within* builds, so if you're 
rebuilding OpenOffice it doesn't help you :-(

>  > It would be interesting to have an experimental maybe unpure use numcores
>  > environment option which would be 1 in all production cases (buildfarm)
>  > but might be set to another value on your local developement machine.
> 
> What exactly makes a parallel build "unpure"? Are you referring to broken
> makefiles that contain race conditions and thus fail when make executes
> build jobs concurrently?

Exactly.  There is no way to guarantee that a parallel Make produces the same 
result as a sequential Make.  But I have no idea how many packages have broken 
makefiles (my guess is a lot).

OTOH, I'm mostly worried about race conditions that don't cause the build to 
fail but produce a different result, and that's probably very rare.  So turning 
on "-j" globally might be acceptable, but some empirical results would be needed :-)

(Undoubtedly Gentoo people regularly build packages with specific -j flags, so 
they probably know how much breakage it causes...)

-- 
Eelco Dolstra | http://www.st.ewi.tudelft.nl/~dolstra/