[Nix-dev] Re: Another approach at parallelizing in-build jobs

[Michael Raskin]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On 06/20/2010 12:06 AM, Peter Simons wrote:
> Hi Michael,
> 
>> The fear of opt-out mechanism is based on the fear of _few_ small rogue
>> packages that will be built incorrectly.
> 
> there seems to be a misunderstanding.
> 
> We both agree that an opt-out policy for parallel building is a bad
> choice. For any random package, the default assumption should be that
> parallel building is unsafe. Parallel building should be enabled only
> for those package for which we can be reasonably sure that parallel
> builds are supported, well tested, and reliable.
> 
> In other words, parallel builds should be performed using an explicit
> opt-in policy.

Technically, what I said in the previous letter partially explains that
some things may be missed even with opt-in. As you correctly noted,
finding a race condition in a GCC build is not an easy task, and some
people are afraid to be bitten by it - even if the risk is ridiculously
low. After all, GHC people cannot find the problem with -j8.

>  >> In fact, Makefiles don't even specify that order. Whether make
>  >> builds foo.o before bar.o or bar.o before foo.o is implementation
>  >> defined.
>  > Note - many people pretend that there is only one implementation of
>  > "make", namely "GNU Make". Once "GNU Make" and "BSD Make" do
>  > something in the same order, for many packages that means "make
>  > always does that".
>  > In this situation hoping that everyone keeps in mind every thing that
>  > can be implementation-dependent is too optimistic.
> Arguably, that problem doesn't exist in Nix, because Nix generally uses
> GNU Make, so there is only one de facto implementation. Of course,
> there's an odd chance that the implementation of GNU Make changes
> behavior in future versions, but that doesn't seem to be particularly
> likely.
> Anyway, the point I was trying to make is a slightly different one. I
> was trying to illustrate the basic principles that say that the exact
> build order of independent files is irrelevant.

My point is that "order is not defined" doesn't preclude people from
relying on GNU Make behavior being the only one possible. And sometimes
they only use -j1.

> Yes, you are right. Personally, I believe that parallel builds are not
> very important for small or obscure packages, simply because those
> packages are built rarely (or quickly) anyway. My main interest in
> parallel builds is with regard to big, popular, frequently changing
> packages, such as GHC, boost, gcc, glibc, etc.
> 
> Would you agree that building those packages -- and only those packages
> -- with parallelism enabled might be beneficial?

As I have already stated, I think that this is valuable, and that the
risk is very small (and testsuites can be run later).
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.15 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJMHSccAAoJEE6tnN0aWvw3t1YIALLoiMEs3UmnUqUMOCr4YtBd
l9fFX+vMxDVoKZoqKdP3M0ZR6zWjz1C1h9et2JYKY6Ccj8UKdAeM9gUGF9RX4QCW
8kztDs4XjIe1Pi+FcW3pD4/PSUdj1hvsK1xXZaJ1UuB0wjxhlOjfo9TMrVaf/IkT
nODAHWsVR/UHlDeG6NM6FZDp6pTOTonAOPx7jOlvwPu1mJ2GXHo3DYs/hgPpP5na
Pc5x86q3rfmxyoj9u0vWsX86i2hMt0et7fpPdCKsyoQN1P3sNkWSOlH/qPVV//9p
o8DXiZMd1fQ7XBY0CR4Ed0n9tHqNRHn5qgu77et5kguOYXdq7U3qJk+3ax54OZo=
=jcY1
-----END PGP SIGNATURE-----