[Nix-dev] Re: Another approach at parallelizing in-build jobs

[Peter Simons]

Hi Michael,

 > The fear of opt-out mechanism is based on the fear of _few_ small rogue
 > packages that will be built incorrectly.

there seems to be a misunderstanding.

We both agree that an opt-out policy for parallel building is a bad
choice. For any random package, the default assumption should be that
parallel building is unsafe. Parallel building should be enabled only
for those package for which we can be reasonably sure that parallel
builds are supported, well tested, and reliable.

In other words, parallel builds should be performed using an explicit
opt-in policy.


 >> In fact, Makefiles don't even specify that order. Whether make
 >> builds foo.o before bar.o or bar.o before foo.o is implementation
 >> defined.
 >
 > Note - many people pretend that there is only one implementation of
 > "make", namely "GNU Make". Once "GNU Make" and "BSD Make" do
 > something in the same order, for many packages that means "make
 > always does that".
 >
 > In this situation hoping that everyone keeps in mind every thing that
 > can be implementation-dependent is too optimistic.

Arguably, that problem doesn't exist in Nix, because Nix generally uses
GNU Make, so there is only one de facto implementation. Of course,
there's an odd chance that the implementation of GNU Make changes
behavior in future versions, but that doesn't seem to be particularly
likely.

Anyway, the point I was trying to make is a slightly different one. I
was trying to illustrate the basic principles that say that the exact
build order of independent files is irrelevant.


 > As far as I understand, the concern is about small makefiles with
 > underspecified dependencies that still work because "make" always does
 > it in the correct way.

Yes, that is a real problem. Makefiles may be broken when parallel
building is enabled, but that bug doesn't necessarily occur in every
build. This is the classic problem with race-conditions. Those are
generally hard to debug. I guess is the main reason why opt-in is far
more appealing than opt-out.


 > [concrete example omitted]
 >
 > I will easily believe that Linux Kernel has no such catches now.
 >
 > But for a not-too-popular _small_ package (build time under a minute on
 > single-core?) it can easily go unnoticed.

Yes, you are right. Personally, I believe that parallel builds are not
very important for small or obscure packages, simply because those
packages are built rarely (or quickly) anyway. My main interest in
parallel builds is with regard to big, popular, frequently changing
packages, such as GHC, boost, gcc, glibc, etc.

Would you agree that building those packages -- and only those packages
-- with parallelism enabled might be beneficial?


 > It is a case where everybody is "right" just because it is a value-based
 > choice. So we just need to find out how to give everyone whatever is
 > best for the person's special needs...

Yes, I completely agree with you. Generally speaking, a solution that
satisfies many different needs is probably superior to one that doesn't.

Whatever happens on the "parallel make" front, I feel it should be
entirely optional, and parallel builds should be enabled strictly on an
opt-in basis for well-tested and reliable packages.

Take care,
Peter