[Nix-dev] Re: Another approach at parallelizing in-build jobs

[Michael Raskin]

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

> I have heard this claim several times before, but so far no-one has been able
> to give any reasoning whatsoever to back it up. Because of that, I tend to
> believe that this claim is based on irrational fears that are rooted in a lack
> of understanding.

The fear of opt-out mechanism is based on the fear of _few_ small rogue
packages that will be built incorrectly. As for some of us Nix is about
reproducibility, it causes what some find overreaction. Manually
creating more indeterminism without a reliable tool to check whether it
changed anything makes some of us nervous.

> Assume that we are building an executable "main" from two independent source
> code files "foo.c" and "bar.c". That procedure works as follows:
> 
>  1) Compile foo.c to foo.o.
>  2) Compile bar.c to bar.o.
>  3) Link bar.o and foo.o to main.

<.. skipped ..>

> In fact, Makefiles don't even specify that order. Whether make builds foo.o
> before bar.o or bar.o before foo.o is implementation defined.

Note - many people pretend that there is only one implementation of
"make", namely "GNU Make". Once "GNU Make" and "BSD Make" do something
in the same order, for many packages that means "make always does that".

In this situation hoping that everyone keeps in mind every thing that
can be implementation-dependent is too optimistic.

<.. skipped ..>

> The executable "main", however, is not independent. main depends on both foo.o
> and bar.o, which is why both files must be built before main can be built.
> Figuring that out, i.e. deciding which files need to be built in what order, is
> the purpose of make. The tool has been specifically designed to do that.

As far as I understand, the concern is about small makefiles with
underspecified dependencies that still work because "make" always does
it in the correct way.

I agree that any build _failures_ are not that bad here - I will try to
explain a case of "successfully" building an incorrect package.

Let the package have a markdown documentation that it converts
markdown -> tex -> pdf (for whatever reasons - I agree that examples are
rare). Unfortunately, even packages using "make" sanely are sometimes
seen doing such conversions for documentation with custom scripts.
Sometimes non-trivial related functionality is built into the conversion
tool. At least, we can assume without going too far from realism that
some packages implement a conversion with a cycle over all found files.

I also assume that dependencies are underspecified - I do not believe it
is a rare case...

Let this be implemented like that:

::::
all-tex:
  for i in md/*.md; do markdown-to-tex $i --output-directory tex; done

all-pdf:
  for i in tex/*.tex; do (cd pdf/; pdflatex ../$i); done

all-doc: all-tex all-pdf all-html
::::

What is the chance of this causing a single-threaded build problem? I
bet on zero with GNU Make and BSD Make.

What is the chance of such things going unfixed in a large project
(where there is a reason to build it with -j8)? I guess quite low
(although more complex things happen - like with GHC).

I will easily believe that Linux Kernel has no such catches now.

But for a not-too-popular _small_ package (build time under a minute on
single-core?) it can easily go unnoticed.

I do agree with both sides - using "-j$NUM_CORES" by default for
personal builds is not "picking pennies from under steamroller", more
like picking valuable gems from under child's bicycle. Hydra can always
find better things to do than building in parallel (namely, build other
things). Having a mis-built small package is still annoying, although
quite unlikely.

It is a case where everybody is "right" just because it is a value-based
choice. So we just need to find out how to give everyone whatever is
best for the person's special needs...
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.15 (GNU/Linux)
Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/

iQEcBAEBAgAGBQJMHRYoAAoJEE6tnN0aWvw31P4H+wVP4FXnFiVBaQYiiEk3X4ta
oKjDUba2ejn1lI097oQ43Rzxlcjds5vVWFfuNBwMxdojhmQrBwu5Vd3tLoQZf4HJ
MvjolA+ux8c3PMKWcgyGhbNAw2NivAVlYf4BKLQIx8L0lRrjcI3fL5Ug9dYEzBMH
csUfYn1AHvNK6sFYZgM4WANclbhs/FJY95Y1LrXgn8JbkktWZ6T6zXJkeFMPjFpN
ECWVV4pIHvZQ/C//crvtp+SJMSp7M/fAl7bBsOuyrY8mHPrkd4ZWXOZmkyJVa7l5
DUuXDlVNMeQARog1dpr9liPOoaIBYQqhg5aOrDcTNxzJaBLW3XFTrjHssZNPkds=
=5SPd
-----END PGP SIGNATURE-----