[Nix-dev] Re: Another approach at parallelizing in-build jobs

[Peter Simons]

Hi Lluís,

 > Michael Raskin proposed the idea of using substituters to achieve
 > in-build parallelization.

unfortunately, I don't know much about substituters and their role in the build
process. Can you (or Michael, anyone else) please post a quick explanation of
how that approach would work? When are those "substituters" run? What exactly
do they do?


 > As we have little means to achieve reproduceability once we have the
 > parameters "-j" with a number greater than one in 'make' [...].

I have heard this claim several times before, but so far no-one has been able
to give any reasoning whatsoever to back it up. Because of that, I tend to
believe that this claim is based on irrational fears that are rooted in a lack
of understanding.

Assume that we are building an executable "main" from two independent source
code files "foo.c" and "bar.c". That procedure works as follows:

 1) Compile foo.c to foo.o.
 2) Compile bar.c to bar.o.
 3) Link bar.o and foo.o to main.

Now, the obvious question is: Is it necessary to build foo.o before bar.o?

And the obvious answer is: no, that's not a necessary choice. We could just as
well build bar.o first and foo.o second. The outcome of the build process would
be exactly the same, because the two files are independent of each other,
meaning that foo.c does not affect the compilation of bar.c, and bar.c does not
affect the compilation of foo.c.

In fact, Makefiles don't even specify that order. Whether make builds foo.o
before bar.o or bar.o before foo.o is implementation defined.

Now, the next question is: Is it necessary to build those objects sequentially
at all?

No, it turns out, it is not. It's entirely possible to compile foo.o and bar.o
in parallel without affecting the outcome of the build, because those two files
are independent of each other.

The executable "main", however, is not independent. main depends on both foo.o
and bar.o, which is why both files must be built before main can be built.
Figuring that out, i.e. deciding which files need to be built in what order, is
the purpose of make. The tool has been specifically designed to do that.

Take care,
Peter