[Nix-dev] Re: Another approach at parallelizing in-build jobs

[Peter Simons]

Hi Lluís,

 >>  > Michael Raskin proposed the idea of using substituters to achieve
 >>  > in-build parallelization.
 >>
 >> unfortunately, I don't know much about substituters and their role in the
 >> build process. Can you (or Michael, anyone else) please post a quick
 >> explanation of how that approach would work? When are those "substituters"
 >> run? What exactly do they do?
 >
 > Substituters. There are now (at least) 'download-from-manifest' (take builds
 > available in a manifest through curl and nix-store --import), and
 > 'copy-from-other-stores' (take builds from a remote nix store mounted in the
 > system file tree).

I see, thank you for the quick response. What I don't see, however, is how
substituters might accomplish in-build parallelization?


 > I have a fear (not necesarily irrational) that not many people wrote good
 > Makefiles (in terms of reproduceability of results in parallel builds).

It sounds a lot like your fear *is* irrational, because it is in stark contrast
to the real world. Virtually everyone who owns a multi-core machine does enable
parallel building, because without that feature build times for packages like
the Linux kernel, GHC, GCC, glibc, etc. would be so immense that it would be
extremely unpleasant to actually work on that code. Consequently, build systems
of packages of a significant size tend to be extremely well tested with regard
to parallel building. Also, other Linux distributions like Gentoo or ArchLinux
utilize parallel building, and that seems to work, too. So I wonder on what
rational criteria your fear might be based?

It is no secret that some packages don't support parallel building for whatever
reason, and the commonly utilized approach to deal with that situation is to
build them single-threadedly.

Can you describe an actual problem that you anticipate when building, say the
Linux kernel, with parallelism enabled?

Take care,
Peter