[Nix-dev] Re: Another approach at parallelizing in-build jobs

[Lluís Batlle i Rossell]

On Sun, Jun 20, 2010 at 10:02:16PM +0200, Peter Simons wrote:
> Hi guys,
> 
> Now, there is another matter that needs to be discussed before we can
> make decisions about an actual implementation of that feature. Assuming
> that we can build, say glibc, with parallel compilation enabled, do you
> feel that the choice of the -j flag to GNU Make should influence the
> store path of the resulting binary package?
I think it should not.
> What is your preferred solution? And what are the pros and cons that you
> see?
I think that the user should be able to choose alternative ways of building the
packages, alternatives to "build without any -j or -l flag". For me, this falls
into the same situation of the 'build-remote.pl', 'copy-from-other-stores.pl', and
'download-from-manifest.pl'. We could have a
'build-parallel.pl', that users could push in as done currently
with the NIX_SUBSTITUTERS environment variable. That could take user settings on
the parallelism from any place.

Nevertheless, as far as I understand, currently the build-remote.pl or
build-parallel.pl cannot be implemented over the substituters API for at least
these reasons:
- The API does not distinguish if nix wants a .drv or an output path.
- nix does not provide the .drv file in the store, for an output path wanted
- The substituter calling "nix-store --realise" for a drv (the substituter could
put a new drv in the store for the parallel build) that would provide the
output path wanted would end in a deadlock due to store path locking.

I've not investigated that in deep, so feel free to amend this. I think that the
changes to the nix API are not that heavy to allow all this (the lock problem
apart), but currently I'm not a volunteer to implement that soon.

The opt-in procedure could be based on setting some variable in the drv, so the
build-parallel.pl understood it, and it would not conflict with canonical
builds.

As for hydra... the hydra owners should do a good choosing of substitutes for
their paths, and that good choosing my involve not using the build-parallel at
all.

Regards,
Lluís.