[Nix-dev] proposal: in build parallelization policies

[Lluís Batlle i Rossell]

On Mon, Jul 12, 2010 at 06:29:52AM +0200, Marc Weber wrote:
> I'm unhappy with opt-in - you're unhappy with opt-out.
> So what about this?
> 
> additional point:
> It documents which phases are likely to be parallelizeable
> 
> Matrix illustrating when to use -j -l options for make:
> 
>   builder setting       Nix in-build-parallelization-policy
> 
>                     ||  never      opt-in    i-feel-lucky
>                     ||(default)               = opt-out
>   ==================||==================================
>                     ||  do   parallel    build?
>   always            ||  no           yes       yes
>   often             ||  no           no        yes [1]
>   never             ||  no           no        no
>   not set           ||  no           no        yes

I like this approach, for the jobs not having any note about this being in the
'often' group. For sure I prefer this instead of the dichotonmy
"parallel/not-parallel". I think that from the conversations, some prefered
two kinds of parallelization: opt-out and opt-in, and this solves that little
trouble.

I think that, additionally, people running with opt-out can be a nice test for
moving things from "often" to "always" or "never".

> 
> stdenv.mkDerivation {
>   # build in parallel if policy is "i-feel-lucky"
>   parallelBuildable   = "often" 
> 
>   # never build in parallel
>   parallelInstallable = "never" 
> 
>   # build in parallel if policy is set to etiher "opt-in" or "i-feel-lucky"
>   parallelCheckable   = "always"
> }

I would not do any parallel installable, because I think it will mostly work in
terms of I/O-parallelization and not CPU-parallelization.

> [1]: As check I'd implement two:
> 
>   cd $out
> 
>   if [ $symbolsHash != '' -a $(for x in **/*.so; do symbols $x; | done | md5sum) != "$symbolsHash" ];
>     echo "symbols changed - did parallel build break it?"; exit 1
>   fi
>   if [ $filesHash != '' -a $(find) != "$filesHash" ];
>     echo "file list differs - did parallel build break it?"; exit 1
>   fi

I don't think we need these checks. Maybe we could find better checks, but I'd
go without.

Regards,
Lluís.