[Nix-dev] proposal: in build parallelization policies

[Marc Weber]

I'm unhappy with opt-in - you're unhappy with opt-out.
So what about this?

additional point:
It documents which phases are likely to be parallelizeable

Matrix illustrating when to use -j -l options for make:

  builder setting       Nix in-build-parallelization-policy

                    ||  never      opt-in    i-feel-lucky
                    ||(default)               = opt-out
  ==================||==================================
                    ||  do   parallel    build?
  always            ||  no           yes       yes
  often             ||  no           no        yes [1]
  never             ||  no           no        no
  not set           ||  no           no        yes

  [1] (only if checks are implemented catching bad builds - eg $symbolsHash and $filesHash is set)

Example usage:

stdenv.mkDerivation {
  # build in parallel if policy is "i-feel-lucky"
  parallelBuildable   = "often" 

  # never build in parallel
  parallelInstallable = "never" 

  # build in parallel if policy is set to etiher "opt-in" or "i-feel-lucky"
  parallelCheckable   = "always"
}

[1]: As check I'd implement two:

  cd $out

  if [ $symbolsHash != '' -a $(for x in **/*.so; do symbols $x; | done | md5sum) != "$symbolsHash" ];
    echo "symbols changed - did parallel build break it?"; exit 1
  fi
  if [ $filesHash != '' -a $(find) != "$filesHash" ];
    echo "file list differs - did parallel build break it?"; exit 1
  fi

Do you want to discuss this idea?

Marc Weber