[Nix-dev] Re: [Nix-commits] SVN commit: nix - 22282 - MarcWeber - nixpkgs/branches/stdenv-updates/pkgs/stdenv/generic

[Marc Weber]

May I try to summarize?

topics

parallel build within a derivation

a) for Hydra ( I never intended to talk about this at this stage )

b) for development only on local machines. (I wrote the patch for this
   use case)

c) after global opt-in for each nix-daemon or by setting nix.conf there
  are still two ways:
  a) opt-in per package   ( mark packages or build phases as "good" )
  b) opt-out per package  ( mark packagse or build phases as "bad" ) << current implementation
  c) let the user choose. ( mark both: good and bad packages. So the
  user can control by choosing what should be done with packages which
  are not flaged as good or bad)
  
Is a package safe. Can it be build, installed, checked in parallel?
How to find out? Existing impurities:
  - CPU
  - date/time
  - filesystem being in use (?)
  - is chroot build?
  - hardware configuration
  - possibility of hash collisions ?
  - possible network  access (?)
  - your life in general is .. :)
  - kernel your build runs on
  - xray from space
 [- impurity by parallelism ]

So the question is is whether the benefits of parallel builds within a
build can outperform the "risk" of bad builds.

 benefits:

    - Maybe we can build build jobs which are important (eg security fixes)
    faster.

    - local builds are faster. (Eg apply a kernel patch etc)

Of course if you only have a look at Hydra and its load you can keep
load high with or without parallel builds because Hydra has always
something to do.. 

Looking at the impurity list I have to ask whether we should start
building each package at least once for each kernel, with and without
chroot, and on Monday, Tuesday, Wednesday, Thursday, Friday, Saturday
and Sunday. Maybe a build fails on Sunday only ? Do you know?

I don't want to talk about implementation details. I'm happy to rewrite
the code. The question is: Do you see benefits in having this feature or
not. And if so which opt-in-out policies you prefer. You should make
clear statements about that b),c)

> I'm basically in favor of your patch. I'm just unhappy about the opt-out
> policy decision. I'm also somewhat unhappy about your patch containing
> unrelated re-factoring.

Again "saying' unhappy is not enough to improve it. The "refactoring"
contained several things.
- removing of echo lines
- passing -f $makefile once only
- introducing cmd and runMake.
  cmd was meant to replace echo. It is applying the pattern: Doing
  something which was done multiple times once only.
  And that pattern should is used very often.

Just saying "I'm unhappy" doesn't help. You have to tell me which items
you dislike. Then I can try to find alternative implementations.
Doing things once make things easier in general.

alternative for ugl names:

  PARALLEL_BUILD_OPT_OUT="all" # or "install check" < this means use parallel build for build phase only

  This would be nicer to read and write ?

  of course we could use OPT_IN instead. However this means touching
  almost every file in nixpkgs. Do you want this?

....

Again I'd like to see alternative ways instead of "no".

I don't think NixPkgs should serve Hydra only (it should not harm it
either)

Yours
Marc Weber