[Nix-dev] Yet another flag and derivation arg assembling system proposal

[Marc Weber]

Some time ago I've written all the stuff at the end of lib/default.nix.
The stuff leading to listToAttrs and to some commented out dead code
(using the new derivation style proposal)..
To make the long story short I found it was a way to complicated to pay
off. I don't think anyone else has used it anyway so I'd like to show
you my even newer much more flexible proposal based on foldArgs (used by
composedArgsAndFun as well).

============= composedArgsAndFun - mergeAttrByFunc ====================

To start building a small library of reusable code (stubs for different
purposes such as "build and install a python package") there need to be
kind of inheritance. foldArgs is perfectly usabale.. However it's hard
to say how to merge the basic implementation with what is needed for the
actual derivation. So I'd like to introduce you to the heart of this:

mergeAttrByFunc = attr1 : attr2 : .... merging those attr1 and attr2

so why yet another function? You can already choose one of:
a) mergeAttrsNoOverride (my old function, now depreceated in favour of
   the new one)
b) mergeAttrsConcatenateValues (can only handle lists)
c) mergeAttrsWithFunc (you have to specify one merge func for all attr
   names)
d) mergeDefaultOption (merges a set of list, attrs, functions,
  whatsover, currently used for assembling the nixos configuration)

Because they didn't do exactly what I had in mind. I wanted something
which knows how to merge simple things such as
meta, passthru, configureFlags, buildInputs, propagatedBuildInputs as
well as custom stuff such as assembling test scripts by concatenating
the strings separated by "\n".

Because nix isn't good at typing I didn't try to invent different types,
I just put the merge description within the attrs and remove it again
before passing the attrs to mkDerivation.

  Example:

    x = mergeAttrByFunc  { # implementation attached [1]
     mergeAttrBy = {
        buildInputs = x : y : x ++ y;
        myTestScript = x : y : "${x}\n${y}";
     };

     buildInputs = [ a b ];
     myTestScript = "import foo";
    } { 
      buildInputs = [ c d ];
       # you could use mergeAttrBy = { buildInputs = (x : y : y); };
       # to get back the // behaviour (take snd/last)
       # because the mergeAttrBy is merged by // before applying the
       # funtions overriding works fine..
     myTestScript = "import bar";
    };


  result:
  x = { 
    mergeAttrBy = ...;
    buildInputs = [ a b c d];
    myTestScript ="import foo\nimport bar";
  };

So that's nice: using this way you can add stuff in a sensible way.. but
how to remove stuff / shrink the attrs again?

MichaelRaskin has written foldArgs for this. What it does is: Merge
attrs with another attr set, then add to the resulting set a function to
which you can apply either
a) another attrs (so the sensible merge above will take place)
b) a function replacing the attrs with another one (so you can remove it again)

  Example assuming you already have a
  base = { 
    [ .. stuff ..]; 
    passthru = { function = < the nice thing> ; };
  };
  created by a composedArgsAndFun using foldArgs like function.

  a) looks like this:

    baseMerge = base.passthru.function {
      configureFlags = ["additional stuff"];
    };

  b) looks like this. This example eg shrinks the attrs by removing the
     configureFlags name:

    baseMerge = base.passthru.function (x : removeAttrs x ["configureFlags"]);

    note: I like this behaviour, MichaelRaskin prefers to not allow this.
      That's why composedArgsAndFun uses something (x : x // removeAttrs x [ "configureFlags"])
      in a hidden way so even if you shrink the attrs everything gone
      will be merged back by //. I think that's a minor but important
      difference.

============= prepareDerivationArgs ==================================

Of course we're not done yet at all. What about building different
variations of the same type? There are many use cases where you want to
bulid support for xy in or you don't and some dependencies have to
verify that a feature has been compiled in. Some use cases are:

- jdk with plugin support for firefox
- svn with perl support needed by git-svn
- ...

a typical example on how this has been done most often is
development/interpreters/python/2.5/default.nix

stripping it down to the important things it looks like this:


  { stdenv, fetchurl, zlib ? null, zlibSupport ? true, bzip2
  , gdbmSupport ? true, gdbm ? null
  , sqlite ? null

  [...]

    buildInputs = [..]
      ++ optional zlibSupport zlib
      ++ optional gdbmSupport gdbm [..];

    preConfigure = " [..] "
    + (if readline != null then ''
      export NIX_LDFLAGS="$NIX_LDFLAGS -lncurses"
    '' else "");

    passthru = {
      sqliteSupport = sqlite != null;
      [..];


pro: It's somewhat easy to understand because the most complex function
  beeing used is optional. This makes it easy for everyone to add just
  another if then else ..
con: There are two different ways to enable, disable features:
  passsing 
  - gdbmSupport = true /false 
  - passing db4 = <derivation> or null.. 
  This makes it harder to for users to see what you can change with that
  existing expression.
  It's nearly impossible to write some kind of convinient gui tool
  installer where you click on soem check boxes to enable or disable
  some features also viewing some description.

My old way tried to solve this problem as well also suffering from this
problem because there has been various way to enable and disable
features and pass configuration options etc.. That makes the code hard
to mantain because even i had to lookup stuff again and again.

That's why I've written prepareDerivationArgs [3]  from scratch also reusing
the merging stuff found above.. This leads to a very flexible
implementation still beeing very similar to using stdenv.mkDerivation
so I hope it's still easy to understand and work with.
One example showing a reimplementation of the python derivation [2]
using the prepareDerivationArgs function. Of course you can grab that
derivation, use the passthru.function to add additonal flags, test
script lines etc easily.

  pythonMinimal = ( (import ./python.nix) {
    name = "python-2.5.2";
    inherit fetchurl stdenv lib bzip2 ncurses composableDerivation;
    inherit zlib sqlite db4 readline openssl gdbm;
  });

  # all features
  pythonFull = pythonMinimal.passthru.function {
   name = "python-2.5.2-full";
    cfg = {
      zlibSupport = true;
      sqliteSupport = true;
      db4Support = true;
      readlineSupport = true;
      opensslSupport = true;
      gdbmSupport = true;
    };
  };

  # python.nix contents
  composableDerivation {
    # hack to define C_INCLUDE_PATH, LIBRARY_PATH before passing the attr to
    # mkDerivation.. Normally you can just stick to default (args: stdenv.mkDerivation (lib.prepareDerivationArgs args))
    f = args: let attr = lib.prepareDerivationArgs args; in 
        stdenv.mkDerivation ( attr // { 
        C_INCLUDE_PATH = concatStringsSep ":" (map (p: "${p}/include") attr.buildInputs);
        LIBRARY_PATH = concatStringsSep ":" (map (p: "${p}/lib") attr.buildInputs);
      });

    initial = {

      # removed before reachungi mkDerivation
      # =====================================

      mergeAttrBy = { pyCheck = x : y : "${x}\n${y}"; };

      # assuming that if a module can be loaded that it does also work..
      # all the sub attrs will be merged in the sensible way if the flag
      # is set. You can also use zlib = { set = { [. if set .]; }; unset = { [. if not set . ]; }; };
      # instead to use add --disable-feature the configureFlags if zlibSupport is not set etc.
      # for each flag a feature = true or false will be added to
      # passthrough. If you need an env var add pass = { envvar = 1; }
      # to the flag description.
      flags = {
        zlib = { buildInputs = [ zlib ]; pyCheck = "import zlib"; };
        gdbm = { buildInputs = [ gdbm ]; pyCheck = "import gdbm"; };
        sqlite = { buildInputs = [ sqlite ]; pyCheck = "import sqlite3"; };
        db4 = { buildInputs = [ db4 ]; }; # TODO add pyCheck
        readline = { buildInputs = [ readline ]; }; # doesn't work yet (?)
        openssl = { buildInputs = [ openssl ]; pyCheck ="import socket\nsocket.ssl"; };
      };
    
      # passed to mkDerivation maybe merged with flag data
      # =================================================

      postPhases = ["runCheck"];

      # should be last because it sources setup-hook of this package
      # itself
      runCheck = ''
        PATH=$out/bin:$PATH; . $out/nix-support/setup-hook;
        echo -e "import sys\n$pyCheck\nprint \"import pyCheck ok\"" | python
      '';

      inherit (args) name;

      src = fetchurl {
        url = http://www.python.org/ftp/python/2.5.2/Python-2.5.2.tar.bz2;
        sha256 = "0gh8bvs56vdv8qmlfmiwyczjpldj0y3zbzd0zyhyjfd0c8m0xy7j";
      };
      
      buildInputs = [...];
      [ ...]

I hope you've enjoyed reading this post and even understood some parts
of it? I apprecatiate any feedback. I'll continue to improve the idea. Then I'll
send a full patch to the mailinglist so that we can discuss it further.

At least I feel most comfortable with this design at the moment.
Thanks to MichaelRaskin for inventing composedArgsAndFun idea - I love it
(I feel so for nix(os) in general :)

Marc Weber

[1]

  mergeAttrByFunc = x : y :
    let mergeAttrBy2 = { mergeAttrBy=mergeAttr; }
                      // (maybeAttr "mergeAttrBy" {} x)
                      // (maybeAttr "mergeAttrBy" {} y); in
    mergeAttrs [
      x y 
      (mapAttrs ( a : v : # merge special names using given functions
          if (__hasAttr a x)
             then if (__hasAttr a y) 
               then v (__getAttr a x) (__getAttr a y) # both have attr, use merge func
               else (__getAttr a x) # only x has attr
             else (__getAttr a y) # only y has attr)
          ) (removeAttrs mergeAttrBy2 
                         # don't merge attrs which are neither in x nor y
                         (filter (a : (! __hasAttr a x) && (! __hasAttr a y) )
                                 (__attrNames mergeAttrBy2))
            )
      )
    ];

[2] I've rewritten that for two reasons:
  a) I've tried upgrading pstPython and I couldn't make it work using
      PYTHONPATH or such yet. That's why I try to reimplement it using
      kind of policy (each package has to provide PYTHONPATH by
      setup-hook so that it can be used easily and that you can write a
      python wrapper just adding PYTHONPATH to the env of python to
      allow using all the libs from command line..
  b) I don't like this
  ### PYTHON 2.5 modules ...
  within all-packages.nix. It makes it hard to upgrade because there is
  a time frame where the new libs don't work yet and you already pass
  the new python to the old ones.. or you start copying everything
  within all-packages leading to
  ### PYTHON 2.5 modules ...
  ### PYTHON 2.6 modules ...
  ### PYTHON 2.7 modules ...
  if we start doing this with haskell, perl, python, ruby, php..
  I don't want to imagine..
  On the other hand if we have
    python25Attrs = import ../python25-all ..
    python25Wrapper = python25Attrs.python25Wrapper;

  It'll be much cleaner because within that python25-all you have only one
  python version and everything ranging from version 2.3 up to 2.5.
  When upgrading you can clone the whole directory and start working
  without interfering with the already existing and working stuff..
  c) Having all python modules in a single file makes it really easiy to
    just copy paste one to add another library..
    I'd propose this style for all totally different languages.
  
[3]

  prepareDerivationArgs = args:
    let args2 = { cfg = {}; flags = {}; } // args;
        flagName = name : "${name}Support";
        cfgWithDefaults = (listToAttrs (map (n : nv (flagName n) false) (attrNames args2.flags)))
                          // args2.cfg;
        opts = flattenAttrs (mapAttrs (a : v : 
                let v2 = if (v ? set || v ? unset) then v else { set = v; };
                    n = if (__getAttr (flagName a) cfgWithDefaults) then "set" else "unset";
                    attr = maybeAttr n {} v2; in
                if (maybeAttr "assertion" true attr)
                  then attr
                  else throw "assertion of flag ${a} of derivation ${args.name} failed"
               ) args2.flags );
    in removeAttrs
      (mergeAttrsByFunc ([args] ++ opts))
      ["flags" "cfg" "mergeAttrBy" ];