[Nix-dev] python proposal

[Marc Weber]

On Tue, Dec 02, 2008 at 01:51:51PM +0100, Eelco Dolstra wrote:
> Marc Weber wrote:
> 
> > I've comitted my work now.
> > Let me know what you think about it.
> > If there is any trouble tell me to fix it immediately :)
> > 
> > I've told you about the basic features in previous posts.
> > Would you benefit of a small composableDerivation tutorial?
> 
> Please do.  What does it do?  (I'm kind of concerned that we get yet another
> style of writing expressions only used by some small subset of packages that I
> don't really understand...)

Don't mind. I'll remove my *old* one in favour of the new one.
And it has been desigen to almost look and feel like the default
stdenv.mkDerivation. But you can use additional features if you have to.

By the way: would it make sense to have kind of lib_test.nix file.
It could test the functions and also serve as example usage ?

topics:

  composableDerivation is made up of
    a) mergeAttrsByFuncDefaults
             mergeAttrsByFuncDefaults = foldl mergeAttrByFunc { inherit mergeAttrBy; };
    b) prepareDerivationArgs
    c) applyAndFun
    d) composableDerivation calling those above

long description is following now:

  a) mergeAttrsByFuncDefaults is calling mergeAttrsByFunc with some
    defaults.
    I think an example usage does show best how it's used:
        mergeAttrsByFuncDefaults [
        {
          A = 10;
          B = 10;
          mergeAttrBy = {
            B = x : y : __add x y;
          };
        }
        {
          A = 20;
          B = 20;
          # [2]
        }
        ]
     will result in
     {A = 20; B = 30;} because B has a merge func. 
     the merge func can be overridden eg at [2].
     for common names there are sane defaults (// or ++) (-> mergeAttrBy in lib)

  b) prepareDerivationArgs is a helper function handling configuration
     options. The best way to learn about it is having a look at an
     example:

    composableDerivation { #0
      name = "gst-python-0.10.13";
      flags = {
        pluginsGood = { propagatedBuildInputs = [C]; }; #1
        ffmpeg = { propagatedBuildInputs = [B]; }; #2
      };
      cfg = {
        pluginsGoodSupport = true;
        ffmpegSupport = true;
      };
      propagatedBuildInputs = [ A ];
    };

    The main attr #0 is merged by a) with sub attrs #1 and #2 only when the
    configuration option has been set to true.
    In this example both are set to true so the result of

    composableDerivation looks like this:
    {
      name = ...;
      propagatedBuildInputs = [ A B C ];
      passthru = { pluginsGoodSupport = true; ffmpegSupport = true; }
    }

    Note that passthru has been set automatically for you so that you can
    check for features in depending derivations.

        ffmpeg = { propagatedBuildInputs = [B]; }; #2
    is a shortcut for
        ffmpeg = { 
          set = {
            propagatedBuildInputs = [B]; 
          };
          unset = {
          };
        }

    This way you can pass things if a flag is not set as well.
    example:
      flags = {
        ffmpeg = {
            unset = {
              configureFlags = ["--disabel-feature"];
              assert = !dep.xySupport; #a
            }; 
        };
      };

    Configurations even supports kind of assertion (#a)
    Will trigger this error
        throw "assertion of flag ${a} of derivation ${args.name} failed"

  c) applyAndFun is function based on the work and ideas by Michael Raskin.
    Basic idea: compose arguments before passing them to function.

    The first design (by Michael R.) looked like this:
      name f { attrs } { attrs } null;

    which has been superseded by this design (Michael R again)
      deriv = composedArgsAndFun mkDerivation { name = .. ; buildInputs = ..; };
      myDeriv = deriv.passthru.function {
        # additional args to be merged.
      }

    So there is no longer a "null" termination  but a
    .passthru.function composition continuation. Basically its' the same.
    But there is no longer the need to explain what "null" does :-)

    Here another exapmle to make it most obvious how it works:
      ( ( (name f { attrs }).passthru.function 
        { attrs2 }).passthru.function 
        { attrs3 }).passthru.function )

    How attrs, attrs2, attrs3 are merged depends on which merge function
    has been specified resulting in function "name". In both cases
    (composedArgsAndFun and my applyAndFun) version you can also pass a
    function instead of an attr set. Eg replace { attrs 3 } in the exapmle
    above by (composition : { foo = composition.A }) to access things
    which has been passed before.

    So how does applyAndFun differ from composedArgsAndFun ?
      * It let's you remove attrs as well
      * Its stripped down a little bit because you don't have to pass an
        initial attrs (see definition of composedArgsAndFun which gets
        passed an empty attr set )
      * it implements fix (more bloat again :-)

    fix feature of applyAndFun
    =========================
    start = applyAndFun mkDerivation {
      start = foo;
    };
    deriv = ( composition: {
      foo = composition.fixed.bar; # << fixed is used here
      start2 = composition.start; # you can use what has been passed without "fixed"
    });
    final = deriv.passthru.fun {
      bar = "this value is used above";
    }
      
    Merging is the behaviour you want most often. That's why I've been
    following the idea of Michael Raskin again by introducing an
    additional function: .passthru.funMerge which does merge by default.
    
    summary / comparison:
    =====================
    composedArgsAndFun:
      - interface:
          .passthru.function (does merge automatically)
      - you can pass a function or attrs

    applyAndFun (my new version of composedArgsAndFun):
      - interface:
          .passthru.funMerge (does merge automatically - only matters when passing a function)
          .passthru.fun      (doesn't merge. passing (x : {}) will remove all attrs)
      - you can pass a function or attrs
      - fix feature: you have .passthru.funMerge (x : x.fixed ) availible..

  d)  composableDerivation
    You know what ? You already know what it does:

    It let's you compose args by applyAndFun, applying the result to
    prepareDerivationArgs (handling configuration options) passing that
    result to mkDerivation.

    So as long as you don't use
    .passthru.fun(Merge) or cfg = {...} or  flags = {..} or fixed
    it should behave exactly like stdenv.mkDerivation does except that you
    have to use composableDerivation { initial = { name = ...; meta =...; } ; }

    The implementation looks like this:

        # see new python derivations for example..
        # You should be able to override anything you like easily
        composableDerivation = {
                # modify args before applying stdenv.mkDerivation, this should
                # remove at least attrs removeAttrsBy
              f ? lib.prepareDerivationArgs,
              stdenv ? pkgs.stdenv,
                # initial set of arguments to be passed to stdenv.mkDerivation
                # passing prepareDerivationArgs by default
              initial ? {},
                # example func :  (x: x // { x.buildInputs ++ ["foo"] }), but
                # see mergeAttrByFunc which does this for you
              merge ? (lib.mergeOrApply lib.mergeAttrByFunc)
            }: lib.applyAndFun
                  (args: stdenv.mkDerivation (f args))
                  merge
                  (merge { inherit (lib) mergeAttrBy; } initial);
      
Hopefully you're able to read the new python derivations now?

Sincerly
Marc Weber