[Nix-dev] Yet another flag and derivation arg assembling system proposal

[Nicolas Pierron]

Hi Marc,

On Sat, Nov 15, 2008 at 00:57, Marc Weber <marco-oweber at gmx.de> wrote:

> ============= composedArgsAndFun - mergeAttrByFunc ====================
>
> To start building a small library of reusable code (stubs for different
> purposes such as "build and install a python package") there need to be
> kind of inheritance. foldArgs is perfectly usabale.. However it's hard
> to say how to merge the basic implementation with what is needed for the
> actual derivation. So I'd like to introduce you to the heart of this:
>
> mergeAttrByFunc = attr1 : attr2 : .... merging those attr1 and attr2
>
> so why yet another function? You can already choose one of:
> a) mergeAttrsNoOverride (my old function, now depreceated in favour of
>   the new one)
> b) mergeAttrsConcatenateValues (can only handle lists)
> c) mergeAttrsWithFunc (you have to specify one merge func for all attr
>   names)
> d) mergeDefaultOption (merges a set of list, attrs, functions,
>  whatsover, currently used for assembling the nixos configuration)

Have you look at mergeOptionSets, I think this almost match your expectations.

> Because they didn't do exactly what I had in mind. I wanted something
> which knows how to merge simple things such as
> meta, passthru, configureFlags, buildInputs, propagatedBuildInputs as
> well as custom stuff such as assembling test scripts by concatenating
> the strings separated by "\n".

So you want to declare some kind of type for each attribute in order
to use the right merge function on them if they are many.

> Because nix isn't good at typing I didn't try to invent different types,
> I just put the merge description within the attrs and remove it again
> before passing the attrs to mkDerivation.

Even with types you may want to apply different merge functions.  So
types are useful for documentation and to remove default merge cases
but they cannot fill up all merge cases.  In Nixos we use mkOption for
this purpose to add a description to each option and to specify how
they have to be merged and applied.

>  Example:
>
>    x = mergeAttrByFunc  { # implementation attached [1]
>     mergeAttrBy = {
>        buildInputs = x : y : x ++ y;
>        myTestScript = x : y : "${x}\n${y}";
>     };
>
>     buildInputs = [ a b ];
>     myTestScript = "import foo";
>    } {
>      buildInputs = [ c d ];
>       # you could use mergeAttrBy = { buildInputs = (x : y : y); };
>       # to get back the // behaviour (take snd/last)
>       # because the mergeAttrBy is merged by // before applying the
>       # funtions overriding works fine..
>     myTestScript = "import bar";
>    };
>
>
>  result:
>  x = {
>    mergeAttrBy = ...;
>    buildInputs = [ a b c d];
>    myTestScript ="import foo\nimport bar";
>  };

> So that's nice: using this way you can add stuff in a sensible way.. but
> how to remove stuff / shrink the attrs again?

Right, we should find a way to define this in the merge construct,
like an "undefined" value and a post-process step to remove all
undefined values from the attribute set.


>      # removed before reachungi mkDerivation
>      # =====================================
>
>      mergeAttrBy = { pyCheck = x : y : "${x}\n${y}"; };
>
>      # assuming that if a module can be loaded that it does also work..
>      # all the sub attrs will be merged in the sensible way if the flag
>      # is set. You can also use zlib = { set = { [. if set .]; }; unset = { [. if not set . ]; }; };
>      # instead to use add --disable-feature the configureFlags if zlibSupport is not set etc.
>      # for each flag a feature = true or false will be added to
>      # passthrough. If you need an env var add pass = { envvar = 1; }
>      # to the flag description.
>      flags = {
>        zlib = { buildInputs = [ zlib ]; pyCheck = "import zlib"; };
>        gdbm = { buildInputs = [ gdbm ]; pyCheck = "import gdbm"; };
>        sqlite = { buildInputs = [ sqlite ]; pyCheck = "import sqlite3"; };
>        db4 = { buildInputs = [ db4 ]; }; # TODO add pyCheck
>        readline = { buildInputs = [ readline ]; }; # doesn't work yet (?)
>        openssl = { buildInputs = [ openssl ]; pyCheck ="import socket\nsocket.ssl"; };
>      };

I think this start to be very similar to nixos configuration files in
a smaller syntax.  Declared flags correspond to enable options and
buildInputs correspond to an option declared by mkDerivation.
Therefore to detect options you just have to check for declared
options as done in Nixos.

I think this syntax is something that could be used in Nixos with some
kind of mkService in order to reduce the writing.

> I hope you've enjoyed reading this post and even understood some parts
> of it? I apprecatiate any feedback. I'll continue to improve the idea. Then I'll
> send a full patch to the mailinglist so that we can discuss it further.

I suggest you to use some parts of mergeOptionsSets for your
implementation of mergeAttrByFunc as started in the following example:

mergeAttrByFunc = x : y :
  let
    mergeAttrBy2 =
       { mergeAttrBy=mergeAttr; }
    // (maybeAttr "mergeAttrBy" {} x)
    // (maybeAttr "mergeAttrBy" {} y);

    merge = name: values:
      let tailValues = tail values; x = head values; y = head tailValues; in
      if tailValues = [] then x
      else (__getAttr name mergeAttrBy2) x y;
  in
    zip merge [x y];

I think that options used in Nixos are really similar to what you have
introduced in this mail.  To compare both approaches I have made this
small listing:

__
- Feature:
1) Pkgs config
=> comments
2) Nixos config
=> comments

- config:
1) *Support = true/false
=> easy to handle by a gui.
2) any kind
=> Give more information like description, ...

- merge:
1) mergeAttrBy = { foo = ...; };
=> As the merging is order, merging functions can be override.
2) foo = mkOption { merge = ...; };
=> Only one option can be declare for one name.

- sub-config:
1) flags = { foo = {..}; bar = {..}; };
=> sub-configuration are included in function of the value of the
corresponding Support flag.
2) { require = [ (inherit ./foo) (inherit ./bar)]; .. }  &   config: {
foo = if config.bar then .. else ..; }
=> configuration can be highly configure in many files by the result
of the fix operation.

-- 
Nicolas Pierron
http://www.linkedin.com/in/nicolasbpierron
- If you are doing something twice then you should try to do it once.