[Nix-dev] python proposal

[Eelco Dolstra]

Hi,

Armijn Hemel wrote:

> On Tue, 2008-12-02 at 15:11 +0100, Marc Weber wrote:
>> Hopefully you're able to read the new python derivations now?
> 
> It makes my brain hurt.
> 
> All these abstractions make it really hard for mere mortals like me, who
> do not make their shoppinglists using Lambda calculus.

:-)

> If it were up to me, I'd rather go for simpler expressions, instead of
> more abstractions I don't understand. These abstractions make it just
> too much of an effort to keep up with.

I also have a hard time understanding a lot of this stuff.  This may have
something to do with the coding style, in particular the names of some
identifiers: for instance, a name like "composedArgsAndFun" doesn't tell me a
lot about what that function is supposed to do, yet I see it everywhere.  And
this new name "applyAndFun" isn't even grammatical.

Marc, you gave a few examples, but little *motivation*.  What do they allow us
to accomplish (better) that we couldn't do before?

(If the motivation is to make aspects of package functions overridable, it may
be worth considering that if you make everything overridable from the outside,
then by definition you have no abstraction: the entire internal operation of the
function is exposed to the caller.)

I'm in general worried that some of these new-style Nix expressions are much
harder to understand than the old style.  For instance, who wants to argue that
this expression (random example):

https://svn.nixos.org/repos/nix/nixpkgs/trunk/pkgs/os-specific/linux/sdparm/1.03.nix

is more readable and maintainable than (another random example):

https://svn.nixos.org/repos/nix/nixpkgs/trunk/pkgs/os-specific/linux/hdparm/default.nix

In part, this has to do with the indentation style, but also the different
"layers" of the expression that make it hard to follow what happens.  E.g.
there's a "let localDefs = builderDefs.passthru.function { ... }", the result of
which is included with a "with", making it hard to follow where identifiers come
from.  The "src" is an attribute of localDefs - why not of stdenv.mkDerivation?
Where does "version come from?  Why do I have to write all this boilerplate
"writeScript ... (textClosure [...])"?  And so on.  Such a "layered" expression
is harder to read than the basic template

  {dependencies}:

  stdenv.mkDerivation {
    ... attributes of the derivation ...
  }

Of course, there's nothing sacred about stdenv.mkDerivation.  Making
higher-level functions that simplify common code (like setting up configureFlags
on the basis of user options) is great.  But you do want to keep the general
template

  {dependencies}:

  foo {
     ... attributes ...
  }

where "foo" is some function that implements the desired functionaly, like the
the support for "flags" in the new Python expression, which I like in principle:

    flags = {
      zlib = { buildInputs = [ zlib ]; pyCheck = "import zlib"; };
      gdbm = { buildInputs = [ gdbm ]; pyCheck = "import gdbm"; };
      ...
    };

as this kind of thing does make the expression easier to read and write.

OTOH, I have a hard time understanding the top of
https://svn.nixos.org/repos/nix/nixpkgs/trunk/pkgs/development/interpreters/python-new/2.5/python.nix,
in particular

===
args: with args;
let inherit (lib) optional prepareDerivationArgs concatStringsSep fix;  in

composableDerivation {
  f = args: let attr = lib.prepareDerivationArgs args; in stdenv.mkDerivation (
attr // {
      C_INCLUDE_PATH = concatStringsSep ":" (map (p: "${p}/include")
attr.buildInputs);
      LIBRARY_PATH = concatStringsSep ":" (map (p: "${p}/lib") attr.buildInputs);
    });
  initial = {
     ... attributes ...
  };
}
===

is hard to understand because of all the "non-flat" stuff happening.  E.g.
what's "initial" and what's "f"?

In pkgs/development/interpreters/python-new/2.5/default.nix I see

    pythonMinimal = ( (import ./python.nix) {
      name = "python-${t.version}";
      inherit (p) fetchurl stdenv lib bzip2 ncurses composableDerivation;
      inherit  (p) zlib sqlite db4 readline openssl gdbm;
    });

which is fine, but then

    pythonFull = t.pythonMinimal.passthru.fun {
     name = "python-${t.version}-full";
      cfg = {
        zlibSupport = true;
        sqliteSupport = true;
        db4Support = true;
        readlineSupport = true;
        opensslSupport = true;
        gdbmSupport = true;
      };
    };

In other words, pythonFull is implemented as a post-hoc modification of
pythonMinimal.  That's kind of asymmetric.  Isn't it just as easy to do it like

  pythonFun = cfg: import ./python.nix {
    inherit ... common dependencies ...;
    inherit cfg;
  };

  pythonMinimal = pythonFun {
  };

  pythonFull = pythonFun {
    zlibSupport = true;
    sqliteSupport = true;
    ...
  };

This also scares me a lot:

===
p: # p = pkgs
let
  inherit (p) lib fetchurl stdenv getConfig;
  # withName prevents  nix-env -qa \* from aborting (pythonLibStub is a
derivation but hasn't a name)
  withName = lib.mapAttrs (n : v : if (__isAttrs v && (!__hasAttr "name" v))
then null else v);
in
  withName ( lib.fix ( t : { # t = this attrs
===

It looks like really complicated Nix attribute set hackery.  Is it really
necessary?  In other words I agree with Armijn to Keep It Simple and hide
complexity from the package expressions - the expressions that build stdenv are
pretty complicated (they give me a headache and I wrote them) but at least they
don't bother you if you're just using them.

-- 
Eelco Dolstra | http://www.st.ewi.tudelft.nl/~dolstra/