[Nix-dev] python proposal

[Marc Weber]

Hi Eelco, I'd like to start with your last statement:
>  but at least they
> don't bother you if you're just using them.
That's what I've had in mind as well thinking about that many different
python packages that are out there and want to be packaged by someone
:-)

Thanks for telling me that I should tell you more about my motivation..


> identifiers: for instance, a name like "composedArgsAndFun" doesn't tell me a
> lot about what that function is supposed to do, yet I see it everywhere.  And
> this new name "applyAndFun" isn't even grammatical.
How would you name them?
  apply = function application 
  AndFun = also expose function interface to inherit stuff.
At least that is what I had in mind.
If you have a better name I'd like to change it.

> Marc, you gave a few examples, but little *motivation*.  What do they allow us
> to accomplish (better) that we couldn't do before?

Sure. I'll give you one example:
I've tried updating octave because I needed a more recent version to see
wether I could help a friend. I ended in trying different things:
atlas: yes and no
hg development version: yes and no

The dev version needs some extra configuring etc.. So it looks like [1]
now. (It finally builds, but without docs and still fails to install) 
By writing this new style I had in mind making this kind of experiments
more easy without having to clone the whole expression.

You've given this example which is easier to read:
expr = cfg : mkDerivation {...}
exprMinimal = expr {...};
exprMaximal = expr {...};

Now it happens that you want to use a new version which also supports a
new flag or configuration option? How to add another flag without
cloning the whole expression?
You've been totally right that my goal was to make inheriting stuff more
easily.
Using composableDerivation you can just use this and you're done

  myPython = pythonMinimal.passthru.funMerge (a: {
      flags = {
        # new custom flag
      };
      src = { /* my src */ };
      meta = { description = "patched!!" + a.meta.description; };
    }
  )

  That's even short enough to add it using __override to ~/.nixpkgs/config.nix
  But let's not get that complicated: Just assume you want to have a
  python-dev package (from repository, bleeding edge for whatever
  reason).. How can you add to your
  expr = cfg : mkDerivation {..}
  a simple preConfigure commad to run autoconf?
  That's what the more complex functions composedArgsAndFun and
  applyAndFun are about.

While writing prepareDerivationArgs I've had more in mind.
I think there will be a time when some people might think about how to
make nix more attrcative to masses. One missing piece is a installer
providing a gui also supporting configuration options.
Then we should somehow define an interface to tell the gui applications
about what options are availible. And you'll have a much harder time
adjusting all those getConfig occurences to that interface compared to
using one function. But I don't think that prepareDerivationArgs adds
hat much to complexity. So I'll stop talking about that.


> which is included with a "with", making it hard to follow where identifiers come
That's why I've used p. (for all-packages) and t. (for this = defined
within this rec) I've removed with for exactly that reason. You don't
have the list at the top but you can search for p\. and t\. to find out.
I think this makes sense here because python is kind of a sub universe.
And if two people are adding different packages with additional deps you
get merge conflicts in that dependency list { .. ... .. }: { deriv1 = ;
deriv 2 = ;} without benefit.

> then by definition you have no abstraction: the entire internal operation of the
> function is exposed to the caller.)
Why not? I'd say I have kind of abstraction because I can modify some
internal aspects while reusing the existing code only modifying the
parts I have to.


> Where does "version come from?  Why do I have to write all this boilerplate
> "writeScript ... (textClosure [...])"?  And so on.  Such a "layered" expression
Ahm. Let's forward this question to the author..

> OTOH, I have a hard time understanding the top of
> https://svn.nixos.org/repos/nix/nixpkgs/trunk/pkgs/development/interpreters/python-new/2.5/python.nix,
> in particular
> 
> ===
> args: with args;
> let inherit (lib) optional prepareDerivationArgs concatStringsSep fix;  in
> 
> composableDerivation {
>   f = args: let attr = lib.prepareDerivationArgs args; in stdenv.mkDerivation (
>    attr // {
>       C_INCLUDE_PATH = concatStringsSep ":" (map (p: "${p}/include") attr.buildInputs);
>       LIBRARY_PATH = concatStringsSep ":" (map (p: "${p}/lib") attr.buildInputs);
>     });
>   initial = {
>      ... attributes ...
>   };
> }
> ===
> is hard to understand because of all the "non-flat" stuff happening.  E.g.
> what's "initial" and what's "f"?
f = the function creating the derivation. In this case it just adds
  include paths and library paths in a python build system specific way.
  I've copied that part hoping that it just works..
  Different speaking: Here it does the same as composableDerivation but
  after collecting the args it defines C_INCLUDE_PATH and LIBRARY_PATH.
  Maybe I should have rewritten that as well. On the other side it works
  and nobody should bother about it (same as your stdenv case).
initial = initial set of arguments. So this is what you feed into
  mkDerivation. Because I've tried two different pygtk versions I didn't
  pass the name at this point but later on using .passthru.funMerge {
    name = ..
  } which made nix-env -qa \* fail because it found a derivation without
  name. That's what the head was about (filtering those out) and that's
  what made the buildfarm fail because I missed to remove one name..


Another point of view about simplicity and 

pythonMinimal = ( (import ./python.nix) { [..]     });
pythonFull = t.pythonMinimal.passthru.fun { [ all options set ]   };

vs

pythonFun = cfg: import ./python.nix {
pythonMinimal = pythonFun { .. }
pythonFull = pythonFun { .. }

In the second (more simple) case you have to read 3 blocks. In the first
one you to read 2 to see the relation?

Drawn as graph it makes this difference:

A -> min
 `-> full

vs

min -> full

At least the dependency graph is easier in the second case :)

> This also scares me a lot:
> 
> ===
> p: # p = pkgs
> let
>   inherit (p) lib fetchurl stdenv getConfig;
>   # withName prevents  nix-env -qa \* from aborting (pythonLibStub is a
> derivation but hasn't a name)
>   withName = lib.mapAttrs (n : v : if (__isAttrs v && (!__hasAttr "name" v))
> then null else v);
> in
>   withName ( lib.fix ( t : { # t = this attrs
> ===
I hope my explanation above (initial = ) has answered why I've added
this. If you want to propose an easier way to do things go on and let me
know. I'm thankful for this kind of enhancement.

But let's have a closer look at what has caused all this: two different
verions of pygtk (one of them doesn't work and could be removed.. I
didn't knew it at that time)
The reason for not using
pyGTK = gtk : glib : ...;
  is that when you write such a function its final. No way to change
  internals (See top of this mail)

The existing pygtk package did pass glib and gtk using passthru for
depending packages: (simplified)
  pygtk = {
    propagatedBuildInputs = [glib  pygtk];
    passthru = {inherit glib pygtk; }
  };

Using the final function style there is no problem. But how to do this?
You want to abstract:
  pasing glib in passthru and add it to propagatedBuildInputs while
  keeping the ability to change internals?
That's why I had need for fixed. Thats why it looks like this:

pygtkBaseFun = (t.pythonLibStub.passthru.funMerge (a :
    let inherit (a.fixed) glib gtk; in lib.mergeAttrsByFuncDefaults [
    {
      # use glib and gitk and pass it using passthru
    }
    {
    }
    ] );

Now add name and glib and gtk version:
pygtk212 = t.pygtkBaseFun.passthru.funMerge (a : {
  version = "2.12.1";
  name = "pygobject-${a.fixed.pygobjectVersion}-and-pygtk-${a.fixed.version}";
  pygtkSrc = fetchurl { 
    url = http://ftp.acc.umu.se/pub/GNOME/sources/pygtk/2.12/pygtk-2.12.1.tar.bz2;
    sha256 = "0gg13xgr7y9sppw8bdys042928nc66czn74g60333c4my95ys021";
  };
  passthru = { inherit (p.gtkLibs) glib gtk; };
  '';
});

You can repeat the last step for different pygtk versions and still
override preConfig hooks etc..

Anyway I think there will always be some experiments to enhance writing
nix expressions. So can we also talk about how to document them so even
if they cause headache that they won't cause breakdowns :)?

I appreciate talking about it. Maybe we can find one of the best ways to
write expressions which can still be customized easily?
Of course I know that nix is not only about me :-)
I still consider this beeing a proposal. And it surely is better than my
first one. I had to get the code in to be able to remove that.

Sincerly
Marc Weber


attempt to get a more recent octave:
[1]:
{stdenv, fetchurl, g77, readline, ncurses, perl, flex,
 bison, autoconf, automake, sourceByName, getConfig, lib, atlas, gperf,
python, glibc, gnuplot, texinfo, texLive}:

assert readline != null && ncurses != null && flex != null;
assert g77.langF77;

let commonBuildInputs = [g77 readline ncurses perl glibc]; in

stdenv.mkDerivation ({
  NIX_LDFLAGS = "-lpthread";
  configureFlags = "--enable-readline --enable-dl --disable-static --enable-shared";
  meta = { 
      description = "High-level interactive language for numerical computations";
      homepage = http://www.octave.org;
      license = "GPL-3";
    };
} // (
  if (getConfig ["octave" "devVersion"] false) then {
    name = "octave-hg"; # developement version mercurial repo
    src =  sourceByName "octave";
    # HOME is set to $TMP because octave needs to access
    # ${HOME}/.octave_hist while running targets
    # in doc/interpreter.. Maybe this can be done better. This hack is
    # fastest :)
    preConfigure = ''
        # glob is contained in glibc! Don't know why autotools want to
        # use -lglob
        sed -i 's/-lglob//' configure.in
        ./autogen.sh
        export HOME=$TMP
        '';
    buildInputs = commonBuildInputs ++ [ flex bison autoconf automake gperf gnuplot texinfo texLive ]
                  ++ lib.optionals (getConfig ["octave" "atlas"] true) [
python atlas ];
    # it does build, but documentation doesn't.. So just remove that
    # directory
    # from the buildfile
    buildPhase = ''
      sed -i octMakefile \
        -e 's/^\(INSTALL_SUBDIRS = .*\)doc \(.*\)$/\1 \2/' \
        -e 's/^\(SUBDIRS = .*\)doc \(.*\)$/\1 \2/'
      make
    '';
  } else {
    name = "octave-3.1.51";
    src =  fetchurl {
      url = ftp://ftp.octave.org/pub/octave/bleeding-edge/octave-3.1.51.tar.bz2;
      sha256 = "0v0khhpmydyimvdl2rswfd0jrcqa9rhd3cyi60zhqv2hi0bhmkh8";
    };
    buildInputs = commonBuildInputs ++ [ flex bison autoconf automake python ]
                  ++ lib.optionals (getConfig ["octave" "atlas"] true) [ python atlas ];
  }
))