[Nix-dev] Re: Python packages in peril

[Marc Weber]

Excerpts from ludo's message of Mon Apr 19 23:31:10 +0200 2010:
> Hello,
> 
> Marc Weber <marco-oweber at gmx.de> writes:
> 
> > Author: MarcWeber
> > Date: 2010-04-19 19:41:41 +0000 (Mon, 19 Apr 2010)
> > New Revision: 21171
> >
> > You can view the changes in this commit at:
> >    https://svn.nixos.org/viewvc/nix?rev=21171&view=rev
> >
> > Removed:
> >    nixpkgs/trunk/pkgs/top-level/python-packages.nix
> 
> You must be joking.  Can you please revert the patch?

No I dislike joking.
In fact I didn't even remove python-packages.nix.
Let me explain:

The situation was:
we have:

python-2.5
python-2.6
python-3 is coming (and most libs are not compatible to python 2 because
  the language changed the syntax a little bit. I'm not a python guru so I
  can't tell details. It was said that python 3 even might split the
  community..)

That's why I renamed python-packages.nix to python2-packages.nix.
The idea was to add python3-packages later because many libraries may
have two different versions targeting python 3.x or 2.x.

So python-packages.nix was fine. You choose which python-2.x version to
pass to that file.
But there have been many other python packages laying around in
all-packages.nix. How to get a version for 2.5 or 2.6?
Copy paste it? That was even done for some packages.

Example:

  setuptools = builderDefsPackage (import ../development/python-modules/setuptools) {
    inherit python makeWrapper;
  };

  setuptools_python26 = builderDefsPackage (import ../development/python-modules/setuptools) {
    inherit makeWrapper;
    python = python26;
  };

I love keeping things short only duplicating code which is necessary.
So setuptools had to go into the python2-packages.nix file



  flup = builderDefsPackage ../development/python-modules/flup {
    inherit fetchurl stdenv;
    python = python25;
    setuptools = setuptools.passthru.function {python = python25;};
  };


What was that?
Of course this passthru thing vanishes when moving this package into
python2-packages.nix as well.

That's why I moved all python packages I saw into
python2-packages.nix so that python-2.5 or python-2.6
(or maybe even python-2.4) is passed only *once*.

Inkscape broke for me in the past. It illustrates the issue:

  inkscape = import ../applications/graphics/inkscape {
    inherit [..]  python pyxml ;
    inherit (pythonPackages) lxml;
    [..]
  };

We pass a pyhton version (2.4 or 2.5) 3 times:
python, pyxml and lxml.

After my change this would become:
  inkscape = import ../applications/graphics/inkscape {
    inherit python;
    inherit (python26Packages) lxml pyxml;
    [..]
  };

(so maybe python should be added to python26Packages as well ?)

Nevertheless there may be some packages which are known to build with
python-2.{4,5,6} only.

That's why I added a attr set which removes those packages which are
known to not building:


  b = builtins.removeAttrs allPythonPackages
          ([ "setuptoolsTrial"
             "simplejson"
           ]
           # these packages don't build with specific python versions..
           ++ (pkgs.lib.optionals (python.libPrefix == "python2.6") [])
           ++ (pkgs.lib.optionals (python.libPrefix == "python2.5")
             ["pycairo" "pygtk" "pyGtkGlade" "rhpl" "pygobject" "pythonSexy"])
          );

The goal I achieved was: Get evaluation errors rather than build time
errors. Eg
simplejson is not defined instead of a build error.

So 
rec {
  pythonLib = {
    buildInputs = [ depA depB ];
    ..
  }
  depA = ..
  depB = ..
}

becomes something like this:

rec {
  b = # remove attrs which are known to not build to cause early failures
    filter ... allPythonPackages;

  allPythonPackages = {

    pythonLib = {
      buildInputs = [ b.depA b.depB ];
      ..
    }

    depA = ..
    depB = ..
  }
}


That's why I introduced yet another style here. I hope it is easy to
understand though.

Why did so many things break? I'd like to know myself. I started this
change some weeks ago and missed some updates. My fault. I didn't test
enough.

Anyway I added a new attr called "all". So you can test that all python
packages build easily:

nix-build -A python25Packages.all
nix-build -A python26Packages.all

Everything except the packages listed above when defining the attr "b"
are known to built on my system (x86_64).
I'm trying to find out why pygtk and pythonSexy broke. I'm really sorry
about this. I didn't want to break the buildfarm for 24h.

Lluís Batlle  told me to run these commands to verify that I don't break
things in the future:

  export NIXOS=${NIXOS:-/etc/nixos/nixos}
  export NIXPKGS_ALL=${NIXPKGS_ALL:-/etc/nixos/nixpkgs}

  # evaluate all packages and dependencies in all-packages.nix:
  nix-env -qa '*' -P --out-path -f $NIXPKGS_ALL/pkgs/top-level/all-packages.nix

  # I still have to digg into code and see what they really do:
  nix-instantiate --readonly-mode $NIXPKGS_ALL/pkgs/top-level/release.nix --show-trace
  nix-env -qa '*' $NIXPKGS_ALL/pkgs/top-level/release.nix
  nix-build -A tarball $NIXOS/release.nix


So if you don't have anything to add I'll put this script on this wiki
page for reference:
http://wiki.nixos.org/wiki/Contributing

I hope you can forgive me for causing that much trouble for so long.
I really only wanted to improve the situation for everybody - because
the python2.6 update hit me two times.

Ludovic asked me to revert my changes, then start the discussion.

If I can't fix the remaining issues within 2 hours I'll revert
everything.

Marc Weber