[Nix-dev] How to set python's module search path (was: nix store: hardlink vs symlink)

[Peter Simons]

Hi Florian,

 > The prefix determines where python searches for site-packages. [...]

my impression is that you are not concerned about the use of symlinks
versus hardlinks, but you're really interested in ways to make Python
find its extra libraries. It looks on first sight as if those two issues
are related, but they are really not. The proper way to make Python find
its extra libraries is through use of environment variables, and there
are two alternative solutions:

 > Python applications with an isolated environment
 > ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 > They call python as:
 > /nix/store/vzpvrymynp4n93bznxha6hadj0ww68vx-python-2.7.1/bin/python
 >
 > the prefix is:
 > /nix/store/vzpvrymynp4n93bznxha6hadj0ww68vx-python-2.7.1
 >
 > and python will load its empty site-packages in there. Modules needed
 > by the application are made available via PYTHONPATH.

This approach has extremely reliable behavior. It is easy to predict how
Python is going to have, because the set of available libraries does not
depend on any external factors, such as the contents of the profile that
Python happens to be installed in. Suppose I wanted Python to know about
the Cheetah library, then would configure

  {
      pythonCheetah = pkgs.pythonFull.override {
        extraLibs = with pkgs.python27Packages; [ cheetah ];
      };
    };
  }

in ~/.nixpkgs/config.nix, install

  nix-env -p /tmp/foobar -iA pythonCheetah

..., and I'd be able to run

  /tmp/foobar/bin/python -c 'import Cheetah'

just fine. That particular approach allows me to control which libraries
Python knows, and which libraries Python *doesn't know*, and Python is
going to behave that way no matter what I other things I do.


 > Python development
 > ~~~~~~~~~~~~~~~~~~
 > You currently can install python modules into a profile, together with a
 > python interpreter (of the same python version) they can build a python
 > environment. Not meant for running applications, but for example
 > development.
 >
 > For this environment to function properly python needs to treat the
 > profile as its home. virtualenv for example currently breaks because of
 > modules coming from different prefixes.
 >
 > If I install ipdb and ipython into a profile, I'd expect them to see
 > all modules I added to the profile.

This approach doesn't work well within Nix, because Nix calls things
directly from their store path (i.e. there is no "profile"), but it's a
fine solution for end users who to install things into their home
profile and just run them without re-configuring their expressions every
time. The behavior is inherently unpredictable, though. When a certain
library is installed in the profile, Python is going to know about it.
When that library is missing, however, Python won't know about it. In
other words, there are hidden dependencies.

One could argue that PYTHONHOME is not the proper way to accomplish this
configuration, though. Personally, I'd rather set

  export PYTHONPATH=$HOME/.nix-profile/lib/python2.7/site-packages

in my ~/.bashrc or something, but that's a matter of taste.

Anyway, both solutions have their individual strengths and weaknesses,
and I believe that we should support both.

The "pythonhomeWrapper" expression seems to be superfluous, though.
There is no need for two different wrapper scripts, so I'd much rather
see that functionality incorporated into the existing wrapper.

Take care,
Peter