[Nix-dev] Python: getting rid of PYTHONPATH in Nixpkgs

[Freddy Rietdijk]

Hi,

Currently we use PYTHONPATH
<https://docs.python.org/2/using/cmdline.html#envvar-PYTHONPATH> a lot in
Nixpkgs to let applications and the interpreter find Python modules. This
typically works fine, but there are problems with this method and so I
would like to get rid of it and use only `python.withPackages` which uses
`python.buildEnv`.

The two main issues with the use of PYTHONPATH:

   1. Before we used `--prefix $PYTHONPATH`, but this would leak PYTHONPATH
   into subprocesses, which is especially problematic when both parent and
   child depend on Python but of different versions.  `--set $PYTHONPATH`
   would solve that issue, but that makes it impossible to add other (impure)
   paths, which is an important feature. This also breaks alternative shells
   like `ipython`. This issue is currently solved by extending `sys.path` in
   the Python applications.
   2. Limits the use of multiple outputs. When moving a module of a package
   into a separate output it becomes problematic to load this again, since
   just adding the module to PYTHONPATH typically doesn't work because the
   order matters. While Python modules are typically small, and build fast,
   rebuilding can take a lot of time in cases like `matplotlib` which supports
   multiple backends and is depended on by quite some packages.

A method that is more reliable is building an environment with symbolic
links to all the modules, and wrapping the applications with a wrapper that
sets PYTHONHOME
<https://docs.python.org/2/using/cmdline.html#envvar-PYTHONHOME>. This is
exactly what `python.buildEnv` does, and it solves both 1) and 2).

`PYTHONPATH` is mainly constructed with the Python interpreter setupHook.
It is used in `buildPythonPackage` for building the package, and after
installing it is extended so the tests can run. Furthermore, `PYTHONPATH`
is set by the `setupHook` when using `nix-shell` like `nix-shell -p
pythonPackages.numpy`.

I think we can get rid of the setupHook. For the building we can create an
env. This would be able to support multiple outputs as inputs, but will be
more expensive than setting PYTHONPATH. For the tests we do add the newly
constructed package to PYTHONPATH; there's no way around it and it doesn't
cause any problems either.

The biggest impact will be on how `nix-shell` is used. It won't be possible
anymore to use it as shown before, instead `nix-shell -p
'python.withPackages(ps:[ps.numpy])'` would have to be used. While it is
possible to keep the setupHook (or use it as a shellHook) it will be
unreliable in the case of multiple outputs.

What do you think about this change? Do you see any problems with it? Any
other suggestions?

Freddy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.science.uu.nl/pipermail/nix-dev/attachments/20161101/a8bcbd63/attachment.html>