[Nix-dev] Python: getting rid of PYTHONPATH in Nixpkgs

[Thomas Hunger]

I'm +1 on this because I encountered lots of problems with PYTHONPATH,
especially for programs that have their own module loading logic (e.g.
gunicorn, some Django code).

I don't know what new problems this change will introduce (other than what
you mentioned) but since this is closer to how virtualenv works I suspect
it'll be manageable.

~

On 1 November 2016 at 10:22, Freddy Rietdijk <freddyrietdijk at fridh.nl>
wrote:

> Hi,
>
> Currently we use PYTHONPATH
> <https://docs.python.org/2/using/cmdline.html#envvar-PYTHONPATH> a lot in
> Nixpkgs to let applications and the interpreter find Python modules. This
> typically works fine, but there are problems with this method and so I
> would like to get rid of it and use only `python.withPackages` which uses
> `python.buildEnv`.
>
> The two main issues with the use of PYTHONPATH:
>
>    1. Before we used `--prefix $PYTHONPATH`, but this would leak
>    PYTHONPATH into subprocesses, which is especially problematic when both
>    parent and child depend on Python but of different versions.  `--set
>    $PYTHONPATH` would solve that issue, but that makes it impossible to add
>    other (impure) paths, which is an important feature. This also breaks
>    alternative shells like `ipython`. This issue is currently solved by
>    extending `sys.path` in the Python applications.
>    2. Limits the use of multiple outputs. When moving a module of a
>    package into a separate output it becomes problematic to load this again,
>    since just adding the module to PYTHONPATH typically doesn't work because
>    the order matters. While Python modules are typically small, and build
>    fast, rebuilding can take a lot of time in cases like `matplotlib` which
>    supports multiple backends and is depended on by quite some packages.
>
> A method that is more reliable is building an environment with symbolic
> links to all the modules, and wrapping the applications with a wrapper that
> sets PYTHONHOME
> <https://docs.python.org/2/using/cmdline.html#envvar-PYTHONHOME>. This is
> exactly what `python.buildEnv` does, and it solves both 1) and 2).
>
> `PYTHONPATH` is mainly constructed with the Python interpreter setupHook.
> It is used in `buildPythonPackage` for building the package, and after
> installing it is extended so the tests can run. Furthermore, `PYTHONPATH`
> is set by the `setupHook` when using `nix-shell` like `nix-shell -p
> pythonPackages.numpy`.
>
> I think we can get rid of the setupHook. For the building we can create an
> env. This would be able to support multiple outputs as inputs, but will be
> more expensive than setting PYTHONPATH. For the tests we do add the newly
> constructed package to PYTHONPATH; there's no way around it and it doesn't
> cause any problems either.
>
> The biggest impact will be on how `nix-shell` is used. It won't be
> possible anymore to use it as shown before, instead `nix-shell -p
> 'python.withPackages(ps:[ps.numpy])'` would have to be used. While it is
> possible to keep the setupHook (or use it as a shellHook) it will be
> unreliable in the case of multiple outputs.
>
> What do you think about this change? Do you see any problems with it? Any
> other suggestions?
>
> Freddy
>
> _______________________________________________
> nix-dev mailing list
> nix-dev at lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.science.uu.nl/pipermail/nix-dev/attachments/20161101/c5b5570a/attachment-0001.html>