[Nix-dev] Python: getting rid of PYTHONPATH in Nixpkgs

[Dmitry Kalinkin]

> On 01 Nov 2016, at 11:56, Freddy Rietdijk <freddyrietdijk at fridh.nl> wrote:
> 
> 
> > To solve 2) you could also make a wrapper that combines selected outputs of the multiple output package without python itself. I think, this is what texlive does. That way all the burden of wrapping will be limited only to the multiple output packages.
> 
> I'm not sure whether I understand what you mean. What kind of wrapper do you mean? What does it wrap, and what does it set? The problem with the multiple outputs is that when you would add each to PYTHONPATH, the extra outputs won't be importable unless the 'main' output is first. I don't see how that can be solved. One that sets PYTHONPATH? Or some kind of package
I had in mind a simple ```combine [blah.foo blah.bar]``` that would produce a derivation with symlinks to both $foo/lib/python2.7/blah/foo and $foo/lib/python2.7/blah/bar, so multiple outputs are combined back and can be importable with the current envHook setting the PYTHONPATH. This is similar to the current python wrapper that additionally throws in the python interpreter itself along with wrapper to set PYTHONHOME.
> 
> > As for the first point, I couldn’t very well understand what the problem is. Are you talking about conflict between instances of a package that were built against different version of the python? Then shouldn’t `--prefix` override the incompatible version, since, as you said, order matters?
> 
> Example with discussion: https://github.com/NixOS/nixpkgs/pull/17749
> 
> 
> On Tue, Nov 1, 2016 at 4:12 PM, Dmitry Kalinkin <dmitry.kalinkin at gmail.com> wrote:
> 
> > On 01 Nov 2016, at 06:22, Freddy Rietdijk <freddyrietdijk at fridh.nl> wrote:
> >
> > Hi,
> >
> > Currently we use PYTHONPATH a lot in Nixpkgs to let applications and the interpreter find Python modules. This typically works fine, but there are problems with this method and so I would like to get rid of it and use only `python.withPackages` which uses `python.buildEnv`.
> >
> > The two main issues with the use of PYTHONPATH:
> >       • Before we used `--prefix $PYTHONPATH`, but this would leak PYTHONPATH into subprocesses, which is especially problematic when both parent and child depend on Python but of different versions.  `--set $PYTHONPATH` would solve that issue, but that makes it impossible to add other (impure) paths, which is an important feature. This also breaks alternative shells like `ipython`. This issue is currently solved by extending `sys.path` in the Python applications.
> >       • Limits the use of multiple outputs. When moving a module of a package into a separate output it becomes problematic to load this again, since just adding the module to PYTHONPATH typically doesn't work because the order matters. While Python modules are typically small, and build fast, rebuilding can take a lot of time in cases like `matplotlib` which supports multiple backends and is depended on by quite some packages.
> > A method that is more reliable is building an environment with symbolic links to all the modules, and wrapping the applications with a wrapper that sets PYTHONHOME. This is exactly what `python.buildEnv` does, and it solves both 1) and 2).
> >
> > `PYTHONPATH` is mainly constructed with the Python interpreter setupHook. It is used in `buildPythonPackage` for building the package, and after installing it is extended so the tests can run. Furthermore, `PYTHONPATH` is set by the `setupHook` when using `nix-shell` like `nix-shell -p pythonPackages.numpy`.
> >
> > I think we can get rid of the setupHook. For the building we can create an env. This would be able to support multiple outputs as inputs, but will be more expensive than setting PYTHONPATH. For the tests we do add the newly constructed package to PYTHONPATH; there's no way around it and it doesn't cause any problems either.
> >
> > The biggest impact will be on how `nix-shell` is used. It won't be possible anymore to use it as shown before, instead `nix-shell -p 'python.withPackages(ps:[ps.numpy])'` would have to be used. While it is possible to keep the setupHook (or use it as a shellHook) it will be unreliable in the case of multiple outputs.
> >
> > What do you think about this change? Do you see any problems with it? Any other suggestions?
> >
> > Freddy
> > _______________________________________________
> > nix-dev mailing list
> > nix-dev at lists.science.uu.nl
> > http://lists.science.uu.nl/mailman/listinfo/nix-dev
> 
> Hi,
> 
> I’m far from being an expert in python, but here are my thoughts:
> 
> To solve 2) you could also make a wrapper that combines selected outputs of the multiple output package without python itself. I think, this is what texlive does. That way all the burden of wrapping will be limited only to the multiple output packages. As for the first point, I couldn’t very well understand what the problem is. Are you talking about conflict between instances of a package that were built against different version of the python? Then shouldn’t `--prefix` override the incompatible version, since, as you said, order matters?
> 
> Also, if you remove setupHook, won’t you also lose ability to specify python dependencies directly in buildInputs? Put that inside mkDerivation?
> 
> Dmitry
>