[Nix-dev] Python: getting rid of PYTHONPATH in Nixpkgs

[Freddy Rietdijk]

> To solve 2) you could also make a wrapper that combines selected outputs
of the multiple output package without python itself. I think, this is what
texlive does. That way all the burden of wrapping will be limited only to
the multiple output packages.

I'm not sure whether I understand what you mean. What kind of wrapper do
you mean? What does it wrap, and what does it set? The problem with the
multiple outputs is that when you would add each to PYTHONPATH, the extra
outputs won't be importable unless the 'main' output is first. I don't see
how that can be solved. One that sets PYTHONPATH? Or some kind of package

> As for the first point, I couldn’t very well understand what the problem
is. Are you talking about conflict between instances of a package that were
built against different version of the python? Then shouldn’t `--prefix`
override the incompatible version, since, as you said, order matters?

Example with discussion: https://github.com/NixOS/nixpkgs/pull/17749


On Tue, Nov 1, 2016 at 4:12 PM, Dmitry Kalinkin <dmitry.kalinkin at gmail.com>
wrote:

>
> > On 01 Nov 2016, at 06:22, Freddy Rietdijk <freddyrietdijk at fridh.nl>
> wrote:
> >
> > Hi,
> >
> > Currently we use PYTHONPATH a lot in Nixpkgs to let applications and the
> interpreter find Python modules. This typically works fine, but there are
> problems with this method and so I would like to get rid of it and use only
> `python.withPackages` which uses `python.buildEnv`.
> >
> > The two main issues with the use of PYTHONPATH:
> >       • Before we used `--prefix $PYTHONPATH`, but this would leak
> PYTHONPATH into subprocesses, which is especially problematic when both
> parent and child depend on Python but of different versions.  `--set
> $PYTHONPATH` would solve that issue, but that makes it impossible to add
> other (impure) paths, which is an important feature. This also breaks
> alternative shells like `ipython`. This issue is currently solved by
> extending `sys.path` in the Python applications.
> >       • Limits the use of multiple outputs. When moving a module of a
> package into a separate output it becomes problematic to load this again,
> since just adding the module to PYTHONPATH typically doesn't work because
> the order matters. While Python modules are typically small, and build
> fast, rebuilding can take a lot of time in cases like `matplotlib` which
> supports multiple backends and is depended on by quite some packages.
> > A method that is more reliable is building an environment with symbolic
> links to all the modules, and wrapping the applications with a wrapper that
> sets PYTHONHOME. This is exactly what `python.buildEnv` does, and it solves
> both 1) and 2).
> >
> > `PYTHONPATH` is mainly constructed with the Python interpreter
> setupHook. It is used in `buildPythonPackage` for building the package, and
> after installing it is extended so the tests can run. Furthermore,
> `PYTHONPATH` is set by the `setupHook` when using `nix-shell` like
> `nix-shell -p pythonPackages.numpy`.
> >
> > I think we can get rid of the setupHook. For the building we can create
> an env. This would be able to support multiple outputs as inputs, but will
> be more expensive than setting PYTHONPATH. For the tests we do add the
> newly constructed package to PYTHONPATH; there's no way around it and it
> doesn't cause any problems either.
> >
> > The biggest impact will be on how `nix-shell` is used. It won't be
> possible anymore to use it as shown before, instead `nix-shell -p
> 'python.withPackages(ps:[ps.numpy])'` would have to be used. While it is
> possible to keep the setupHook (or use it as a shellHook) it will be
> unreliable in the case of multiple outputs.
> >
> > What do you think about this change? Do you see any problems with it?
> Any other suggestions?
> >
> > Freddy
> > _______________________________________________
> > nix-dev mailing list
> > nix-dev at lists.science.uu.nl
> > http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
> Hi,
>
> I’m far from being an expert in python, but here are my thoughts:
>
> To solve 2) you could also make a wrapper that combines selected outputs
> of the multiple output package without python itself. I think, this is what
> texlive does. That way all the burden of wrapping will be limited only to
> the multiple output packages. As for the first point, I couldn’t very well
> understand what the problem is. Are you talking about conflict between
> instances of a package that were built against different version of the
> python? Then shouldn’t `--prefix` override the incompatible version, since,
> as you said, order matters?
>
> Also, if you remove setupHook, won’t you also lose ability to specify
> python dependencies directly in buildInputs? Put that inside mkDerivation?
>
> Dmitry
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.science.uu.nl/pipermail/nix-dev/attachments/20161101/73f6a830/attachment.html>