[Nix-dev] python search path and PyQt4&dbus

[Marc Weber]

Excerpts from Yury G. Kudryashov's message of Wed Aug 25 20:39:40 +0200 2010:
> Hi!
> 
> I'm working on d-bus support in PyQt4. See current patch.
> 
> There is one issue that stops me from commiting this patch to SVN: it 
> doesn't work ;)
> 
> PyQt4 installs file $pyqt/lib/python2.6/site-packages/dbus/mainloop/qt.so 
> but if someone writes
> 
> import dbus.mainloop.qt
> 
> then Python finds dbus.mainloop in $pythonDBus/lib/python2.6/site-
> packages/dbus/mainloop/__init__.py, then looks for dbus.mainloop.qt *only* 
> in $pythonDBus, not in full PYTHONPATH.
> 
> Any ideas how to fix it?

There is no trivial fix. All python package were designed to be run from
one directory only. This means everything "magically" works when
assembling packages in ~/.nix-profile - but it fails when configuring
etc, because there are many store paths.

The gtk issue is / was:

import pygtk
pygtk.require('2.0')

now the require adds *one* path by default, which is a store/pygtk*/
path.

The issue is that some other packages such as gnome which depend on
pygtk also install into their pygtk 2.0 dir which pygtk doesn't know
about.

I patched it so that it looks in all NIX_PYTHON_SITES.
NIX_PYTHON_SITES is similar to PYTHON_PATH, but behaves exactly as
PYTHON_USERBASE (which is the correct way to tell python which library
directory should be used). The issue with PYTHON_USERBASE is that it
only supports one directory. So I patched python 2.5 and 2.6.

Why (wasn't?) not PYTHONPATH ? That time I tried that idea the first
time some packages depended on .pth files - which were ignored by
PYTHONPATH. .pth files tell python to look in a subdir for .py modules.
I think its usage decreased - cause this does no longer seem to be a
problem.

The current (hacky) way in nixpkgs is:

- ignore the pygtk.require not adding all paths to sys.path.

- instead manually add those PYTHONPATH which is tedious because
  you always have to add those PATH's by try and error until something
  works.

This worked somewhat reasonable because nobody was using the packages
depending on pygtk (such as pygnome) extensively - so nobody had to
manually add that much PATHs.
Also we only support one gtk version. That allows overriding of the 
pygkt.require('2.0') by manually adding the modules to sys.path in a
"feel lucky" way which didn't seem to break in the past.

If you grep for PYTHONPATH you'll find many examples of this usage which
I consider being suboptimal.

Because I messed up python once I didn't commit. I  still have a local
git branch which at least fixes the pygtk issue in a sane extensible
way.

I don't have to repeat myself - you all know I want a git mirror being
able to officially maintain branches / ideas like this within the Nix
project :-/

So what's the fix? patch it. Do you know how to start a python debugger?

You can also follow the hacky way, 

Anyway: The python mess doesn't end here. What if you want to support
both:  python-3.1 and python-2.6?  http://bugs.python.org/issue2375
I think such a PYTHON3PATH would solve much hassle and prevent many
issues. We should keep that solution in mind.

I'm not a python guru - so my view on this topic may be limited.

Marc Weber