[Nix-dev] python packages and dependencies

[Florian Friesdorf]

A non-text attachment was scrubbed...
Name: 0001-wip-pure-python-dependencies.patch
Type: text/x-patch
Size: 2398 bytes
Desc: not available
Url : http://lists.science.uu.nl/pipermail/nix-dev/attachments/20110923/3851a263/attachment.bin 
-------------- next part --------------

Hi Peter,

On Wed, 21 Sep 2011 22:09:02 +0200, Peter Simons <simons at cryp.to> wrote:
> Hi Florian,
> 
>  > buildPythonPackage happily fetches all dependency of a package defined in
>  > setup.py. The dependencies in there are not necessarily fixed to a
>  > specific version. Therefore, it's not guaranteed that one and the same
>  > nix expression produces the same derivation. How is this handled?
> 
> I have noticed that, too, and I'm not about that "feature" either. I assume
> that there is a way (command-line flag) to tell 'distutils' to disable
> network access? Does anyone know how to do that?

Rok Garbas and me are sprinting currently in Arnhem on the Living
Statues Sprint. The attached patch tells easy_install not to install
dependencies, cleans a python package's site-packages and creates an
<eggdirname>_nix.pth file for the python package. We get some
interesting behaviour:

Three environments:

config.nix:
{
  packageOverrides = pkgs:
  {
    good1 = pkgs.buildEnv {
      name = "good1";
      paths = [
        pkgs.python27
        pkgs.python27Packages.simplejson
      ];
    };
    bad = pkgs.buildEnv {
      name = "bad";
      paths = [
        pkgs.defaultStdenv
        pkgs.python27
        pkgs.python27Packages.simplejson
      ];
    };
    good2 = pkgs.buildEnv {
      name = "good2";
      paths = [
        pkgs.zlib
        pkgs.python27
        pkgs.python27Packages.simplejson
      ];
    };
  };
}

The egg in site-packages is loaded without specifying PYTHONPATH:

$ nix-env -p good1 -i good1 && ./good1/bin/python -c 'import sys;import pprint;pprint.pprint(sys.path); print sys.prefix; import site; print site.PREFIXES'
installing `good1'
['',
 '/tmp/cfl/good1/lib/python27.zip',
 '/tmp/cfl/good1/lib/python2.7',
 '/tmp/cfl/good1/lib/python2.7/plat-linux2',
 '/tmp/cfl/good1/lib/python2.7/lib-tk',
 '/tmp/cfl/good1/lib/python2.7/lib-old',
 '/tmp/cfl/good1/lib/python2.7/lib-dynload',
 '/tmp/cfl/good1/lib/python2.7/site-packages',
 '/tmp/cfl/good1/lib/python2.7/site-packages/simplejson-2.1.3-py2.7-linux-x86_64.egg']
/tmp/cfl/good1
['/tmp/cfl/good1', '/tmp/cfl/good1']


The prefix is different and site-packages are not loaded:

$ nix-env -p bad -i bad && ./bad/bin/python -c 'import sys;import pprint;pprint.pprint(sys.path); print sys.prefix; import site; print site.PREFIXES'
installing `bad'
['',
 '/nix/store/vzpvrymynp4n93bznxha6hadj0ww68vx-python-2.7.1/lib/python27.zip',
 '/nix/store/vzpvrymynp4n93bznxha6hadj0ww68vx-python-2.7.1/lib/python2.7',
 '/nix/store/vzpvrymynp4n93bznxha6hadj0ww68vx-python-2.7.1/lib/python2.7/plat-linux2',
 '/nix/store/vzpvrymynp4n93bznxha6hadj0ww68vx-python-2.7.1/lib/python2.7/lib-tk',
 '/nix/store/vzpvrymynp4n93bznxha6hadj0ww68vx-python-2.7.1/lib/python2.7/lib-old',
 '/nix/store/vzpvrymynp4n93bznxha6hadj0ww68vx-python-2.7.1/lib/python2.7/lib-dynload',
 '/nix/store/vzpvrymynp4n93bznxha6hadj0ww68vx-python-2.7.1/lib/python2.7/site-packages']
/nix/store/vzpvrymynp4n93bznxha6hadj0ww68vx-python-2.7.1
['/nix/store/vzpvrymynp4n93bznxha6hadj0ww68vx-python-2.7.1', '/nix/store/vzpvrymynp4n93bznxha6hadj0ww68vx-python-2.7.1']


zlib in contrast to defaultStdenv does not trigger the prefix change and
site-packages is loaded correctly:

$ nix-env -p good2 -i good2 && ./good2/bin/python -c 'import sys;import pprint;pprint.pprint(sys.path); print sys.prefix; import site; print site.PREFIXES'
replacing old `good2'
installing `good2'
these derivations will be built:
  /nix/store/l6nyb7f1rp4m8vmc07d4pq4fnjcq9shm-good2.drv
building path(s) `/nix/store/kpzbwlzcxshf4xbbrfh2c9a1ms4c7mw4-good2'
building /nix/store/kpzbwlzcxshf4xbbrfh2c9a1ms4c7mw4-good2
created 657 symlinks in user environment
building path(s) `/nix/store/1sw7x7pdz1z9bibc2i2b5r50aws1l4kg-user-environment'
created 4 symlinks in user environment
['',
 '/tmp/cfl/good2/lib/python27.zip',
 '/tmp/cfl/good2/lib/python2.7',
 '/tmp/cfl/good2/lib/python2.7/plat-linux2',
 '/tmp/cfl/good2/lib/python2.7/lib-tk',
 '/tmp/cfl/good2/lib/python2.7/lib-old',
 '/tmp/cfl/good2/lib/python2.7/lib-dynload',
 '/tmp/cfl/good2/lib/python2.7/site-packages',
 '/tmp/cfl/good2/lib/python2.7/site-packages/simplejson-2.1.3-py2.7-linux-x86_64.egg']
/tmp/cfl/good2
['/tmp/cfl/good2', '/tmp/cfl/good2']


Theory: python only loads site-packages that have the same prefix.
What is the difference of zlib vs defaultStdenv that could trigger the
different sys.prefix?

regards
-- 
Florian Friesdorf <flo at chaoflow.net>
  GPG FPR: 7A13 5EEE 1421 9FC2 108D  BAAF 38F8 99A3 0C45 F083
Jabber/XMPP: flo at chaoflow.net
IRC: chaoflow on freenode,ircnet,blafasel,OFTC
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
Url : http://lists.science.uu.nl/pipermail/nix-dev/attachments/20110923/3851a263/attachment-0001.bin