[Nix-dev] Patch for pythonPackages.numpy

[Florian Friesdorf]

Hi Konrad, hi Peter,

thank you for your patch and sorry for the late response, too busy with
other things. Should get better now.

Could you try whether the attached patch also solves building on OS X
(Konrad) and whether matplotlib builds for you (Peter)?

more comments inline:

On Mon, 30 Apr 2012 16:24:18 +0200, Konrad Hinsen <konrad.hinsen at fastmail.net> wrote:
> Peter Simons writes:
> 
>  >  > Building NumPy on MacOS fails with the current state of nixpkgs, and I
>  >  > suspect it can fail on other platforms as well, as the problems I found
>  >  > are not specific to MacOS. With the attached patch, it builds correctly,
>  >  > at least under MacOS.
>  > 
>  > I applied your patch in r33941, but that change caused build failures
>  > for matplotlib on Linux (and probably on other platforms as well which
>  > Hydra doesn't automatically check):
>  > 
>  >   http://hydra.nixos.org/build/2490567
>  > 
>  > Do you by chance know how to fix these problems?
> 
> The error message "You must install numpy 1.1 or later to build" can
> only have one cause: an "import numpy" failed during installation of
> matplotlib.  And that means that the numpy in your Nix store is not on
> PYTHONPATH when matplotlib is built. I haven't yet studied how Nix
> handles Python packages, so I'd better shut up now. Meaning that I
> don't have a solution at this time.

We are reworking python dependency handling in the python branch[1].

Nix figures out runtime dependencies by scanning the installed files of
a package for hashes of other packages. This works very well with
ELF files as they contain the path to their dependencies.

It also works with pth files, which python (or at least
setuptools/distribute) uses to add dependencies to PYTHONPATH. In the
python branch this works nicely with all packages using setuptools.

However, matplotlib does things differently and does not contain
references to its python dependencies.


Solution path:

- all dependencies are handled via pth files

- pth files are explicitly created for all python dependencies that are
  added to propagatedBuildInputs (in contrast to buildInputs).

- dependencies that fail to show up as runtime dependencies by natural
  means are added to propagatedBuildInputs

There is work in progress to create such pth files[2]. I hacked the
create_pth_for_propagated_inputs, it did not work, and I had to move
focus to more urgent matters.

Any help is greatly appreciated.

> Note that matplotlib fails to build under MacOS because some of its
> dependencies fail, so I can't analyze the problem myself for now.
>
> Are you sure that matplotlib builds correctly without my patch for
> NumPy? It doesn't affect the accessibility of NumPy at all, so I don't
> see what could be the problem. It could potentially cause a build
> failure for NumPy, but not for matplotlib.

Maybe the ATLAS stuff in the patch is guilty?

With the attached modified patch (applied to the python branch) numpy
and matplotlib compile fine. Please check whether it solves your problem
and I will commit it.

> BTW, my NumPy patch mainly removes the default directories for library
> searches. Matplotlib needs a similar patch as well, the evidence being
> the line
> 
>    basedirlist is: ['/usr/local', '/usr']
> 
> I'll look into that when I get the dependencies to build.

Great!

regards
florian

[1] https://github.com/chaoflow/nixpkgs
[2] https://github.com/chaoflow/nixpkgs/blob/python/pkgs/development/python-modules/generic/wrap.sh#L65

-------------- next part --------------
A non-text attachment was scrubbed...
Name: 0001-numpy-empty-default-paths-modified-patch-from-Konrad.patch
Type: text/x-patch
Size: 1317 bytes
Desc: not available
Url : http://lists.science.uu.nl/pipermail/nix-dev/attachments/20120617/6b3bc857/attachment-0002.bin 
-------------- next part --------------

-- 
Florian Friesdorf <flo at chaoflow.net>
  GPG FPR: 7A13 5EEE 1421 9FC2 108D  BAAF 38F8 99A3 0C45 F083
Jabber/XMPP: flo at chaoflow.net
IRC: chaoflow on freenode,ircnet,blafasel,OFTC
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
Url : http://lists.science.uu.nl/pipermail/nix-dev/attachments/20120617/6b3bc857/attachment-0003.bin