[Nix-dev] Upcoming PyPi URL Scheme Change

[Dario Bertini]

I also started to write some code to automate discovery of python package dependencies.

Unfortunately I haven't had the chance to keep working on it. And some of the formats are quite ambiguous  (does the lack of a run_requires key mean that we should look for the information somewhere else, or does this package have no dependencies?)

I started by writing some code to munge the setup.py files, to extract some information from them. Unfortunately it won't be able to work on any setup.py  (unless by using something like fuckit.py, ugh)... Also, due to some grammar changes, it currently only works with Python3.4

(I wanted to write it for Nix purposes, but the code that I wrote up to now is not nix-specific, and I thus chose the pypi4all name)

I'll try to add the few other incomplete changes that I have now, and add a couple or tests...

It uses a little bit of the internal pip api, which is not stable (and requires a recent enough version of pip+setuptools) , but at least it means that it shouldn't be affected by changes like the one in the subject.

You also don't want to be executing this on a trusted machine, since it'll fetch stuff from pypi that we don't know in advance if it could be malicious

https://github.com/berdario/pypi4all

On 21 April 2016 07:02:17 BST, Freddy Rietdijk <freddyrietdijk at fridh.nl> wrote:
>Thanks for the update.
>
>There are indeed some things we can automate. Before, I experimented
>with
>using one of the API's to get out as much metadata as possible. We
>could
>also use pypi2nix, which can give for more information, but requires
>downloading all files.
>Unfortunately, the old site still uses MD5 so I quit my effort using
>the
>API. The new site (https://warehouse.python.org/) uses SHA256 though.
>
>If this change in URL scheme is really going to happen I think we
>should
>start using the API to find the correct file, version, hash,
>description
>and license. Optionally, we should make it possible to run pypi2nix to
>extract more, and more precise, information.
>
>See also https://github.com/NixOS/nixpkgs/issues/11587.
>
>On Thu, Apr 21, 2016 at 12:31 AM, Profpatsch <mail at profpatsch.de>
>wrote:
>
>> On 16-04-20 11:41am, Graham Christensen wrote:
>> > I recently got word that PyPi is changing their URL scheme.
>> >
>> > Old example:
>> >
>>
>https://pypi.python.org/packages/source/a/ansible/ansible-1.8.2.tar.gz#md5=c2ac0e5a4c092dfa84c7e9e51cd45095
>> >
>> > New example:
>> >
>>
>https://pypi.python.org/packages/62/18/91f0e5059373e9b87588c2a1c3b4c3c08ee89e0443aa2017469a4cdae41c/SCRY-1.1.2-py2-none-any.whl#md5=a3c636c4e94df1f0644b6917a9c05e67
>>
>> This is going to be a lot of work.
>>
>> >     Yet another option is to run a sort of "translator" service
>that can
>> consume
>> >     the PyPI JSON API and will output the URLs in whatever format
>best
>> suites you.
>> >     An example of this is pypi.debian.net (which I don't know where
>the
>> code base
>> >     for it is, but the proof of concept I wrote for it is at
>> >     https://github.com/dstufft/pypi-debian). These translators are
>> fairly simple,
>> >     they take an URL, pull the project and filename out of it and
>then
>> use the JSON
>> >     API to figure out the "real" URL and then just simply redirects
>to
>> that.
>>
>> Maybe it’s time to automate what we can? Similar to Hackage?
>>
>> --
>> Proudly written in Mutt with Vim on NixOS.
>> Q: Why is this email five sentences or less?
>> A: http://five.sentenc.es
>> May take up to five days to read your message. If it’s urgent, call
>me.
>> _______________________________________________
>> nix-dev mailing list
>> nix-dev at lists.science.uu.nl
>> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>>
>
>
>------------------------------------------------------------------------
>
>_______________________________________________
>nix-dev mailing list
>nix-dev at lists.science.uu.nl
>http://lists.science.uu.nl/mailman/listinfo/nix-dev

-- 
Sent from mobile. Please excuse my brevity.