[Nix-dev] Upcoming PyPi URL Scheme Change

[Domen Kožar]

This is now fixed in master, we should also backport to 16.03

Thanks to Freddy:
https://github.com/NixOS/nixpkgs/commit/d5e6a4494a2eb00e52b309fc7a196d84ff8625ec

On Thu, Apr 21, 2016 at 7:27 AM, Dario Bertini <berdario at gmail.com> wrote:

> I also started to write some code to automate discovery of python package
> dependencies.
>
> Unfortunately I haven't had the chance to keep working on it. And some of
> the formats are quite ambiguous  (does the lack of a run_requires key mean
> that we should look for the information somewhere else, or does this
> package have no dependencies?)
>
> I started by writing some code to munge the setup.py files, to extract
> some information from them. Unfortunately it won't be able to work on any
> setup.py  (unless by using something like fuckit.py, ugh)... Also, due to
> some grammar changes, it currently only works with Python3.4
>
> (I wanted to write it for Nix purposes, but the code that I wrote up to
> now is not nix-specific, and I thus chose the pypi4all name)
>
> I'll try to add the few other incomplete changes that I have now, and add
> a couple or tests...
>
> It uses a little bit of the internal pip api, which is not stable (and
> requires a recent enough version of pip+setuptools) , but at least it means
> that it shouldn't be affected by changes like the one in the subject.
>
> You also don't want to be executing this on a trusted machine, since it'll
> fetch stuff from pypi that we don't know in advance if it could be malicious
>
> https://github.com/berdario/pypi4all
>
> On 21 April 2016 07:02:17 BST, Freddy Rietdijk <freddyrietdijk at fridh.nl>
> wrote:
> >Thanks for the update.
> >
> >There are indeed some things we can automate. Before, I experimented
> >with
> >using one of the API's to get out as much metadata as possible. We
> >could
> >also use pypi2nix, which can give for more information, but requires
> >downloading all files.
> >Unfortunately, the old site still uses MD5 so I quit my effort using
> >the
> >API. The new site (https://warehouse.python.org/) uses SHA256 though.
> >
> >If this change in URL scheme is really going to happen I think we
> >should
> >start using the API to find the correct file, version, hash,
> >description
> >and license. Optionally, we should make it possible to run pypi2nix to
> >extract more, and more precise, information.
> >
> >See also https://github.com/NixOS/nixpkgs/issues/11587.
> >
> >On Thu, Apr 21, 2016 at 12:31 AM, Profpatsch <mail at profpatsch.de>
> >wrote:
> >
> >> On 16-04-20 11:41am, Graham Christensen wrote:
> >> > I recently got word that PyPi is changing their URL scheme.
> >> >
> >> > Old example:
> >> >
> >>
> >
> https://pypi.python.org/packages/source/a/ansible/ansible-1.8.2.tar.gz#md5=c2ac0e5a4c092dfa84c7e9e51cd45095
> >> >
> >> > New example:
> >> >
> >>
> >
> https://pypi.python.org/packages/62/18/91f0e5059373e9b87588c2a1c3b4c3c08ee89e0443aa2017469a4cdae41c/SCRY-1.1.2-py2-none-any.whl#md5=a3c636c4e94df1f0644b6917a9c05e67
> >>
> >> This is going to be a lot of work.
> >>
> >> >     Yet another option is to run a sort of "translator" service
> >that can
> >> consume
> >> >     the PyPI JSON API and will output the URLs in whatever format
> >best
> >> suites you.
> >> >     An example of this is pypi.debian.net (which I don't know where
> >the
> >> code base
> >> >     for it is, but the proof of concept I wrote for it is at
> >> >     https://github.com/dstufft/pypi-debian). These translators are
> >> fairly simple,
> >> >     they take an URL, pull the project and filename out of it and
> >then
> >> use the JSON
> >> >     API to figure out the "real" URL and then just simply redirects
> >to
> >> that.
> >>
> >> Maybe it’s time to automate what we can? Similar to Hackage?
> >>
> >> --
> >> Proudly written in Mutt with Vim on NixOS.
> >> Q: Why is this email five sentences or less?
> >> A: http://five.sentenc.es
> >> May take up to five days to read your message. If it’s urgent, call
> >me.
> >> _______________________________________________
> >> nix-dev mailing list
> >> nix-dev at lists.science.uu.nl
> >> http://lists.science.uu.nl/mailman/listinfo/nix-dev
> >>
> >
> >
> >------------------------------------------------------------------------
> >
> >_______________________________________________
> >nix-dev mailing list
> >nix-dev at lists.science.uu.nl
> >http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
> --
> Sent from mobile. Please excuse my brevity.
> _______________________________________________
> nix-dev mailing list
> nix-dev at lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.science.uu.nl/pipermail/nix-dev/attachments/20160426/0abbc88e/attachment.html