[Nix-dev] idea for nixpkgs native virtualenv like python package management

[Marc Weber]

Excerpts from Florian Friesdorf's message of Mon Feb 07 19:11:08 +0100 2011:
> Do you already have some ideas to build upon?
Yes - I did the same for hackage and for ruby packages with some
success. hack-nix is used ony by me and it works fine.
The ruby implementation is very experimental. Its able to build sup
dependencies and rails - howover rails does currently not work because
it depends on an older version of "publish" than the latest.
My current ruby gems import script only cares about the latest versions.

I've been talking to  Cillian  about it and he wants to push this as
well.

Current status is this small script which is able to fetch all packages
from pypi using the xmlproc API - however it takes hours so maybe a
snapshot should be taken once - and then the get updates function should
be used:
http://mawercer.de/~marc/python.py

The hard thing about packaging Haskell, Ruby, Python (Perl ?), ...
(Eclipse plugins?) is that they all are a universe themselves having
packages and dependencies.

IMHO one of the reasons why packaging stuff for nixos/nixpkgs is easy is
that the feature which is provided by nixpkgs being able to install
multiple versions is unique. Thus the world tries to avoid that case
making it also simple for us.

However in the Ruby/ Python (Haskell?) case its easy to get an
individual environment having the dependencies you or a package
requires.

If you try to package everything the usual nixpkgs way you'll soon it
the case that you have to think about creating:

python-packages-for-plone.nix (versions which satisfy plone)

pythor-packages-for-wicd.nix (versions which satisfy wicd)

Now if you have these dependency chains:

A - B(2) - C  - D

A - B(1)  - C' - E

You have to tell C whether it should depend on B(1) or B(2)

I'm not sure how often this case happen in practise. But if it does its
causing trouble.

Deep overrides can solve this. However it still means that you have to
put much effort into maintaining the optimal set of packages.

That's why I'd like to give up this idea of optimal set of packages
satisfying most needs.

The idea is to have a pool of packages. Each package knows about its
dependencies. Then you have a simple solver which tries to take the
package versions out of the pool to build the final environment.

The problem is that the solution space may be huge. In Haskell and Ruby
I implemented a brute force solver. However I also limited the solution
space by only using the latest packages which means that in most cases
there is only one choice. In Haskell you can put older package versions
into the pool explicitly which still works reasonable well for my use
cases. That's what I want to do for ruby as well (?).

If you have such a pool the other problem is that it gets bigger and
bigger. The textfiles alone containing only package names and dependency
information are several 100k.

That's one of the main reason why I'll never ever think about putting it
into the main distribution.

Of course there are alternatives. Have a lazy pool - the internet.
Get package info as needed. (That's what is done for ruby in trunk).
So you run a script "calculate some packages and write nix info".

While it worked good enough for sup in the past it will fail horribly if
you want to target many different packages because you'll end up having
those dep-for-target-X-ruby/python/..-packages.nix files which only appear in
my head right now.

So the plan is: create a snapshot of the package pool found on the
internet - even if this will be a huge pool. Write it into a nix file
and ask nix to try finding a solution in reasonable time. If it takes to
long help by removing packages from pool and retry.

The final usage could look like this:

pool = import internet-dump.nix;

rails = rubyPackagesFromPool pool {
  filter = (p: p.name == "builder" 
            then p.version = 2.x
            else p.isLatest
           )
}

This would allow finding a set of dependencies for target (in this case
it could be ruby rails) with minimal manual effort.

This is a simplification. But it should be enough to get the idea.
Eg Ruby has some funny version selectors etc.

I hope this makes sense to you if not write back again.

Marc Weber