[Nix-dev] Language-specific package managers

[Marc Weber]

I created 
hack-nix for Haskell, which dumps hackage. contains a brute force
dependency solver

nixpkgs-ruby-overlay [1] which dumps rubyforge (which is quite usable, but not perfect yet)
nixpkgs-python-overlay [2] which dumps PyPi (experimental, dependency inforamtion is not complete enough)
does not do backtracking, if dependencies fails its you having to to
tell it "try lib-A version 2.0.0".

They are special because they all work on a "dump of packages" creating
.nix derivations on the fly whereas cabal2nix creates .nix files very
close to what you find in nixpkgs.

They all create kind of shell script you can source to augment the
environment variables, so that dependencies are found.
Thus you can have multiple "sets of packages" for different targen
porjects within the same user account without conflicts - however you
always have to load such an environment, eg by

  # run bash with ruby packgaes found in environment "name"
  ruby-env-name bash

Known additional universes:
  - perl
  - java, scala (ivy, maven, sbt)
  - ....

There are many ideas and ways to implement such.
Eg for scala/maven/ivy/... one way to think about it would be using the
store as "installation place", but not using much about the nix* tools
otherwise.

then sbt build would just store everything in store.
Other ways are creating .nix files for a target on the fly - such as
sbt/ivy/mvn create-nix-derivations (which is close to what cabal2nix
does).

The downside is that you may have to run a tool before you can succeed
with nixos-rebuild-switch, because not everything may be packaged.

The perfect way would be including a SOT solver in nix, which hasn't
beeen done yet - and which was not favored by Eelco in the past (maybe
for good reason). Eg Eclipse plugin system works this way: the SAT
solver tries to find one working set of dependencies to satisfy the
setup you want - however the search space may be very big - which is why
I limit the simple brute force solver used by hack-nix by passing only a
subset of all packages found on hackage (latest versions & same manually
selected ones).

While such a generic approach may seem perfect, there are these
downsides:

- its harder to controls when rebuilds will take place, because small
  changes in the pool may cause the solver arbitrarely choose a
  different solution, otherwise its you having to force eg library-A
  version should be 1.0 like thing.

  and such rebuilds are bad, because its easy to loos track about which
  combinations actually work, because while constrtaints are fine, they
  are never complete.

  Thus in any case there will be lots of maintainance effort.

- its also hard and time consuming (for humans and the cpu) to evaluate
  all solutions over and over again - which may not be the perfect end
  user experience. Eg you do'nt want wont to wait 30secs for the
  evaluation to finish just to install "gnu sed"

> translation apps for all language package manager types?  Specifically I
> am looking at opam, the new ocaml package manager.
I never used opam.

Can you copy paste a package description with dependency information so
that we can get an idea about how it actually looks like?

Cabal is kind of "static", but very complex.
Example:
http://hackage.haskell.org/packages/archive/darcs/2.8.3/darcs.cabal
scroll down to "build-depends" which depend on "flags" which are
automatiacally chosen depending on the ghc version available - but
flags are also used to enable/disable features or test cases

for python and ruby there are .py or gemspec files.
The problem is that they may even run python or ruby code - thus
there may be packages whos dependencies may depend on configuration
options and or the system which makes it harder to to dump such info
into something which can be read and used by nix.

[1] ruby: http://gitorious.org/nixpkgs-ruby-overlay
[2] python: http://gitorious.org/nixpkgs-python-overlay

For ruby and Haskell I also have some code which can create package
descriptions for dev versions of packages which then can be read by the
code creating the derivations on the fly.

And then there is stills the question:
Is it efficient to download 40.000 package descriptions if you need only
10? The lazy behaviour of the native package managers for ruby (gem),
python (eg pip) etc somehow make this question obsolete.

Just think about how many perl packages there are available.

Well - you don't have to download 40.000 packages, cabal2nix only dumps
the the maintained packages which are selected manually.
Another solution would be making nix connect to a server to access a
versioned well known "dump state" about known packages. Then only those
package infos could be fetched which are required to fullfill would have
to be fetched.

The ivy case may be not trivial cause you can configure dependencies in
a transient way which means if you have A < B < C you can configure C to
modify dependency A or such (I never fully had the time to learn about
all features).

So the topic is hot - and there is still quite some work left to be
done.

The cabal2nix way may work well if you need some packages only and is
easy to understand and to debug.

I hope I was able to shed some light into current state I know about -
others may know more.

Let's not forgett that eg most xorg packages are generated
automatically, too. The some additional manual work is required to make
everything build.

There are more sub universes, such as gnome which is provides a central
download folder hirarchy you could use to dump package information from.
Nobody did it yet.

Marc Weber