[Nix-dev] Language-specific package managers

[Malcolm Matalka]

Thanks for the complete response Marc.

For the mean time, the current solution I'm working on is manually
converting packages to Nix.  The translation is pretty straight forward
and I should be able to write a tool to automatically do it.  It's not
optimal but I have been able to get all of the packages I care about
into my local nixpkgs repo and will submit them to the master repo once
I'm satisfied with them.

/M

Marc Weber <marco-oweber at gmx.de> writes:

> I created 
> hack-nix for Haskell, which dumps hackage. contains a brute force
> dependency solver
>
> nixpkgs-ruby-overlay [1] which dumps rubyforge (which is quite usable, but not perfect yet)
> nixpkgs-python-overlay [2] which dumps PyPi (experimental, dependency inforamtion is not complete enough)
> does not do backtracking, if dependencies fails its you having to to
> tell it "try lib-A version 2.0.0".
>
> They are special because they all work on a "dump of packages" creating
> .nix derivations on the fly whereas cabal2nix creates .nix files very
> close to what you find in nixpkgs.
>
> They all create kind of shell script you can source to augment the
> environment variables, so that dependencies are found.
> Thus you can have multiple "sets of packages" for different targen
> porjects within the same user account without conflicts - however you
> always have to load such an environment, eg by
>
>   # run bash with ruby packgaes found in environment "name"
>   ruby-env-name bash
>
> Known additional universes:
>   - perl
>   - java, scala (ivy, maven, sbt)
>   - ....
>
> There are many ideas and ways to implement such.
> Eg for scala/maven/ivy/... one way to think about it would be using the
> store as "installation place", but not using much about the nix* tools
> otherwise.
>
> then sbt build would just store everything in store.
> Other ways are creating .nix files for a target on the fly - such as
> sbt/ivy/mvn create-nix-derivations (which is close to what cabal2nix
> does).
>
> The downside is that you may have to run a tool before you can succeed
> with nixos-rebuild-switch, because not everything may be packaged.
>
> The perfect way would be including a SOT solver in nix, which hasn't
> beeen done yet - and which was not favored by Eelco in the past (maybe
> for good reason). Eg Eclipse plugin system works this way: the SAT
> solver tries to find one working set of dependencies to satisfy the
> setup you want - however the search space may be very big - which is why
> I limit the simple brute force solver used by hack-nix by passing only a
> subset of all packages found on hackage (latest versions & same manually
> selected ones).
>
> While such a generic approach may seem perfect, there are these
> downsides:
>
> - its harder to controls when rebuilds will take place, because small
>   changes in the pool may cause the solver arbitrarely choose a
>   different solution, otherwise its you having to force eg library-A
>   version should be 1.0 like thing.
>
>   and such rebuilds are bad, because its easy to loos track about which
>   combinations actually work, because while constrtaints are fine, they
>   are never complete.
>
>   Thus in any case there will be lots of maintainance effort.
>
> - its also hard and time consuming (for humans and the cpu) to evaluate
>   all solutions over and over again - which may not be the perfect end
>   user experience. Eg you do'nt want wont to wait 30secs for the
>   evaluation to finish just to install "gnu sed"
>
>> translation apps for all language package manager types?  Specifically I
>> am looking at opam, the new ocaml package manager.
> I never used opam.
>
> Can you copy paste a package description with dependency information so
> that we can get an idea about how it actually looks like?
>
> Cabal is kind of "static", but very complex.
> Example:
> http://hackage.haskell.org/packages/archive/darcs/2.8.3/darcs.cabal
> scroll down to "build-depends" which depend on "flags" which are
> automatiacally chosen depending on the ghc version available - but
> flags are also used to enable/disable features or test cases
>
> for python and ruby there are .py or gemspec files.
> The problem is that they may even run python or ruby code - thus
> there may be packages whos dependencies may depend on configuration
> options and or the system which makes it harder to to dump such info
> into something which can be read and used by nix.
>
> [1] ruby: http://gitorious.org/nixpkgs-ruby-overlay
> [2] python: http://gitorious.org/nixpkgs-python-overlay
>
> For ruby and Haskell I also have some code which can create package
> descriptions for dev versions of packages which then can be read by the
> code creating the derivations on the fly.
>
> And then there is stills the question:
> Is it efficient to download 40.000 package descriptions if you need only
> 10? The lazy behaviour of the native package managers for ruby (gem),
> python (eg pip) etc somehow make this question obsolete.
>
> Just think about how many perl packages there are available.
>
> Well - you don't have to download 40.000 packages, cabal2nix only dumps
> the the maintained packages which are selected manually.
> Another solution would be making nix connect to a server to access a
> versioned well known "dump state" about known packages. Then only those
> package infos could be fetched which are required to fullfill would have
> to be fetched.
>
> The ivy case may be not trivial cause you can configure dependencies in
> a transient way which means if you have A < B < C you can configure C to
> modify dependency A or such (I never fully had the time to learn about
> all features).
>
> So the topic is hot - and there is still quite some work left to be
> done.
>
> The cabal2nix way may work well if you need some packages only and is
> easy to understand and to debug.
>
> I hope I was able to shed some light into current state I know about -
> others may know more.
>
> Let's not forgett that eg most xorg packages are generated
> automatically, too. The some additional manual work is required to make
> everything build.
>
> There are more sub universes, such as gnome which is provides a central
> download folder hirarchy you could use to dump package information from.
> Nobody did it yet.
>
> Marc Weber
> _______________________________________________
> nix-dev mailing list
> nix-dev at lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev