[Nix-dev] Language-specific package managers

Fri Nov 30 01:27:38 CET 2012

Hi Malcolm,

Malcolm Matalka <mmatalka at gmail.com> writes:
> Thanks for the complete response Marc.
>
> For the mean time, the current solution I'm working on is manually
> converting packages to Nix.  The translation is pretty straight forward
> and I should be able to write a tool to automatically do it.  It's not
> optimal but I have been able to get all of the packages I care about
> into my local nixpkgs repo and will submit them to the master repo once
> I'm satisfied with them.

I like the approach of (semi-)automatically creating nix expressions.

In my understanding (and I am pretty determined but happy to change it
based on good arguments):

A set of nix expressions always produces the very same output.

For me this means (so far) that ad-hoc resolving dependencies within nix
is not needed. I want to resolve dependencies ad-hoc when creating a new
set of nix expressions, but not when I install a package via nix.

Therefore the resolver belongs outside of nix.

If I want to install things ad-hoc in a nix system, I need to create a
bridge between a nix world where everything is determined and a fluent
world where things are installed as they are available. These fluent
things cannot reside in the nix store. Whether such dynamically managed
things belong into /var or a user's home is another discussion.

For nodejs I was quit happy that I got such a dynamic profile via:
% npm config set prefix ~/.nodejs

Now installing things "globally" end up in ~/.nodejs and the bins in
~/.nodejs/.bin, which I included in my path. Perfect quick and dirty
installation of the view nodejs packages I need.

For python we are aiming at something similar: generate nix expression
and check them into git for commonly needed packages, use the rest via
easy_install, but give the people the tool to create nix expressions for
whatever they need.

Looking forward to your ocaml work!

regards
forian

> Marc Weber <marco-oweber at gmx.de> writes:
>
>> I created 
>> hack-nix for Haskell, which dumps hackage. contains a brute force
>> dependency solver
>>
>> nixpkgs-ruby-overlay [1] which dumps rubyforge (which is quite usable, but not perfect yet)
>> nixpkgs-python-overlay [2] which dumps PyPi (experimental, dependency inforamtion is not complete enough)
>> does not do backtracking, if dependencies fails its you having to to
>> tell it "try lib-A version 2.0.0".
>>
>> They are special because they all work on a "dump of packages" creating
>> .nix derivations on the fly whereas cabal2nix creates .nix files very
>> close to what you find in nixpkgs.
>>
>> They all create kind of shell script you can source to augment the
>> environment variables, so that dependencies are found.
>> Thus you can have multiple "sets of packages" for different targen
>> porjects within the same user account without conflicts - however you
>> always have to load such an environment, eg by
>>
>>   # run bash with ruby packgaes found in environment "name"
>>   ruby-env-name bash
>>
>> Known additional universes:
>>   - perl
>>   - java, scala (ivy, maven, sbt)
>>   - ....
>>
>> There are many ideas and ways to implement such.
>> Eg for scala/maven/ivy/... one way to think about it would be using the
>> store as "installation place", but not using much about the nix* tools
>> otherwise.
>>
>> then sbt build would just store everything in store.
>> Other ways are creating .nix files for a target on the fly - such as
>> sbt/ivy/mvn create-nix-derivations (which is close to what cabal2nix
>> does).
>>
>> The downside is that you may have to run a tool before you can succeed
>> with nixos-rebuild-switch, because not everything may be packaged.
>>
>> The perfect way would be including a SOT solver in nix, which hasn't
>> beeen done yet - and which was not favored by Eelco in the past (maybe
>> for good reason). Eg Eclipse plugin system works this way: the SAT
>> solver tries to find one working set of dependencies to satisfy the
>> setup you want - however the search space may be very big - which is why
>> I limit the simple brute force solver used by hack-nix by passing only a
>> subset of all packages found on hackage (latest versions & same manually
>> selected ones).
>>
>> While such a generic approach may seem perfect, there are these
>> downsides:
>>
>> - its harder to controls when rebuilds will take place, because small
>>   changes in the pool may cause the solver arbitrarely choose a
>>   different solution, otherwise its you having to force eg library-A
>>   version should be 1.0 like thing.
>>
>>   and such rebuilds are bad, because its easy to loos track about which
>>   combinations actually work, because while constrtaints are fine, they
>>   are never complete.
>>
>>   Thus in any case there will be lots of maintainance effort.
>>
>> - its also hard and time consuming (for humans and the cpu) to evaluate
>>   all solutions over and over again - which may not be the perfect end
>>   user experience. Eg you do'nt want wont to wait 30secs for the
>>   evaluation to finish just to install "gnu sed"
>>
>>> translation apps for all language package manager types?  Specifically I
>>> am looking at opam, the new ocaml package manager.
>> I never used opam.
>>
>> Can you copy paste a package description with dependency information so
>> that we can get an idea about how it actually looks like?
>>
>> Cabal is kind of "static", but very complex.
>> Example:
>> http://hackage.haskell.org/packages/archive/darcs/2.8.3/darcs.cabal
>> scroll down to "build-depends" which depend on "flags" which are
>> automatiacally chosen depending on the ghc version available - but
>> flags are also used to enable/disable features or test cases
>>
>> for python and ruby there are .py or gemspec files.
>> The problem is that they may even run python or ruby code - thus
>> there may be packages whos dependencies may depend on configuration
>> options and or the system which makes it harder to to dump such info
>> into something which can be read and used by nix.
>>
>> [1] ruby: http://gitorious.org/nixpkgs-ruby-overlay
>> [2] python: http://gitorious.org/nixpkgs-python-overlay
>>
>> For ruby and Haskell I also have some code which can create package
>> descriptions for dev versions of packages which then can be read by the
>> code creating the derivations on the fly.
>>
>> And then there is stills the question:
>> Is it efficient to download 40.000 package descriptions if you need only
>> 10? The lazy behaviour of the native package managers for ruby (gem),
>> python (eg pip) etc somehow make this question obsolete.
>>
>> Just think about how many perl packages there are available.
>>
>> Well - you don't have to download 40.000 packages, cabal2nix only dumps
>> the the maintained packages which are selected manually.
>> Another solution would be making nix connect to a server to access a
>> versioned well known "dump state" about known packages. Then only those
>> package infos could be fetched which are required to fullfill would have
>> to be fetched.
>>
>> The ivy case may be not trivial cause you can configure dependencies in
>> a transient way which means if you have A < B < C you can configure C to
>> modify dependency A or such (I never fully had the time to learn about
>> all features).
>>
>> So the topic is hot - and there is still quite some work left to be
>> done.
>>
>> The cabal2nix way may work well if you need some packages only and is
>> easy to understand and to debug.
>>
>> I hope I was able to shed some light into current state I know about -
>> others may know more.
>>
>> Let's not forgett that eg most xorg packages are generated
>> automatically, too. The some additional manual work is required to make
>> everything build.
>>
>> There are more sub universes, such as gnome which is provides a central
>> download folder hirarchy you could use to dump package information from.
>> Nobody did it yet.
>>
>> Marc Weber
>> _______________________________________________
>> nix-dev mailing list
>> nix-dev at lists.science.uu.nl
>> http://lists.science.uu.nl/mailman/listinfo/nix-dev
> _______________________________________________
> nix-dev mailing list
> nix-dev at lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev

-- 
Florian Friesdorf <flo at chaoflow.net>
  GPG FPR: 7A13 5EEE 1421 9FC2 108D  BAAF 38F8 99A3 0C45 F083
Jabber/XMPP: flo at chaoflow.net
IRC: chaoflow on freenode,ircnet,blafasel,OFTC
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
Url : http://lists.science.uu.nl/pipermail/nix-dev/attachments/20121130/5bb187fd/attachment.bin 