[Nix-dev] nixpkgs and external package databases

[Marc Weber]

Excerpts from Florian Friesdorf's message of Mon Feb 06 04:10:03 +0100 2012:
> the topic was brought already but I could not find a conclusive answer
> to it - if I missed it, I would be glad if you could point me to it.
Let me keep the story short:
I've posted about it:

http://article.gmane.org/gmane.linux.distributions.nixos/6812/match=marc+eclipse+emacs+vim
Which is the best way? I don't know.

The approach 'create snapshots of hackage/rubyforge' once a day and
commit all data to git works for me - but also requires lots of disk
space and bandwidth.

Because nix is able to make derivations depend on the output of other
derivations you can even do crazy things like this:

  pypi_dump = dump_Py { hash_of_last_mondey_will_be_invalid_due_to_many_updates = "xxx"; };

  plone = create_derivations_for pypi_dump [ "plone" ];

drawback: You do no longer know how many derivations have to build in
advance - to find out you have to build some packages first.
the pypi_dump is likely to be out of date etc.

pro: You can use python native tool to find out about dependencies (may
too slow though)

For hackage the current way "create .nix files" only succeeds by
accident. However this accident is very likely because most other
distributions such as debian can't install many different versions of a
package at the same time. And that was the main reason why I didn't even
try to create .nix files anymore. What am I talking about ?

:: means depends on.

( T depends on both B and D. B and D both depend on C)
T  :: ( B :: C >= 1.0) 
T  :: ( D :: C  = 1.0)

Thus C must be version 1.0

T2 :: ( B :: C >= 1.0)
T2 :: ( E :: C = 2.0) 

Thus C must be = 2.0.

I think you agree that it makes sense to assume that B and E should
depend on the same version of C in both cases (Else you may get into
trouble when requiring files when PYTHONPATH is searched etc).

This this means that you require two version of B:
  B' depnds on C-1.0
  B'' depnds on C-2.0

And that's why the haskell approach may fail in such insane cases (which
are unlikely to happen).

Thus which version of B and C is required depends on whether B is a
dependency of T or T2.

But as I said: its not that likely to happen. But if that case happens I
want code to fail early (before starting to download anything).
That's why the design of my python and ruby overlays are the way they
are.

Why did nobody use such automatic dependency analysis for nixpkgs (c
libraries, gnome, kde, ...) ?

Because maintaining all sets of possible combinations is also a
nightmare. Worse: compilation time.

Eelco told me once he intentionally didn't try to apply
automatic dependency management on nixpkgs (eg the way Eclipse installs
plugins using equinox P2 using SAT solvers).

Think about the SAT solver choosing a different coreutils version by
accident because openoffice compiles with both ..

Summary: I don't know yet which is the best version. Also if you are
working on libraries all those layers in between get into your way.
Eg using ruby you can replace any library using a "git" version easily -
thus you can fix bugs and continue distributing your work easily.

For hack-nix I have such a patch layer: You can add git sources to the
pool - and you can patch dependencies. However when fixing dependencies
it always takes time to recreate the dump file.

> Do you think there is something wrong with that approach
See the stupid case above. If you care about it - you may end up
creating a set of .nix files for each target package.
If it doesn't happen nothing is wrong with it (except redundancy in code
which is not that bad).

Also if I need a library I want it fast. And my attempts to generate
.nix derivations for ruby showed that it may take quite some time.

> Also, if you think the way hackage is handleđ is good, a short note
> would be great.
Well - Peter is happy - it seems to get the job done equally well as
hack-nix for the average case (except that hack-nix also creates tag
files for sources - but that is unrelated)

So IMHO without try and error there will be no success anyway.

> Peter or Andres, would you mind to explain the workflow and technology
> used to keep haskell packages up-to-date or if you've done so already
> give us a pointer to it?
Have a look at the commit logs: there is a tool hackage2nix which reads
the dependencies of cabal files creating the .nix files.

However cabal is kind of nightmare because eg darcs (the version control
system) has more than 8 flags you can enable/disable. Thus the amount of
possible ways to build packages explodes. That's why hack-nix (probably
the same way as hackage2nix) chooses sensible defaults.

Maybe my mail is biased :) Maybe it also gives you an understanding that
the perfect solution may not even exist - it depends on what matters to
you most. (install time, dump time, being accurate, minimizing nixpkgs
noise , ...) there are many things you could think about - even worse:
Tools like Eclipse have their own "package managers".. duplicating all
that effort would be insane in the Eclipse case IMHO - at least unless
we have 100 times more man power.

Yours
Marc Weber