[Nix-dev] Anarchy

[Marc Weber]

Excerpts from Michael Weiss's message of Wed Jun 27 14:02:47 +0200 2012:

I have a hard time following you. Having a global namespace for python
and ruby's 'expect' or 'mechanize' libraries causing kind of failure
doesn't seem to be what we want? Or how would you call them? prefix by
python- or ruby-?

> Intuitively, adding something to a collection should never break what's 
> already there.
Well - you have to understand dependencies:

simplified example:

kernel-3.6 -> glibc-1.0 -> curl-library-1.0 -> firefox-1.0
                          ^ http lib           ^ target application
              ^ system lib

Now what should happen if you add a glibc-1.1?
Does this require resigning of firefox (yes, it does).

Do you want to change firefox executable or just try your new glibc
version with your command line application you might care about right
now?

That's what all issues are about: There are so many different ways to
combine packages and versions - that packages have to make a decision 
and only test some (or only one) combination which is called the
"distributtion". That's what gentoo, mandrake, debian, nixpkgs, ... do.

Why? Because its already enough work to test this one "combination of
glibc, curl, firefox, kernel".
And you also want to install only one version of glibc if you reinstall
your system - for that reason you __want to break "existing packages"__
if you add a new version of glibc - other wise you would still install 10
different kernel versions today just because any small thing was using
it in the past ..

The "don't break any existing stuff" policy you talk about is applied by
many companies - which has these consequences:
  - they are still using technology 20 years ago
  - there is no chance to change it ever - because that would take many
    years (nobody can pay)

In fact its easy to imagine the "perfect world" you're talking about
but in the real world you want to optimize the work (testing,
downloading, writing .nix files, ...) often results in the less perfect
quick and dirty sets of things.

It looks like what you're asking for is a "pool of recipes" signed by
devs which can be used to build up a system using another recipe (you
don't talk about). And that last recipe is the one saying: prefer glibc-1.1
over glibc-1.0 but only when building firefox, not when building mutt
email client.
And its saying: build package foo with expat and not with libxml even
though both are possible - so that the user has to download libxml only.

And that's the "manual work" maintainers do - limit the amount of
possible combinations of package recipes creating a distribution
which is much more likely to work.

In the very long run nixpkgs may be refactored and look like

a) pool of sources

b) pool of recipes knowing how to build the referenced sources of a
for various platforms 

c) a dependency solver which uses packages recipes from b) + additional
info about which combinations to force or ignore yielding the final system.

Now imagine you have such a complex system - you still have an issues:
Its testing. Using the manually written all-packages.nix its very easy
to bisect problems: follow the revisions, rebuild, test .. ah rev 527
caused issues.

If you start thinking about software the way I sketched its much much
harder to determine why something was assembled in a different by a
solver eventually causing breakage.

That's the same reason why kernel is only changed in very small steps -
to make it possible to find bugs.

Nevertheless packages systems such as rubyforge, hackage (haskell) and
even Eclipse (equinox p2) show that such automation is possible, but
evaluation time also increases very very much unless you take effort to
limit the space of solutions (which I did in hack-nix).

The great thing is: nix is still the perfect tool to give this all a try
- because the idea putting each package into its own directory /nix/store/hash-package-name
  using a garbage collector etc - it applies to all different models we
  can think of.


If software distribution and upgrades was so easy less people would work
on it :)

Marc Weber