[Nix-dev] Re: overview about hack-nix and how it works

Sun Aug 7 18:30:09 CEST 2011

I built leksah yesterday from scratch on debian 6 after being unable to do
so earlier on deb 5. I was surprised how many dependencies there were on
ghc6 and ghc6 plgs and how much ad hackery was necessary to get the
app built.

Nonetheless this should be modelable and it's a situation that's a) not
likely to go away soon, b) must be dealt with, and c) should be well
within the capabilities of the Nix ecosystem.

I'm glad to see this is being talked about and hopefully eventually
addressed in the large, i.e. the issue of lang specific pkg managers
running under Nix (gems, cabal, pear, whatever).

I wouldn't want to try build leksah on NIxOS but I don't want to run
a desktop on NixOS anyway. Similar problems are encountered in
other non-GUI projects though that I might like to run on NixOS.

On Sun, Aug 7, 2011 at 10:45 AM, Marc Weber <marco-oweber at gmx.de> wrote:

> Excerpts from Peter Simons's message of Sun Aug 07 15:23:47 +0200 2011:
> > That would be part (2) then. What do you mean by "brute force solving of
> > dependencies"? How exactly does hack-nix determine which versions can
> > co-exist?
> >
> > Also, it is not obvious to me why this whole procedure requires
> > end-users to run a special tool.
> You've hit the key. Because if you upgrade (or force a specific package
> version) all dependencies can change.
>
> Example:
>
> "->" means depends on
>
> TARGET -> P1,P2
>
> P1 -> B 2.0 or 1.0 -> C
>
> P2 -> B 1.0 -> C
>
>
> If you install P1 B 2.0 will be tried first. Everything is fine.
> Smae for P2
>
> Let's have a look at what the solver does when trying to build TARGET:
>
> It finds TARGET in the pool, then continues resolving P1 first, then P2.
> When resolving P1 it'll also visit deps finding B-2.0 and C.
>
> When trying to resolve P2 it will stop because B-1.0 must not be used
> because when linking it could cause issues. So it'll try seciond option
> for dependency B of P2 which is version 1.0. This time everything will
> pass.
>
> Because things get even more complicated and version 2 or 1 may depend
> uppon ghc version it is impossible to have on cabal-packages.nix only.
> You may need cabal-packages-for-ghc-6.10.nix
> cabal-packages-for-ghc-6.11.nix, ... the nightmare begins.
>
> Even worse if you want to support packages with newer and older HaXML ..
>
> That's why I don't try to generate on set of .nix files. Patching cabal
> to dump .nix expression was my first attempt. But I gave up on it soon.
> Instead this all is done on the fly as needed.
>
> Thus in the example above you'll get two versions of P1 depending on
> whether B 2.0 may be used or not without doing any manual work but
> including B 1.0 and B 2.0 in the pool.
>
> That's why you should run hack-nix for each TARGET once. Because the set
> of dependencies may be different anyway. I agree that it most cases you
> can get along with having one set of packages (cabal-packages.nix
> demonstrates it). But as soon as you want to escape you have to start
> from scratch. And that's when hack-nix starts paying you back each
> minute you've spend on it.
>
> > Why don't you generate plain Nix expressions that can be checked into
> nixpkgs?
> So the short answer is: Because in the worst case you would have to
> create cabal-packages.nix for each target package and maintain them all
> because one needs newer bytestring, but the other don't compile with
> that newer one.
>
> The interesting point to note is than nixpkgs is built around the
> assumption that there is one set of packages you can build almost
> everything with it. Eelco Dolstra told me on irc once that there is no
> dependency solver by design. And I have to agree that it works pretty
> good in almost all cases. More importantly: You'll get deterministic
> results. If hack-nix was able to cope with the huge solution space
> you'd get the same darcs hash by accident only.
>
> If you require workarounds you can do so by overwriting dependencies
> deeply which has been done for webkit in the past. Today
> "applyGlobalOverrides" seems to be the tool to use.
>
> > they would be able to build and install Haskell packages using the
> standard nix-env
> > interface, which seems to be very desirable.
> You can! Have a look at the [default.nix] in the overlay file and look for
> xmonad,darcs,hackNix itself, .. The function exeByName which is used
> will calculate different dependencies for each (which may look like a
> waste of resources - but it was the only way I came up supporting
> everything possible.)
>
> > to use a separate tool to install Haskell packages. Besides, such a tool
> > already exists, namely cabal-install.
> Yes - but it does not know about Nix. And if dependencies are wrong on
> hackage you're doomed. And that happened too often in the past to
> ignore. Plus when I wrote it gtk2hs was not cabalized. Thus hack-nix was
> able to workaround it, cabal was not.
>
> > Am I missing something?
> No, you've done a great job identifying the missing pieces helping you
> understand the whole thing :)
>
> > So when a package cannot be built the very latest version of its
> > dependencies, I'll have to choose the version that ought be used
> > manually. Is that right?
> Correct. You add it to ~/.hack-nix/config and run hack-nix to regenerate
> the package pool. Then you should be fine (hopefully).
>
> In the example above depending on B-1.0 and B-2.0 is considered fine if
> B is used in executable statements only (thus don't propagate to TARGET
> when linking)
>
> I also agree that we should think about adding the haskell platform
> versions (instead of latest?) to the pool because packages are more
> likely to build ..
>
> However if you add too many versions to the pool brute force will kill
> Nix. So the perfect replacement would be doing it the way Eclipse
> Equinox does: use logical solvers which identify bad paths early
> visiting them once only improving speed.
>
> I also had long discussions with Andres Loeh who thinks the design is
> not perfect. AFAIK he didn't came up with something better which can be
> implemented in reasonable time.
>
> Side note: For python and ruby overlays I didn't implement brute force
> solving. Instead if version mismatches are found an error message is
> given and you can force a version manually which is a compromise.
> For Python there are additional issues such as you having to evaluate
> the description .py files to know about dependencies. So a lot of
> dependencies have to be added manually. Ruby is stable, Python is still
> experimental.
>
> In the end you also have to think about whether you want to download
> 50.000 package descriptions (ruby case) if you only want to build 20 out
> of them. I don't have a solution to this problem which applies to ruby,
> python, haskel, cepan (perl), ..
>
> Marc Weber
> _______________________________________________
> nix-dev mailing list
> nix-dev at cs.uu.nl
> https://mail.cs.uu.nl/mailman/listinfo/nix-dev
>

-- 
===============================================
  juan at acm.org     01 716 524 0542   skype; juan.daugherty
===============================================
60011024 Juan Daugherty (Ren Ren-Juan) <juan at acm.org>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.science.uu.nl/pipermail/nix-dev/attachments/20110807/5bdcf9ed/attachment.html 