[Nix-dev] Re: overview about hack-nix and how it works

[Marc Weber]

Excerpts from Peter Simons's message of Mon Aug 08 01:10:31 +0200 2011:
> just fine. So why would you want to treat Haskell packages differently?
What makes haskell packages different? Its cabal. Its hackage.
You can automate package dependencies. You can't for gnome.
Yes - there should be "hackage" for gnome (= all C/C++ like packages)
and everything would be easier.

> fundamentally different system. Now, I'm sure a package manager that
> works like this can be very useful, but I don't see how this has
> anything to do with Haskell, nor do I believe that this kind of thing
> should be written in the Nix language.
Haskell? See above. Its the cabal system making the difference.
You're right, it should not have been written in Nix. What do you
suggest as alternative? Patch Nix and code it in C++ ? Yes, that would
have been another option. Now I'd even do it. That time I haven't had my
git branches (adding Nix patches as "arbitrary strings as names" patch)

Or what about making Nix call into Haskell via C and reuse cabal's
solver?

> Nix is great for specifying build
> instructions, but it's not a general purpose programming language.
Again - how would you do it? :(
I totally agree. And I told you that its a night mare previously.
This doesn't change the fact that the work has been done and that it
works. If you have time to rewrite it.. go on and do it.

> I don't see how that follows. There is a finite number of ways in which
> the package set can be permuted, so it is clearly possible to configure
> all those combinations in static Nix files.
Yes, you can also write a nix interpreter in brainfuck! But this doesn't
imply that it makes sense to do so. Its about not duplicating code.

Example:

package A depends on
B (if build with ghc-6.x)
C (if build with ghc-7.x)

Now have fun creating a cabal extension writing nix files encoding
this..

Builds in Nix are
> *functions*, so they are really good at being re-used in different
> configurations. Doing these things "on the fly" doesn't gain anything,
You're wrong, it does. Its not me "reusing them" cause the Nix language
is doing it for me. its about my time. Even if its about 10% only -
that's enough to legitimate automating it IMHO. And there is the "human
failure" factor. I know that I don't break anything by changing
configurations. I know that I don't forget to remove dependency inputs
when upgrading packages, ...

> it just causes the tool to be slow and inefficient, because it makes the
> same decisions over and over again, every time it's called.
Unless you want to start testing different configurations (flag settings)
and different ghc versions.

> I trust you mean haskell-packages.nix? Yes, that function works really
> well. It supports a significant number of packages in several
> architectures and several different versions of GHC.
Yet it failed on scion. You didn't knew before building it.
Same for leksah, .. so does it really perform that well?
I mean you usually build both versions: debug and no debug because you
might need them. Then you have to start from scratch because you notice
later that dependencies are wrong?

> Why would anyone want to "escape" that? What would you like to do that
> haskell-packages.nix doesn't allow you to?
Because sometimes you want dev packages. And their dependencies may be
totally different (eg scion no longer depending on syb, ..).

> I beg to differ. haskell-packages.nix is a *function*. That expression
> is extremely generic, and you can customize it for all kinds of
> different situations without ever editing the source code. In the
> presence of override, you can change virtually anything about that
> package set without editing haskell-packages.nix itself.
Agreed. Again - its about my time. I don't want to care about it.

> I don't think that is true. Nixpkgs has built-in support for overrides
> that allow you to customize the package set to your liking.
I'm not talking about svnSupport. I'm talking about: "rebuild everything
and replace gtk libs-2.6 with gtk libs-2.8 in the whole dependency
chain".

And then try to package something else which depends on that and on a
default package. Then the nightmare begins. Happily it doesn't happen
often.
 
>  > Eco Dolstra told me on irc once that there is no dependency solver
>  > by design. And I have to agree that it works pretty good in almost
>  > all cases.
> What you mean by "almost all cases"? What would be a use case that
> doesn't work?
I didn't say "doesn't work" I said "being pretty good at".
Try to package all version of all ghc and packages on hackage manually.
Start using as much overrides as you like (you'll die before having
finished it)
 
> Well, what I meant that I would like users to be able to install Haskell
> packages without having to worry about hack-nix. That tool sounds like
> it's intended for package set maintainers, but not for end users.
That's why I wrote the solver in Nix. Its run automatically in those
cases printing some lines so that you know what its doing because nix-*
evaluations do take some additional seconds.

> existence of hack-nix should not be exposed to end users.
There are two types of users:
- a) the let me try darcs user, they don't care
- b) the let me hack on my library and verify that it still builds with
    ghc-6, ghc-7 and ghc-dev.
    They have simple questions like "Does my package still build if I
    use newest version of Y?" In those cases they change the cabal
    constraints, run hack-nix and they are done.

> Maybe I misunderstand the situation, but I have the impression that I
> would have to install hack-nix before I could install any of the
> packages in that pool.
You have. See B1 B2 in my very first long reply. One is using hack-nix,
the other doesn't.

hack-nix --build-env just encodes cabal dependencies of *.cabal
of the package you're working on in a temporary .nix file so that
dependencies can be resolved. The current package get's version 9999 so
that it always wins over packages found on hackage.

> I believe you are overstating the magnitude of that problem a bit. Cabal
> files on Hackage are usually quite accurate.
We have different experience here. Even tools like alex had to be
patched because they failed due to split base.. Again, I pointed you to
the kind of trivial patches I applied in the repo. Just check them out
and read a couple of them. Most are trivial, I agree.

If you manage dependencies manually you can just pass in the right deps
and be happy. If you have a solver you have to fix them occasionally.

> you can always ask upstream to fix them.
And what? waiting 7 days until you can continue if you want to be done
tomorrow? You're kidding. Because of this issue (interlude being broken
due to a simple typo) I started:
http://haskell.org/haskellwiki/Hackage_wiki_page_per_project_discussion
In that interlude case I waited 2 weeks and wrote two mails. When the
maintainer replied it was fixed fast.

When I want to package something I want to make it work for me and
others immediately if possible.

> you can upload a fixed version yourself. If a Cabal file has bugs, then
> you are by no means "doomed"; there are plenty of ways to remedy the
> situation.
Right. Usually maintainers should do it and they may be on holiday.

> Well, if that is true, then hack-nix is not particularly helpful.
> Figuring out those dependencies is by far the most difficult part of the
> whole task. What would anyone need a dependency resolver for if it
> doesn't resolve those dependencies?
Automate what can be automated: fetch hashes, write .nix code, ..

> doing in haskell-packages.nix already. We don't need a tool that allows
> us to continue to do that manually -- we need a tool that does that for
> us.
If you update a package, do you retest all the others depending on it?
What if they require the older one by cabal constraints? Good luck with
retesting them all over and over again.

Then you updated 10 packages. Then you notice that you also want to give
library Z a try. But because nobody updated Z for 3 years it depends on
many old packages. What do you do then ? Checkout an older revision of
nixpgks 3 years ago? Good luck. Sources will be gone.

You're right, we need a tool which is perfect. hack-nix is only a small
step towards that perfect solution. I can't afford being perfect always
:(

I totally agree that hack-nix improves a lot of small sources of failure
which are not very obvious at the first glance. Eg that you may break
other packages when updating a library. You'll notice the difference if
you need those "other packages" when they are already broken.

If you update the hackage cache latest versions change. This breaks
packages as well. I think that hack-nix is helpful because you can 
fix it faster because its less likely that you break other
packages by adding additional packages to the pool, you never
forget to remove dead dependencies, you don't overwrite packages
versions deeply then noticing that other packages fail after having
waited a minute for the failure to occur, ...

Currently its great because it also does tagging. I agree that this
could be added to haskell-packages.nix as well.

I'd say we've talked enough. You have an idea about what it is.
You also know about its strength and weaknesses. So either you give it a
try or you don't. I'll not bother you again about this. You've done what
I asked you to do which was learning about the difference - thanks for
taking that time.

Marc Weber