[Nix-dev] Re: overview about hack-nix and how it works

[Marc Weber]

First of all, hack-nix is lacking support for the new conTestSuites
field. Trying to fix it.

> you suggest that Cabal makes a huge difference between Haskell and other
> packages, but I don't believe that is true. Most languages have some
> custom format for build meta data. Perl has it, Python has it, Ruby has
> it, LaTeX has it, and so on. One could even argue that the package
> databases published by Debian, Gentoo, or ArchLinux constitute build
> meta data that can be used to auto-generate Nix expressions.
Yes, but which one to use? What's the difference?
Cabal is a database library authors maintain. That's different form
gentoo, debian, ..

Thus using hackage you have always complete meta data for all versions.
For debian,.. you'll always have meta data for versions they have
packaged only.

Except that, you're right.

Heck, maybe I'm going have a look at extracting gnome dependencies from configure
scripts. Even though it may sound silly it could be a starting point.

> I don't know. Personally, I would have used Prolog because constraint
great. When to run prolog if users want to run nixos-rebuild and get a
working system? 

- option 1:
  Run prolog app before running nixos-rebuild so that .nix files can be
  gnerated

- option 2:
  create builtins.system("..") backdoor, pass pool and read result of
  solving ..

- option 3:
  embed prolog interpreter for C (does this exist?)

> I would just call "cabal install --dry-run" and parse the output it
> generates:

This would be (option 1). You'd force users to call the cabal/prolog app
for each haskell piece they'd want to install. Error prone, you can
forget about it. No go for me.

>  If you have time to rewrite it.. go on and do it.
> Actually, that effort is well underway.
How will the design look like? Does the project already have a name?

>  >> There is a finite number of ways in which the package set can be
>  >> permuted, so it is clearly possible to configure all those
>  >> combinations in static Nix files.
>  > Yes, you can also write a nix interpreter in brainfuck!
> I am sorry, but that is not a helpful response. Do you want to have a
> serious conversation or not?

Well, it looks like you don't have an idea about how many finite ways
there are.
Take darcs: How many flags does it have?

I extracted them, there are 12:

  curl, http, static, terminfo, threaded, type-witnesses, library, color, mmap, test, hpc, deps-only


This means that there are 2 ** 12  = 4096 different ways to build darcs
(assuming that dependencies don't change).

It also depends on bytestring >= 0.9.0 and < 0.10

        0.9, 0.9.0.1, 0.9.0.2, 0.9.0.3, 0.9.0.4, 0.9.1.0, 0.9.1.1,
        0.9.1.2, 0.9.1.3, 0.9.1.4, 0.9.1.5, 0.9.1.6, 0.9.1.7, 0.9.1.8,
        0.9.1.9, 0.9.1.10

So there are 16 different bytestring darcs could be built with.
Thus you already have 4096 * 16 = 65536 ways to build darcs using
different flags and different versions.

Now let's just imagine darcs depends on a second library like bytestring
which also has 16 different verions it can be built with. Then you can
multiply with 16 again.

You're right, the number is finite. But finding a way to build the
package (which in the worst case could be trying them all) is non
trivial. Luckily you often find a working version much faster.

Still its silly to talk about encoding 16 * 65536 version (for darcs
only) in nix files.

Not sure what you meant exactly by "permuting".

>  >> Builds in Nix are *functions*, so they are really good at being
>  >> re-used in different configurations. Doing these things "on the fly"
>  >> doesn't gain anything,
>  > You're wrong, it does.
> Do you have any specific examples to illustrate that assertion?
The difference is those 10% I don't have to spend on thinking about
writing those nix files.

>  > I know that I don't break anything by changing configurations. I know
>  > that I don't forget to remove dependency inputs when upgrading
>  > packages, ...
> I don't understand how hack-nix could possibly guarantee these traits in
> a way that's superior to, or even different from the mechanisms in Nix.
Its because I outsourced the manual task to a script which does this for
me. Its the same reason why you use computer to calculate sums. If they
are programmed correctly they make less mistakes than human.

Nix does not know about dependency constraints. Its you passing them
manually by setting buildInputs. Thus you can pass versions and the
./Setup configure script will tell you 'no version found, I need 2.8 of
bytestring'.

If you use hack-nix the solver will tell you before even starting the
build because it knows about .cabal file contents.

> You said that the package set is determined by a configuration file
> that's written manually?
Yes. But the "text" you write is only a guide telling hack-nix which
packages to pick. It then encodes the cabal contents in a Nix readable
way.
 
> GHC is a parameter to haskell-packages.nix. All the expressions in there
> can be instantiated with any other version of GHC. I don't see how
> hack-nix improves the situation on that.

Heck, Because cabal allows you to change dependencies depending on ghc
version:


if (ghc > 7.1)
  BuildInputs: base-4.0, bytestrings, filesystem
else
  BuildInputs: split-base3.0, posix

I agree that such an extreme example may not exist.
Now you test your haskell-packages with ghc-7.1 and it works.
Then you pass ghc ghc-6.12.3 and it'll expect split-base and posix.

(In those cases you're lucky by accident because posix ships with GHC
AFAIK as well as base and bytestring today. But this is by accident!
You could replace them by any other library found on hackage. And then 
you have to start writing conditional expressions manual -> have fun
maintaining that .. )

hack-nix reads those cabal files and reduces those build dependencies
depending on ghc versions. So it'll know about how to pass
posix in the one case and filesystem in the other.

The less mainstream you are the more likely you're going to hit such
cases. And I don't want to spend my time on troubleshooting them.
That's why I'm happily waiting 10secs on the solver instead.

> Can hack-nix build Scion?
If you tell the solver to use ghc-6.12.3 and if you use sources from
hackage, sure. It would require me adding older multiset and time to the
package pool.

> Again, this build problem is totally unrelated to the issue we are
> discussing.
Reread my previous mail. There have been two failures.
Search for headline:

 "compared to cabal-packages.nix"

quoting myself:
    second attempt haskellPackages_ghc6123.leksah

      Configuring ltk-0.8.0.8...
      Setup: At least the following dependencies are missing:
      glib >=0.10.0 && <0.12, gtk >=0.10.0 && <0.12

In those cases hack-nix is likely to give evaluation failures rather
than build time failures.

If I had fixed that I would have found the second issue (gtksourceview
being too old)

> Could you please be a little more specific? What would you like to do
> that haskell-packages.nix doesn't allow you to?
use scion git version for instance:

How do I do it?
I add these lines to pkgs/additional-packages.nix:

  # REGION HACK_NIX:   { name = "scion"; type = "git"; url = "git at github.com:MarcWeber/scion.git"; branch = "resurrect-vim"; }
  # END

Then I run

  # nix-repository-manager 1 . --update-then-publish scion

which will checkout scion, create a tar snapshot, upload the snapshot to my
server and fill in region contents (the dump of the cabal file).
Then the region looks like [2].

Then I add a top level name to [default.nix]:

    # haddock fails because it can't find transitive dependencies, thus disable it.
    scion = { noHaddock = true; };

    # force ghc-7 because nixpkgs defaults to it
    scion = exeByName { haskellPackages = pkgs.haskellPackages; name = "scion"; };

Then I can install scion by

  nix-env -iA haskellOverlay.scion

Of course I could do this using haskell-packages. But it would require more time.

> As you know, that is quite possible.
Its also possible to use brainfuck for an Nix implementation. It would take
much more time though.

>  > I didn't say "doesn't work" I said "being pretty good at". Try to
>  > package all version of all ghc and packages on hackage manually.
Sorry. I meant "imagine try packaging .." sorry about that.

> done what you suggest I should try, and if you've been reading my
> messages, then you should know that.
I know and I'm still recalling it.

> So, would you like to answer my question about a specific use case, or
> are you just being evasive?
I did, see above :)

> Do I understand it correctly that I can install a Haskell package via
> hack-nix without having to worry about hack-nix at all? That would be
> great, indeed.

> I'd be eager to try that out. Could you please post an
> example of a command line I could execute to install, say, pandoc?

See exeByName (scion dev example) above. You have to add this first. And then
you need the Nix patches which allow me to write:

    {
      "foo-2.0" = "data";
    }
Without that I would have to use listToAttrs all the time.


Then you can start after checking out 

- nixpkgs  (my branch adding info about core packages because they are dropped
            from the pool before solving starts)

- nixpkgs-haskell-overlay (because it contains the solver and the pool)

> Marc, please don't be silly. There are not just two types of users. That
> notion is ridiculous. There are hundreds of different types of users,
> because every user is a unique human being with unique intentions and
> goals.
sorry about my bad wording. I meant: I'm thinking about two use cases users are
likely to use. One is using haskell executables, the others are writing haskell software.
Of course the same user can do both.

>  > Automate what can be automated: fetch hashes, write .nix code, ..
> Huh? You say that everything that can be automated should be, but you
> say that in response to my complaint that dependency resolution has
> *not* been automated. What does that mean?
We've been talking about dependency resolution for haskell packages (hack-nix
solver), and we've been talking about manually setting them in nixpkgs.
So I'm not quite sure what you mean now. Maybe its clear what I've been talking
about now.

Maybe it also helps me telling you that I was talking about different things
when saying "hack-nix". Its
  - the whole idea
  - the executable parsing cabal files (using the API which you called trivial)

The hack-nix solver is written in Nix and does not depend on hack-nix executable.
It's using hack-nix's output (See [2] for example) only.

> Now, if hack-nix would be able to resolve build input dependencies
> automatically, then it would be very useful indeed.
It does. That's the whole point :) But there are smarter implementations
then brute force I used.. Its a draft implementation which works.

Marc Weber

[2]:

  # REGION HACK_NIX:   { name = "scion"; type = "git"; url = "git at github.com:MarcWeber/scion.git"; branch = "resurrect-vim"; }
  {
    name = "scion";  version = "0.3";
    edeps = 
    [
      {
        cdeps = [];
        deps = 
        [
          {n = "scion";}
          {i1 = {gte = "0.2";};  i2 = {lt = "0.3";};  n = "atto-lisp";}
          {
            i1 = {gte = "0.8.5.1";};  i2 = {lt = "0.9";};  n = "attoparsec";
          }
          {i1 = {gte = "4.2";};  i2 = {lt = "4.4";};  n = "base";}
          {i1 = {gte = "0.9";};  i2 = {lt = "0.10";};  n = "bytestring";}
          {i1 = {gte = "0.1";};  i2 = {lt = "0.3";};  n = "multiset";}
          {i1 = {gte = "0.11";};  i2 = {lt = "0.12";};  n = "text";}
          {n = "json";}  {n = "derive";}  {n = "network";}
        ];
      }
      {
        cdeps = [];
        deps = 
        [
          {n = "scion";}
          {i1 = {gte = "4.2";};  i2 = {lt = "4.4";};  n = "base";}
        ];
      }
    ];
    ldeps = 
    {
      cdeps = 
      [
        [
          {or = 
           [
             {compilerFlavor = "GHC";  versionRange = {gte = "6.11.20081113";};}
             {
               compilerFlavor = "GHC";
               versionRange = {i1 = {gte = "6.10.2";};  i2 = {lt = "6.11";};};
             }
           ];}
          {cdeps = [];  deps = [];}
        ]
        [
          {
            compilerFlavor = "GHC";
            versionRange = {i1 = {lt = "6.11";};  i2 = {gte = "6.10";};};
          }
          {cdeps = [];  deps = [];}
        ]
      ];
      deps = 
      [
        {i1 = {gte = "4.2";};  i2 = {lt = "4.4";};  n = "base";}
        {i1 = {gte = "1.8";};  i2 = {lt = "1.12";};  n = "Cabal";}
        {i1 = {gte = "0.3";};  i2 = {lt = "0.5";};  n = "containers";}
        {i1 = {gte = "1.0";};  i2 = {lt = "1.2";};  n = "directory";}
        {i1 = {gte = "1.1";};  i2 = {lt = "1.3";};  n = "filepath";}
        {i1 = {gte = "6.12";};  i2 = {lt = "7.2";};  n = "ghc";}
        {i1 = {lt = "0.2";};  i2 = {gte = "0.1";};  n = "ghc-paths";}
        {i1 = {gte = "0.1";};  i2 = {lt = "0.3";};  n = "multiset";}
        {i1 = {gte = "1.1";};  i2 = {lt = "1.3";};  n = "time";}
        {i1 = {gte = "0.11";};  i2 = {lt = "0.12";};  n = "text";}
        {i1 = {gte = "1.0";};  i2 = {lt = "1.1";};  n = "process";}
        {i1 = {gte = "0.2";};  i2 = {lt = "0.3";};  n = "unix-compat";}
        {i1 = {gte = "0.9";};  i2 = {lt = "0.10";};  n = "bytestring";}
        {i1 = {gte = "0.5";};  i2 = {lt = "0.6";};  n = "binary";}
        {i1 = {gte = "1.0";};  i2 = {lt = "1.1";};  n = "old-locale";}
        {i1 = {gte = "2.3";};  i2 = {lt = "2.4";};  n = "network";}
        {i1 = {lt = "1.2";};  i2 = {gte = "1.1";};  n = "temporary";}
      ];
    };
    srcFile = (fetchurl { url = "http://mawercer.de/~nix/repos/scion-git-826da.tar.bz2"; sha256 = "4b00cf1203f4dece73b27bd2e7c0426b3ed0e3a7510cc8837db50cffc7a08067"; });
  }
  # END