[Nix-dev] Hackage -> Nix

[Marc Weber]

Hi.

I'd like to announce that I'm about finishing a cabal / hackage
dependency solver written in nix. I still have to do some more testing
and improve much. But basically it works.

The final workflow will look like this:

a) compile hack-nix
b) write .hack-nix config  file:

    hackage-index http://hackage.haskell.org/packages/archive/00-index.tar
    target-file Just "/tmp/hack-nix.nix"
    test-cabal-files ["tests/test.cabal"] # this will be added to the package db. used by test cases
    target-packages TPMostRecentPreferred ["Cabal == 1.4.0.0"] # additional list of non recent packages to be added"
    #target-packages TPAll # all packages

c) run hack-nix. It will create a package database file readable by
  nix. By default it only keeps the latest version for all packages.
  However you can explicitely add some package versions

  The created hack-nix.nix db file is about 800bk!
  It's content looks like this:

  == first package of db ==
    [
    {
      name = "AC-Colour";  version = [1  1  1];  edeps = [];
      ldeps = 
      {
        cdeps = [];
        deps = 
        [
          {i1 = {gte = [4];};  i2 = {lt = [5];};  n = "base";}
          {
            i1 = {gte = [0  1  0  0];};  i2 = {lt = [0  2];};  n = "ghc-prim";
          }
        ];
      };
      sha256 = "0q1cl3xxscakpqyfn2dbbbl7ygshm64h2klxlf156z1s8z94m816";
    }
    ...
    ]
  ====



d) select the packages you'd like to install.
   This looks like this:

      myTest = resolveDependenciesBruteforce {
        globalFlags =  {}; # { base4 = false; };
        packageFlags = {};
        targetPackages = ["happs-hsp"];
        packages = preprocessedHackageInput;
        os = "Linux";
        mkHaskellDerivation = args: removeAttrs args ["src"];
        debugS = true;
      };

  Nix then reads the database file figuring out which dependencies to
  are required to install a package. I tried following the idea behind
  cabal closely. So at this moment you can't have two versions of a
  package within a package tree (exception: executables. deps of
  executables don't propagate. So it is allowed here)

  Usually a cabal file allows to set different flags and allows using
  different versions. The amount of solutions would explode if you
  was'nt limiting the search space eg by choosing sane defaults such as
  - use base-4 everywhere (set flag base4=true)
  - only use latest versions of packages.

  Right now the algorithm which is used is brute force. There may be
  better ones I don't know. I didn't find one within reasonable time.
  
  Currently there is no solution for the example given above. You can
  enable debugging to get some hints which look like this:


  || trace: deps of hslogger-1.0.7 don't match resolved versions.
  || resolved: [base-4.1.0.0] [directory-1.0.0.3]:
  || required: [base-3.0.3.1] [containers-0.2.0.1] [directory-1.0.0.3] [mtl-1.1.0.2] [network-2.2.1.5] [old-locale-1.0.0.1] [process-1.0.1.1] [time-1.1.4] [unix-2.3.2.0]

  hslogger has base >=3 and base < 4. So 4.1.0.0 doesn't match.
  Some dependencies selected base-4.1.0.0 in this run.

  So another run is tried using the flag base4=false ending in:

  || trace: deps of time-1.1.4 don't match resolved versions.
  || resolved: [base-3.0.3.1] [old-locale-1.0.0.1]:
  || required: [base-4.1.0.0] [old-locale-1.0.0.1]

  Now time is causing trouble because it doesn't build with base-3 at all!

  I want to say: We will have to do a lot of patching which usually
  means replacing base < 3 by base < 5 or such.

  Of course performance is bad. It takes about 8 secs here
  to figure out that no solution exists. However usually you select
  base4 = true and things are faster again. The debugging system gives
  you lots of hints telling about which dep branches are crawled over
  and over again failing at the same point. so it should really be
  doable keeping it fast enough to use.

  The cpu I used:
  "Intel(R) Core(TM)2 Duo CPU     E6750 @ 2.66GHz"
  


I haven't compiled anything yet. I made sure my test suite passes all
tests now.

How to proceed? Is anyone else interested in helping packaging parts of
hackage (which has > 1500 packages today) writing some simple patches
etc?

Do we want to have this 800kb database file in nixpkgs ?
(I don't think so because it changes very often)

So how do you think about a overlay structure?

The next tasks are: Extend hack-nix so that writing patches is much
faster. Then see how well this all works in practise.

Somewhen in the future: announce this on haskell-cafe to get some
feedback and maybe some more users (for nix as well)?

Drawbacks:  Maybe you don't want a perfect system. Maybe you want to mix
base-3 and base-4 and a lot of other versions to get stuff compiled
taking the risk seeing segfaults in very rare cases.

I know that a student of Anders Löh has done some work on a similar
solution. However I never saw any code. I stopped asking.

I will provide access to my repos soon after rebasing things and tidying
up parts of the code.

Keep in mind that I need some feedback soon about whether / how to
integrate this into nixpgks.

Marc Weber