[Nix-dev] overview about hack-nix and how it works

Sat Aug 6 14:06:27 CEST 2011

Excerpts from Peter Simons's message of Sat Aug 06 13:31:21 +0200 2011:
> do I understand it correctly that you are saying hack-nix can build
> *all* of Hackage? I could pick the latest version of any random package,
> and hack-nix would know how to build it and the build would succeed?
That button to press on "get my job done" does not exist.

So what does it do? (sketchy list):

A) generate package pool
========================

  1) get the index from hackage containing all package versions
    (and calculate hash, this takes time .. [3]) 

  2) filter it keeping latest versions + manually selected ones
     (this can be configure in the ~/.hack-nix/config file)

  The result of 2) is patched [2] (because dependencies in .cabal files are
  very often wrong!) and the result is written to a file.

  This is done by running hack-nix without arguments.

  The result can be found here:
  https://github.com/MarcWeber/nixpkgs-haskell-overlay/raw/master/hackage/hack-nix-db.nix

B) use the pool (the magic?)
===========================
On top of the pool brute force solving of dependencies takes place.
Because having all hackage packages would create a huge solution space I
have to limit it [4]

This pays attention to executables may have additional dependencies
which are not propagated upstream. Thus when a package is built its
guaranteed that no two versions of the same package cause cabal compile
time warnings (or runtime failures).

There are two ways to run the solver:

B1) when developing:
  cd into directory containing cabal file:
  hack-nix --build-env [optional configurations ..]

  the result is a bash script you can source which will add all
  dependencies to PATH so that configure && build just works.
  Additional it'll define TAG_FILES. Then you can configure your editor
  to use them. Tags are generated for all packages automatically which
  is good enough for 90% of all cases (the other 10% being cpp or
  template haskell generated code ..)

B2) in nix(os/pkgs,..)
  [default.nix], line 486, example xmonad
  This way you can use packages in ~/.nixpkgs/config.nix or
  /etc/nixos/configuration.nix when using overlays [6]

Now to reply to your question: No, it won't build all packages on
hackage, because they may not even build whatever configuration you may
choose (eg if they contain perl instead of haskell code ..).
But given the pool it'll tell you whether the cabal dependencies
match and feed deps into cabal builds so that at least this work is
eliminated when maintaining packages. Its still your task to assign
cabal flags (see [default.nix] line 327, darcs example), select
additional not latest version [7], or add additional C dependencies
(mysql, gtk, ..), see [default.nix] line 216.

So if all latest versions on hackage could be build by a single ghc
version I'd happily reply yes to your question. Obviously this
assumption will be false (thinking about 5.000 packages or more, some
being unmaintained).

However very often hack-nix --build-env just does the job without
packaging anything previously.

There are some additional constraints such as packages from the pool
being dropped if ghc ships with basic versions and such. We should talk
about them if this is causing issues.

Nevertheless I have to agree that I picked up a lot of cabal
flag configurations from nixpgks (eg xmonad), because it already worked
there.

I also want to say that coding this kind of stuff in Nix is no fun (no
type system)
So if it breaks it may be some work to fix .. Luckily only small
adjustments had to be done in the past..

Marc Weber

more details and links:

[2]:
See here for a list of cabal file patches:
https://github.com/MarcWeber/nixpkgs-haskell-overlay/tree/master/patches
You can also patch the Haskell code. If you want to create a patch try:
  hack-nix --patch-workflow name-version

[3]:
  That's why I also commit the nix-cache:
  https://github.com/MarcWeber/nixpkgs-haskell-overlay/raw/master/hackage/nix-cache

[4]:
limiting the solution space is done by using latest versions & some
manually selected ones. The second limitation is done by choosing
default cabal flags which sometimes have to be overwritten.

[6]: overlays: They allow moving masses of nix code into external
repositories. Think about it being a simple import passing pkgs.

[default.nix]:
https://github.com/MarcWeber/nixpkgs-haskell-overlay/blob/master/default.nix

[7]: my hack-nix config. It gives you an idea. Note the target-packages
line telling it to add additional versions to the pool.

hackage-index "http://hackage.haskell.org/packages/archive/00-index.tar" # url
all-packages.nix /pr/gitnixdev/nixpkgs/pkgs/top-level/all-packages.nix
# readable description about why some older versions are included:
# scion:
# ghc-syb 0.1.*
# list-tries 0.1.*
# uniplate 1.2.*
# Cabal 1.8.*
# hslogger 1.0.12
#
# syb 0.1.0.3  because of agda
# QuicCheck 2.1  because of agda
# neither because of yesod and persistent
# network-2.2.1.10 because its the latest version without network-bytestring (also because of scion)
# tagsoup 0.11 because of hxt-tagsoup
#
# yi:
# pointedlist >= 0.3.5 && <0.4
# rosezipper >= 0.1 && < 0.2
# unix-compat >=0.1 && <0.2
# uniplate 1.5
# haskell-src-exts 1.9.6
# gtk 0.11.2
# glib 0.11.2
#
# these because of svg which still requires older gtk
# cairo == 0.11.1
# pango == 0.11.1
# plot == 0.1.2.3
# gtk2hs-buildtools 0.11.2: gtk does not build with newer version!
#
# blaze-builder < 0.3 because of hamlet
target-packages TPMostRecentPreferred ["blaze-builder == 0.2.1.4", "gtk2hs-buildtools == 0.11.2", "glib == 0.11.2", "gtk == 0.11.2", "pango == 0.11.1", "cairo == 0.11.1", "haskell-src-exts == 1.9.6", "uniplate == 1.5", "unix-compat == 0.1.2.1", "rosezipper == 0.1", "pointedlist == 0.3.5", "network == 2.2.1.10", "hslogger == 1.0.12", "tagsoup == 0.10", "QuickCheck == 2.1.2", "syb == 0.1.0.3", "HStringTemplate == 0.4.3", "Cabal == 1.8.0.6", "Cabal == 1.4.0.0", "mmap == 0.4", "HaXml == 1.13.3", "uniplate == 1.2.0.2", "ghc-syb == 0.1.*","list-tries == 0.1.*" , "neither == 0.1.0"] "mmap 0.4 has a different API from the more recent 0.52 one - I should patch hashed-storage\n HStringTemplate-0.4.3 because of happstack*" # additional list of non recent packages to be added"
# target-file Just "/etc/nixos/nixpkgs-haskell-overlay/hackage/hack-nix-db.nix" # where to write the result to. Addition
target-file Just "/pr/gitnixdev/nixpkgs-haskell-overlay/hackage/hack-nix-db.nix" # where to write the result to. Addition
test-cabal-files [] # this will be added to the package db. used by test cases
# patch-directory "/etc/nixos/nixpkgs-haskell-overlay/patches" # Path to nix-haskell-repo/patches
patch-directory "/pr/gitnixdev/nixpkgs-haskell-overlay/patches" # Path to nix-haskell-repo/patches
work-directory "/pr/gitnixdev/nixpkgs-haskell-overlay/work-directory"   # source will be put into this directory so that you can write patches easily
nixpkgs-haskell-overlay "/pr/gitnixdev/nixpkgs-haskell-overlay" # Path to nix-haskell-repo/patches
nix-flags ["-j4", "-K"] # -j flag for nix-* commands or -K (keep build) flag
create-haskell-tags TTVim # create-haskell-tags TTVim # TTVim | TTEmacs | TTNone
# parallel parsing of .cabal files"
parallel-parsing 4
# parallel patching using external patch directory. more IO load"
parallel-patching 4
# parallel fetching of hashes"
parallel-fetching 4