[Nix-dev] A Journey into the Haskell NG infrastructure: Part I

[Peter Simons]

Dear Haskellers `intersect` Nixers,

this is the first installment of a series of postings to describe technical
aspects of the re-factored Haskell infrastructure. When this is all done, I
intend to use these articles to create some kind of *gasp* user documentation
for the Nixpkgs manual. So I encourage everyone to point out of omissions and
technical errors, or to ask for clarification if something is vague or
misleading. I've chosen an FAQ-style approach because I felt that this is
easiest for me to write. Hopefully this translates into the document being easy
to read, too! Here we go ...

Why should I care about this "new infrastructure"?
--------------------------------------------------

  The new code will break evaluation of any Haskell-related configuration you
  may have in ~/.nixpkgs/config.nix or /etc/nixos/configuration.nix.

  Privately generated cabal2nix expressions will cease to compile.

  Installations that rely on ghc-wrapper finding GHC libraries automatically in
  your ~/.nix-profile are obsolete. If you use this approach, you won't be able
  to update your profile anymore.


Duh, I don't want that!
-----------------------

  You can stick to the established 'haskellPackages' set, if you prefer. The
  new code exists separately in 'haskellngPackages', and it's invisible to
  "nix-env -i <package-name>" and "nix-env -u". If you don't make a conscious
  effort to switch, you won't see any of it.


So I'll never have to update if I don't want to?
------------------------------------------------

  My guess is that 'haskellPackages' and 'haskellngPackages' will co-exist for
  a while. Personally, I switched to Haskell NG, though, and I won't maintain
  any packages in the old hierarchy anymore. I suppose other contributors will
  do the same thing. Once you've converted your setup to 'haskellngPackages',
  there's no reason to look back, really.

  In other words, you don't have to convert your setup *right now*, but at some
  point in the future you will probably have to.


Is it difficult to migrate to Haskell NG?
-----------------------------------------

  Users of 'ghcWithPackages' can easily switch back and forth between both
  package sets. Your ~/.nixpkgs/config.nix file probably defines an attribute
  like this one:

  | {
  |   packageOverrides = super: let self = super.pkgs; in
  |   {
  |     haskellEnv = self.haskellPackages.ghcWithPackages (p: with p; [
  |       async attoparsec caseInsensitive fgl GLUT GLURaw haskellSrc
  |     ]);
  |   };
  | }

  Change this definition as follows to switch to Haskell NG:

  | {
  |   provideOldHaskellAttributeNames = true;
  |
  |   packageOverrides = super: let self = super.pkgs; in
  |   {
  |     haskellEnv = self.haskellngPackages.ghcWithPackages (p: with p; [
  |       async attoparsec caseInsensitive fgl GLUT GLURaw haskellSrc
  |     ]);
  |   };
  | }

  There are only two differences: the "ng" bit in the name of the package set,
  and the new 'provideOldHaskellAttributeNames' config attribute. (Note that
  this lives at the top level of ~/.nixpkgs/config.nix -- not inside of
  'packageOverrides'.)

  Users of the ghc-wrapper have to make a greater effort, I'm afraid, because
  ghc-wrapper no longer exists in the NG code. You'll have to switch to
  'ghcWithPackages' first. You may find this article helpful for the process:
  <http://permalink.gmane.org/gmane.linux.distributions.nixos/15161>.


What does 'provideOldHaskellAttributeNames' do?
-----------------------------------------------

  The package 'cabal-install' has the attribute 'cabalInstall' in the old
  package set. In the new one, however, the attribute is now called
  'cabal-install' -- just like the package. provideOldHaskellAttributeNames
  enables a compatibility layer that defins the old camel-case style names on
  top of the new ones. This allows you to use the same configuration for both
  the new and old package sets.

  Once you feel ready to migrate permanently to NG code, you can switch the
  compatibility layer off, to ensure that all your definition use the new names
  consistently.


Why is the ghc-wrapper gone?
----------------------------

  ghc-wrapper's underlying promise is that you can install a random Haskell
  package into your Nix profile, and GHC will pick it up from there
  automatically. Now, this worked reasonably well for many packages, but for
  may other packages it didn't work, and those cases caused us a lot trouble,
  bug reports, and heated discussions in the past, and we ended up adding more
  and more layers of plumbing around xmonad, ghc-mod, and whatnot in an attempt
  to rescue the idea that GHC would somehow find packages automatically without
  being told about them.

  In the NG infrastructure, this whole notion has been abandoned. The proper
  way to install ghc is trough the 'ghcWithPackages' function. It knows about
  exactly those packages that you tell it to. It's slightly more effort to
  define your Haskell environment this way, but what you get in return for this
  extra effort is a totally deterministic compiler that doesn't depend on any
  wrapper scripts, it produces predictable results independent of what else you
  may or may not have installed, and it interacts just fine with cabal-install
  -- something the ghc-wrapper never managed to get right.


When will the new code be available?
------------------------------------

  It already is. 'haskellngPackages' was introduced to 'master' by commit
  54baa53d on 2015-01-07, and it has now been picked up by the "unstable"
  channel.


Why would I want to migrate? What are the advantages?
-----------------------------------------------------

  The NG package set contains the latest version of every package available
  from Hackage, plus some older versions that we need to make legacy stuff
  compile. In total, we deliver in excess of 7,500 packages. Just look at

    https://raw.githubusercontent.com/NixOS/nixpkgs/master/pkgs/development/haskell-modules/hackage-packages.nix

  if you don't believe it. The 'hackage-packages.nix' file is generated
  automatically by the hackage2nix utility -- which I'll describe in detail in
  a later installment --, and it allows us to easily track Hackage on a daily
  basis if we want to.

  Haskell NG supports GHC 7.10.x.

  Haskell NG allows you to configure the hell out of your installation. You can
  add new packages easily, you can build your own customized package sets, you
  can change Cabal flags, compiler arguments, or any other aspect of how a
  package is built by means of the 'overrideCabal' function without editing the
  source of Nixpgs. The file

    https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/haskell-modules/configuration-common.nix

  illustrates this point nicely.

  It's possible to specify deep overrides with 'overrideScope', which I'll
  describe in detail in a later installment. To get a first impression, check
  out

    https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/haskell-modules/configuration-ghc-7.8.x.nix#L61

  for a concrete example of a deep override that replaces the version of mtl
  2.1.x with 2.2.x in the context of the 'amazonka' package family.


Is there a binary cache available for the NG packages?
------------------------------------------------------

  Haskelll NG is built by http://hydra.nixos.org/jobset/nixpkgs/haskell-updates
  and http://hydra.cryp.to/jobset/nixpkgs/haskell-ng. You can get binaries from
  these servers by adding the command line flags

    --option extra-binary-caches http://hydra.nixos.org
    --option extra-binary-caches http://hydra.cryp.to

  to your invocations of nix-build, nix-env, etc.

  The normal binary cache that backs the "unstable" channel does not yet have
  binaries for NG.


Does Haskell NG support compilers other than GHC 7.8.4?
-------------------------------------------------------

  Yes, there is a family of package sets 'haskell-ng.packages.ghcXYZ' that
  define package sets for GHC versions 6.10.4, 6.12.3, 7.0.4, 7.2.2, 7.4.2,
  7.6.3, 7.8.4, and HEAD a.k.a. 7.10.1-rc1. For example, to install a version
  of GHC 7.6.3, define

    ghc763Env = self.haskell-ng.packages.ghc763.ghcWithPackages (p: with p; [
      mtl network parallel parsec QuickCheck random regex-base regex-compat
      regex-posix stm syb text zlib alex cabal-install cpphs happy ghc-paths
      monad-par
    ]);

  in ~/.nixpkgs/config.nix. Note that the compiler-specific package sets do
  *not* have the compatibility layer. Inside of 'haskell-ng.packages.ghcXYZ',
  only the new attribute names exist!

  Furthermore, the package sets for GHC 7.6.3, 7.8.4, and 7.10.1-r1 are the
  only ones that can be expected to work. I have never tested any pre-7.6.3
  compiler. Getting those package sets to work well is almost certainly going
  to be a bit of an effort.


What can I do to help develop Haskell NG?
-----------------------------------------

  Use it!

  Switch your configuration to the 'haskellngPackages' package set and try
  whether you can update your profile to the new version. If you cannot, then
  please report the issue in this e-mail thread so that we can try to figure
  out what's going wrong.

  As of today, we can compile approximately 50% of Hackage successfully. See
  http://hydra.nixos.org/jobset/nixpkgs/haskell-updates for the current state
  of affairs. Obviously, we would like to get that percentage way up! Patches,
  bug reports, and feature requests will help us accomplish that.


Have fun,
Peter