[Nix-dev] Use Function Application To Escape Override Hell

[Peter Simons]

Mathnerd314 writes:

 >> "haskellPackages" provides the foundation that other package sets
 >> override according to their needs. As of today, this approach
 >> requires approximately 25,000 overrides.
 >
 > The vast majority are [...] auto-generated and repeated in each LTS
 > file. Excluding those gives only 648 manually-specified overrides.

You are right. It feels like we are talking about different topics,
though. A conversation about the complexity of our Haskell
infrastructure probably ought to distinguish the following aspects:

 (1) How much effort is necessary to maintain the files inside of
     pkgs/development/haskell-modules?

 (2) How effectively can "nix-instantiate" interpret those files and
     turn them into store derivations that "nix-store" can realize?

 (3) How much effort is it for Nix users to understand, customize, and
     use their Haskell development environment?

Now, you seem to be saying that we generate the vast majority of
overrides automatically with a tool, therefore their existence doesn't
increase (1) -- which is true.

I'm concerned about (3), however. Overrides are too messy for my taste.
Most computer-literate people understand what function application does.
But how many people understand "callPackage"? And more importantly: why
should our users even care about it?

Don't get me wrong: "callPackage" and "override" are super useful and we
should absolutely use them whenever appropriate. Their use in the
Haskell infrastructure has gotten out of hand, though, and I believe
that there are better ways to express the variations between our package
sets than the one we currently use. My instinct tells me that plain
functions are the way to go.


 >> [Examples for differences between package sets].
 >
 > These are all configured in generic-builder.nix and have nothing to
 > do with package sets.

The point I was trying to make is that "haskell.packages.ghcXYZ.foo" and
"haskell.packages.ghcZYX.foo" can vary in many different ways:

 - Both attributes can refer to different versions of "foo".

 - Both packages are almost certainly built with different versions of
   their dependencies

 - The builds may be configured differently with regard to shared
   library support, availability of the regression test suite,
   availability of Haddock documentation, etc.

Two Haskell package sets can be really quite different from each other.
In fact, the same package set can be quite different from itself if you
interpret it on another platform. We have provide this staggering amount
of diversity, but we don't want to do it by replicating tens of
thousands of build expressions in Nixpkgs. We would like to keep Nixpkgs
as small as possible. The answer to that challenge is to take advantage
of Nix's ability to define functions. Currently, we do that, but then
hide the fact because we don't expose the underlying functions. I
believe that we might be able to reduce the complexities (2) and (3) in
Nixpkgs if we re-consider that design choice.


 >> Nix can handle this diversity, because it describes builds in a
 >> Turing-complete functional language.
 >
 > Actually, it describes them in a simple ATerm format (.drv file). The
 > Turing-complete functional language is used to generate the build
 > descriptions.

Yeah, I often use the term "build" in a very general way that is not
particularly accurate. You are right, of course: we use the Nix language
to specify expressions that may evaluate to derivations, which we can
instantiate as store derivations and then realize their output paths, if
necessary. (Yes, I am aware that Eelco spells it "realise", but my
aspell dictionary won't let me do that.)


 >> These kind of overrides are so useful that we've defined a whole set
 >> of helper functions that use this technique to customize the default
 >> definitions from "hackage-packages.nix" in any way we need.
 >
 > The only way to use them from a configuration.nix or config.nix is to
 > write out with import <nixpkgs>/pkgs/development/haskell-modules/lib.nix;

pkgs.haskell.lib works too.


 > Yet another approach is to make "hackage-packages.nix" actually be
 > "hackage-packages.json"; i.e., remove all this function nonsense.

I don't think I can follow. "hackage-packages.nix" defines derivations
that we can build. "hackage-packages.json" clearly doesn't do that --
the file contains just data. How exactly would one go from that JSON
file to, say, an Idris binary built with GHC 7.8.4 and shared libraries
disabled?


 > The package sets then can define their own method of resolving
 > packages to package versions.

Well, defining a good method to resolve package names to package
versions and build variants is exactly what this thread is about! Would
you care to elaborate how exactly that would work for Haskell packages
in your proposal?

Best regards,
Peter