[Nix-dev] Use Function Application To Escape Override Hell

[Mathnerd314]

On Wed, Oct 14, 2015 at 8:46 AM, Peter Simons <simons at cryp.to> wrote:
> The Problem
> -----------
>
> override: "haskellPackages" provides the foundation that other package
sets
> override according to their needs. As of today, this approach requires
> approximately 25,000 overrides.
I count 443442 overrides (roughly 8790 packages * 50 package-sets). The
vast majority are of the form
  "<pkg>" = dontDistribute super."<pkg>";
which are autogenerated and repeated in each LTS file.
Excluding those gives 24634 overrides, but once again most are
autogenerated; they are of the form
  "<pkg>" = doDistribute super."<pkg>_<version>";
Excluding those gives only 648 manually-specified overrides. So 99.8% of
the overrides come from two lines [1]. In total there are only 10559 unique
overrides; 8023 of these are dontDistribute, 1989 are doDistribute, and 587
are manual. There are ~767 packages which actually use different versions,
out of Stackage's 1100 packages.

Methodology:
Commit 0a28867303054bbbe504f1c84a382a201581c059 was used.
"override" was approximated as "a reference to super in
pkgs/development/haskell-modules/". Non-autogenerated overrides were
examined by hand and 139 spurious or duplicate results were found (69
unique) [2]. The master command from which other commands were derived was
  git grep super . | grep -v '"\([^"]*\)" = dontDistribute super."\1"' |
grep -v '"\([^"]*\)" = doDistribute super."\1_[^"]*"' | sort -u -t: -k2 |
sort | wc -l
The number of Hackage packages (8790) is from [3].

> Furthermore, we use 3 different types of
> overrides ("override", "overrideCabal", and "overrideScope") that have
subtly
> different purposes and capabilities.
Don't forget overrideDerivation; there are 4 types.

> For example, package sets intended for use with GHC
> versions prior to 7.4.x neither build nor run regression test suites,
because
> Cabal introduced that feature only in later versions of the compiler.
Package
> sets compiling with GHC versions prior to 7.8.x don't support shared
library
> builds, because that feature never worked reliably until more recent
versions
> of the compiler. And builds running on Darwin don't generate the Haddock
> documentation when compiling with GHC versions 7.6.x or earlier, because
the
> Haddock binary on that platform used to interact poorly with Darwin's
"cpp" and
> thus failed until that bug was eventually fixed in GHC 7.8.1.

These are all configured in generic-builder.nix and have nothing to do with
package sets.

> Nix can handle this diversity, because it describes builds in a
Turing-complete
> functional language.

Actually, it describes them in a simple ATerm format (.drv file). The
Turing-complete functional language is used to generate the build
descriptions.

> Builds that exist in multiple variants are functions that

Again, they are not functions. At most they are Nix attribute sets.

> The latest version, 0.4.1, is simply called "hslua".

That was before the Stackage stuff. Now it is the version in Stackage
Nightly that has no suffix.
For example, HUnit is at 1.2.5.2, while HUnit_1_3_0_0 is available, due to
[4].

-----------
I hereby request a configuration-latest.nix which overrides the packages to
actually use the latest versions.
-----------

> It's a mind-boggling way to define a recursive attribute set "rec { ...
}" without actually using the "rec" keyword.

The basics are explained in [5]; including that in the Nixpkgs manual might
be helpful.

> These kind of
> overrides are so useful that we've defined a whole set of helper functions
> that use this technique to customize the default definitions from
> "hackage-packages.nix" in any way we need.

The only way to use them from a configuration.nix or config.nix is to write
out
  with import <nixpkgs>/pkgs/development/haskell-modules/lib.nix;

> One possible approach is to take "callPackage" out of
"hackage-packages.nix".
> Instead of derivations, "hackage-packages.nix" could provide functions
that
> package sets then call with appropriate parameters to define the actual
build.

Yet another approach is to make "hackage-packages.nix" actually be
"hackage-packages.json"; i.e., remove all this function nonsense.
  "hslua": {
       "pname": "hslua",
       "version": "0.4.1",
       "sha256":
"2df2b4f0566ef2244506f9830e0207fce3bece7c331129f69f446c722136173f",
       "configureFlags": [ "-fsystem-lua" ],
       "libraryHaskellDepends": [ "base" "bytestring" ],
       "librarySystemDepends": [ "lua5_1" ],
       "testHaskellDepends": [
         "base" "bytestring" "hspec" "hspec-contrib" "HUnit" "QuickCheck"
         "quickcheck-instances" "text"
       ],
       "description": "A Lua language interpreter embedding in Haskell",
       "license": "mit"
     }
The package sets then can define their own method of resolving packages to
package versions.

This approach is already in use with KDE 5, using autonix [6] [7], which
aims to become a standard for automated Nix packaging [8]. Autonix assumes
the use of stdenv's mkDerivation and would have to be modified
substantially to achieve its goal of becoming a standard [9], but the basic
resolveDeps function is fine and could be used for Haskell as well.

-- Mathnerd314

[1]
https://github.com/NixOS/cabal2nix/blob/255fd253e04f6eed2531eb2a6119bd6a6f4f9445/hackage2nix/src/Main.hs#L225-L226
[2] http://pastebin.com/5pU9G6D2
[3] http://www.modulecounts.com/
[4]
https://github.com/fpco/stackage/blob/2734cf8fe4fd98fb2ea75f865b4b11eb4a82aac2/build-constraints.yaml#L1552
[5] http://r6.ca/blog/20140422T142911Z.html
[6]
https://github.com/NixOS/nixpkgs/blob/master/pkgs/applications/kde-apps-15.04/packages.json
[7]
https://github.com/NixOS/nixpkgs/blob/master/pkgs/applications/kde-apps-15.04/default.nix
[8] https://github.com/NixOS/nixpkgs/pull/5786#issuecomment-71205805
[9]
https://github.com/NixOS/nixpkgs/blob/master/pkgs/build-support/autonix/default.nix#L46
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.science.uu.nl/pipermail/nix-dev/attachments/20151014/a05a1785/attachment.html