[Nix-dev] Publish All of Hackage

[zimbatm]

Are haskell libraries useful other than at compile time ?

Nix doesn't do dependency resolution in regards to version matching and I
don't think that we will ever have a unified hackage where all the latest
versions work together. Tools like cabal2nix seem essential in that regard.

If we where to only package applications, provide cabal2nix and friends and
include common overrides for things that cabal doesn't describe like C
libraries I think that it would be already very good.

On Thu, 19 Nov 2015 11:14 Peter Simons <simons at cryp.to> wrote:

> The Problem
> -----------
>
> Users of Nixpkgs want to build Haskell packages that have no recent
> release, i.e. the software's authors have not updated their code to deal
> with modern versions of the respective build dependencies. The required
> old versions, however, may not be available in Nixpkgs, and then the
> build cannot succeed. At that point, we recommend our users to add the
> missing package versions themselves [1] through the override mechanism
> in ~/.nixpkgs/config.nix. That approach works fine, but it's challenging
> for new users and it's certainly inconvenient.
>
> The Situation Today
> -------------------
>
> We publish 11,216 package versions out of 62,354 available ones, i.e.
> 18% of Hackage. We select packages for distribution using the following
> criteria:
>
> 1. the latest version of every package (9,058 builds),
>
> 2. old versions required by LTS Haskell (2,137 builds), and
>
> 3. old versions manually configured in the "extra-packages" field of
>    configuration-hackage2nix.yaml [2] (21 builds).
>
> The resulting hackage-packages.nix [3] file is ~8.3 MByte large.
>
> Now, we could simply publish all of Hackage, i.e. all 62,354 available
> builds. That would certainly make the lives of many Haskell hackers
> easier, but unfortunately the resulting hackage-packages.nix file would
> require 52 MByte of space. Furthermore, simple benchmarks suggest that
> nix-instantiate needs a whopping 4 seconds to instantiate a
> "ghcWithPackages" environment from the new database where instantiating
> the same environment used to take 0.45 seconds before --- a performance
> drop by a factor of 8.
>
> Possible Improvements
> ---------------------
>
> We could continue to publish the "small" hackage-packages.nix database
> by default, but generate an additional large database that we'd store in
> a separate location. We would then add support for an option to
> ~/.nixpkgs/config.nix users could set to opt in to using the large
> database if they want to. That way, Haskell users would have an option
> to get all of Hackage without causing a performance drop for everyone
> else.
>
> One issue is that checking that ~50MB file into Nixpkgs might be a bad
> idea, because it sets a dangerous precedent. Arguably, if we check all
> of Hackage into Nixpkgs, then we cannot reasonably say *no* to someone
> who wants to check all of CPAN into Nixpkgs too, and before we know it
> our Git repository triples in size. So it might be wise to put the full
> Hackage variant of hackage-packages.nix into a separate repository that
> we fetch via "builtins.fetchurl" if the corresponding option is enabled.
>
> An altogether different approach is to improve our tools like cabal2nix
> to make it convenient to add old versions from Hackage into a local
> Nixpkgs instance, i.e. by extending the program to generate builds for
> all missing versions to complete the transitive closure of the requested
> build so that users don't have to run it a dozen times because multiple
> old versions are missing. Conceivably, we could also provide tools that
> manage ~/.nixpkgs/config.nix, which might be a good idea independent of
> these Haskell-specific issues.
>
>
> [1]:
> http://nixos.org/nixpkgs/manual/#how-to-create-nix-builds-for-your-own-private-haskell-packages
> [2]:
> https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/haskell-modules/configuration-hackage2nix.yaml
> [3]:
> https://github.com/NixOS/nixpkgs/blob/master/pkgs/development/haskell-modules/hackage-packages.nix
>
> _______________________________________________
> nix-dev mailing list
> nix-dev at lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.science.uu.nl/pipermail/nix-dev/attachments/20151119/e9693ee1/attachment-0001.html