[Nix-dev] cabal-install 1.24's Nix-style builds, and a new style of Nix + language package manager integration

[John Ericson]

I wrote an issue for Cabal that goes into more detail at what the Cabal end
of things might look like: https://github.com/haskell/cabal/issues/3882

On Sun, Sep 18, 2016 at 11:48 PM, John Ericson <
john_ericson at alumni.brown.edu> wrote:

> After much pondering, I've decided the best way to work with
> language-specific build managers is for them to come up with the end-to-end
> build plan, and us to build each package in that build plan.
>
> Our current method with Haskell of trying to replicate each package and
> all the possible ways it can be configured just doesn't scale very well
> with respect to configuaration options. Things like `if impl(ghcjs):` have
> long remained unimplemented, and implementing them bakes more and more
> build-plan resolution machinery into machine until we approach
> re-implementing cabal-install. [Peti has discussed this in depth in
> https://github.com/NixOS/cabal2nix/blob/master/doc/03-
> map-cabal-files-to-nix-without-information-loss.md.]
>
> At the same time, or current method isn't great for developing multiple
> packages simultaneously. The default thing to do is make a package set with
> each one, but if one doesn't handle the source very carefully, a change to
> any of them will cause them all to be rebuilt. Furthermore, one needs
> switch shells per each package---which is a pain in itself---and that
> prevents the use of neat things like `stack repl` which interprets multiple
> packages at once for seamless concurrent development. [I discussed part of
> this in https://github.com/NixOS/nixpkgs/issues/10902 a bit.]
>
> `stack --nix` at first glance seems to solve both of these areas, but in
> fact comes with it's own set of downsides. Yes, since Stack manages the
> build plan less knowledge is needed on the nix side, but stack also
> *executes* the build plan. This means that both Nix's superior caching is
> not utilized, and per-package patches (like foreign dependencies) are
> unavailable. [See https://github.com/NixOS/nixpkgs/issues/15995.]
>
> What I'm proposing should give us the best of both worlds in that language
> package manager makes most of the decisions, but we still have an
> opportunity to cache and patch, and development and deployment workflows
> overlap as much as possible. This methodology could in principle work with
> any language and its package manager, but I single out Cabal+Haskell
> because the new Nix-style workflow (http://cabal.readthedocs.io/
> en/latest/nix-local-build.html) makes it the most ready.
>
> For both deployment and development, the idea is we first run
> cabal-install to compute a build plan. During this stage it in principle
> only needs access to .cabal files / the Hackage index and a project
> config---certainly not any already built binaries like today. Next we
> instruct cabal-install to decompose the plan into "sub plans" for each
> dependency/would-be system package---this is the biggest change needed for
> cabal-install, but one that any system package manager could leverage, not
> just us. After that, using a reworked cabal2nix, we convert each of those
> plans to a Nix derivation---this may include turning hackage deps into
> local deps with their source fetched by nix. Finally, each of those
> derivations can use `cabal new-install` to install their root package, and
> the caches/package DBs can be passed downstream and combined just like
> today.
>
> There are two important concepts to note here: First, naively, the use of
> entire sub plans in each Nix derivation would cause cabal-install to
> rebuild each dependency for package in the plan that depends that needs it,
> but since the Nix derivations will also depend on each other, `cabal
> new-build`'s own caching should kick in so each Nix derivation does in fact
> correspond to each Haskell package being built. Second, while cabal-install
> can be instructed to build any package in a plan, the decomposition of
> build plans is necessary so build plans that differ in ways that don't
> affect common dependencies still end up sharing nix-executed builds. In
> both cases, we rely on cabal-install's imitation of our methodology.
>
> For development, we leverage the new nix-style build's preexisting notion
> of a local vs external package. For convenience, it's probably best to let
> cabal-install devise the build plan impurely (this is analogous to
> non-restricted mode for Nix expr evaluation), and not copy any local
> sources to the Nix store. Then, only external packages (and only those
> which are dependencies of the current goal) should be built by Nix in
> accordance with the plan above. In a shell with those built, cabal-install
> can be run again building local packages normally and locally using its
> more fine-grained per-file hashing and enabling tricks analogous to
> `stack-repl`. While the default shell derivation will need tweaking if any
> local packages have special requirements like foreign dependencies, this is
> better than with `stack --nix` where the shell needs tweaking if *any*
> dependency has such special requirements.
> like today.
>
> For deployment we don't need to worry about local vs external packages,
> and on the cabal-install side `cabal new-install` should treat them all the
> same IIUC. However, we do need to make sure that the build plan resolution
> can be done in a pure environment: this means making sure all local
> packages locations are in the Nix store and the Hackage index is
> pre-downloaded by Nix. After that however no further complications arise.
>
> There's a lot of work with nixpkgs and cabal2nix required for all this,
> but comparatively little upstream for cabal-install. And the end result I
> believe is hugely worthwhile, solving just about every pain point with our
> existing infrastructure.
>
> John
>
> N.B. not quite sure on the division of labor between Cabal and
> cabal-install with the new system, and erring on the side of cabal-install.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.science.uu.nl/pipermail/nix-dev/attachments/20160921/09b27254/attachment.html>