[Nix-dev] Our policy for upgrading haskellPackages

[John Wiegley]

This message is a follow-on to a discuss Peter and I were having on GitHub,
since I believe it is of more general interest:

>>>>>> Peter Simons writes:

> Generally speaking, the two goals
>
>   1. have recent versions of all major Haskell packages and
>   2. all Haskell packages should compile
>
> are contradictory. The 2.6.x version of network has been out there since Tue
> Sep 2 18:14:36 UTC 2014, i.e. for more than 1,5 months. Since 2.5.x and
> 2.6.x have incompatible APIs, many package authors don't bother supporting
> the old version: they update their packages to compile against 2.6.x and
> never look back. Now, in that situation, we must switch to 2.6.x eventually,
> because network 2.5.x cannot compile many available updates. At the same
> time, the switch to 2.6.x breaks the packages of all those people who didn't
> update their libraries.
>
> So what are we supposed to do? Forgo the available updates to keep a stable
> system or update at the cost of breaking packages that are sort-of
> unmaintained?
>
> I try to keep as many packages building as possible, and getting those ~200
> updates into master was a major effort for me, i.e. I worked on those
> commits several hours per day for the better part of a week. Even with all
> that effort spent, however, I cannot remedy the fundamental conflict of
> interest between a system that's up-to-date and a system that's stable. At
> some point, I just push whatever I have come up with and I rely on other
> people, like yourself, to help finding the best balance between those two
> contradictory goals.

Hi Peter,

First, let me state how much I appreciate the contribution you're making to
nixpkgs.  Its support of Haskell is superb, and that is in large part due to
your time and effort.  My hope is to support you as best I can, and not to
criticize your efforts in any way.

You are exactly right that we have a tension between those two goals.  I can
think of two things that might be done to remedy this, and perhaps make
updates to master more smooth:

  1. We keep a dedicated branch, "haskell-updates", to which only your Hackage
     updates get pushed, or fixes to those updates.  I will personally pull
     and rebuild this branch every day on my machine, just as I presently
     rebuild master nearly every day -- compiling more than 2,000 packages
     that I keep locally updated through --leq.

     I (and hopefully others) will help to discover which packages can be
     fixed by inserting references to older packages, which requires patches,
     and which must truly be marked as "broken" until the maintainer of that
     package can be contacted.

     Further, I'll help you to maintain a list of outstanding "broken"
     packages, and see what can be done to make sure this list decreases over
     time.

  2. The second option is to create a new haskellPackages set, called
     'stackage'.  The Stackage maintainers already do a lot of the work
     implied by #1, ensuring that every package within the Stackage set can
     build together.  Further, they only upgrade a package once they've either
     created a patch, or worked with upstream to update the package.

     Of course, the downside to this is:

       - less frequent updates of packages
       - a smaller available package set
       - life-draining maintenance of a mostly parallel package set

     The upside being that all patching/curating work is done for us, likely
     for as long as FP Complete keeps funding people to maintain Stackage.

Most of the time I can resolve breakages that occur on master, and I'm getting
up to speed with pushing the right fixes back to you via cabal2nix.  However,
I still rely on 'master' to be working overall on a daily basis, and sometimes
the degree of breakage in haskellPackages is too much to handle all at once,
forcing me to stop tracking 'master' -- which then delays my involvement in
getting those breakages fixed.

I think if we had a separate channel for haskell updates, and that if you and
I both worked together to get that channel ready for inclusion into master, we
could make this upgrade effort smoother for everyone involved, and hopefully
less stressful for you in particular.

The only important part, then, is that we be sure this branch gets on Hydra,
as another check of suitability.

It would also be really nice to see you on IRC more, for asking question about
upgrade decisions more quickly than through GitHub.  But I understand if
that's not possible.

Yours,
  John