[Nix-dev] Our policy for upgrading haskellPackages

[Mateusz Kowalczyk]

On 10/15/2014 08:27 PM, John Wiegley wrote:
> This message is a follow-on to a discuss Peter and I were having on GitHub,
> since I believe it is of more general interest:
> 
>>>>>>> Peter Simons writes:
> 
>> Generally speaking, the two goals
>>
>>   1. have recent versions of all major Haskell packages and
>>   2. all Haskell packages should compile
>>
>> are contradictory. The 2.6.x version of network has been out there since Tue
>> Sep 2 18:14:36 UTC 2014, i.e. for more than 1,5 months. Since 2.5.x and
>> 2.6.x have incompatible APIs, many package authors don't bother supporting
>> the old version: they update their packages to compile against 2.6.x and
>> never look back. Now, in that situation, we must switch to 2.6.x eventually,
>> because network 2.5.x cannot compile many available updates. At the same
>> time, the switch to 2.6.x breaks the packages of all those people who didn't
>> update their libraries.
>>
>> So what are we supposed to do? Forgo the available updates to keep a stable
>> system or update at the cost of breaking packages that are sort-of
>> unmaintained?
>>
>> I try to keep as many packages building as possible, and getting those ~200
>> updates into master was a major effort for me, i.e. I worked on those
>> commits several hours per day for the better part of a week. Even with all
>> that effort spent, however, I cannot remedy the fundamental conflict of
>> interest between a system that's up-to-date and a system that's stable. At
>> some point, I just push whatever I have come up with and I rely on other
>> people, like yourself, to help finding the best balance between those two
>> contradictory goals.
> 
> Hi Peter,
> 
> First, let me state how much I appreciate the contribution you're making to
> nixpkgs.  Its support of Haskell is superb, and that is in large part due to
> your time and effort.  My hope is to support you as best I can, and not to
> criticize your efforts in any way.
> 
> You are exactly right that we have a tension between those two goals.  I can
> think of two things that might be done to remedy this, and perhaps make
> updates to master more smooth:
> 
>   1. We keep a dedicated branch, "haskell-updates", to which only your Hackage
>      updates get pushed, or fixes to those updates.  I will personally pull
>      and rebuild this branch every day on my machine, just as I presently
>      rebuild master nearly every day -- compiling more than 2,000 packages
>      that I keep locally updated through --leq.
> 
>      I (and hopefully others) will help to discover which packages can be
>      fixed by inserting references to older packages, which requires patches,
>      and which must truly be marked as "broken" until the maintainer of that
>      package can be contacted.
> 
>      Further, I'll help you to maintain a list of outstanding "broken"
>      packages, and see what can be done to make sure this list decreases over
>      time.

This already exists, in part: the haskell-updates job at [1]. I say in
part because it:

1. Only builds for x86_64 (so the recent vector failure was not caught)
2. It doesn't make clear what packages were marked as broken.
3. Once it builds, it goes into master (usually) without any further
review: partially a good thing because we get updates fast.

So I think a better plan would be instead to have a separate branch (not
haskell-updates) which contains only Haskell updates that result in some
packages getting broken. Everything else just goes into master (through
haskell-updates or whatever) as it is doing today.

Say, keep the changes in this pending-broken branch for two days behind
master: any interested party has a chance to jump in and say ‘oh, I know
how to make this update not break anything’ or ‘I'll take care of
specifying proper versions for these packages’. Obviously what would go
onto this branch is at updaters discretion: no point delaying update of
<really popular package> because one <really unpopular package> breaks.

This approach of only putting up breaking updates for public review is,
I feel, a better solution than requiring every Haskell update to wait in
a branch for a day or two for no particular reason.

>   2. The second option is to create a new haskellPackages set, called
>      'stackage'.  The Stackage maintainers already do a lot of the work
>      implied by #1, ensuring that every package within the Stackage set can
>      build together.  Further, they only upgrade a package once they've either
>      created a patch, or worked with upstream to update the package.
> 
>      Of course, the downside to this is:
> 
>        - less frequent updates of packages
>        - a smaller available package set
>        - life-draining maintenance of a mostly parallel package set
> 
>      The upside being that all patching/curating work is done for us, likely
>      for as long as FP Complete keeps funding people to maintain Stackage.

This should be pretty easy to maintain on nix side as we should be able
to hackage4nix/cabal2nix all of hackage and have it Just Work™.

Personally I'd like to see it go outside of nixpkgs, like Gentoo
overlays do: I want to say separate channel but I don't know of any
instructions about making channels play well together. This is more out
of the principle of ‘people have different needs for package sets so it
should be separated somehow’ more than any specific use-case though so
if it complicates things then it's probably not worth bothering.

> Most of the time I can resolve breakages that occur on master, and I'm getting
> up to speed with pushing the right fixes back to you via cabal2nix.  However,
> I still rely on 'master' to be working overall on a daily basis, and sometimes
> the degree of breakage in haskellPackages is too much to handle all at once,
> forcing me to stop tracking 'master' -- which then delays my involvement in
> getting those breakages fixed.
> 
> I think if we had a separate channel for haskell updates, and that if you and
> I both worked together to get that channel ready for inclusion into master, we
> could make this upgrade effort smoother for everyone involved, and hopefully
> less stressful for you in particular.
> 
> The only important part, then, is that we be sure this branch gets on Hydra,
> as another check of suitability.
> 
> It would also be really nice to see you on IRC more, for asking question about
> upgrade decisions more quickly than through GitHub.  But I understand if
> that's not possible.
> 
> Yours,
>   John
> _______________________________________________
> nix-dev mailing list
> nix-dev at lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
> 


[1]: http://hydra.cryp.to/project/nixpkgs

-- 
Mateusz K.