[Nix-dev] Help needed: patching Hydra to retry failed builds after a while

[Gergely Risko]

Hi,

Happy new year to all the Nixers around here!

In https://github.com/NixOS/hydra/issues/139 I reported the following issue:

> It happens quite frequently that some build breaks with a transient
> failure on some Hydra machine. The most recent example is GHC on
> i686. The only solution in these situations is to whine on the mailing
> list and hope that some hydra admin will restart the failed build.
>
> It'd be much better to have a TTL for negative build caching and retry
> failed builds e.g. every week at least once even if the derivation
> didn't change. That would ensure that transient errors get fixed even
> without manual intervention.

Since I received no comments on the ticket, may I ask for opinions here?
Is this a good idea to do?  If yes, can someone with actual coding and
design experience with hydra help me please?

Are there any design decisions to make?  Can someone point me to the
relevant parts of the codebase and give a little bit of an overview what
I have to do to achieve this goal?  I'd be happy to figure out the
details and prepare a patch of course.

Currently I can't update my haskell machines for the last 4 months on
i686 because of this and always pinging the hell out of hydra admins
seems to be a waste of everybody's time if this can be automated.

Thanks,
Gergely