[Nix-dev] Travis Testing Needs Rethinking

[Adam Russell]

Wouldn't this all be less of an issue if the build on Hydra wasn't behind
by weeks? Should we talk about how to improve that? Personally I don't even
know how to navigate or interpret Hydra when I go look at it.

On Sat, Feb 13, 2016 at 10:48 AM Jakob Gillich <jakob at gillich.me> wrote:

> I assume PR branch refers to unstable? Or even a stable branch? I don't
> really see how testing against non-master is better when you're submitting
> something for master.
>
> On Sat, Feb 13, 2016, at 02:41 PM, Kevin Cox wrote:
>
> Hello,
>
> I have been trying to submit a pull request for quite some time but keep
> getting bitten by the Travis testing. The problem boils down to a couple of
> things. Firstly travis has build limits both in terms of log size and build
> time. These are expected and while there is possibility of getting them
> raised (at least for log size) I'm going to assume that these are the
> bounds we have to work within.
>
> The reason that this is a problem is because of the way nox (the review
> tool) works. Nox merges your new changes into the master branch then
> attempts to build your PR. The problem with this is that anything you
> depend on that has recently been changed will have to be rebuilt from
> source as they aren't yet in a binary cache. When you have a sufficiently
> complex package you almost always have dependencies that have recently
> changed so you end up rebuilding a lot of unchanged packages which will
> often send you over your resource quotas. This is especially true as these
> complex packages are often close to the limits on their own.
>
> To combat this problem I propose that packages are tested on their current
> branch. To determine what has changed simple identify the changes since the
> latest common ancestor of the pull request and the master branch. This
> means that you can allow your PR to get slightly behind master and now all
> of your dependencies are in the binary cache and don't eat up your
> resources.
>
> The apparent downside is that you aren't testing on the latest code so
> there may have been incompatible changes in the mean time that appear on
> merge. However since there is often a couple of days delay between Travis
> passing and merging of PRs anyways so I don't see this as much added risk.
> If desired an error or warning can be added when a PR is "too far" behind
> master to limit this problem as well. In the rare case of conflict it will
> be caught on the master branch, either by testing master in a similar way
> (although finding the "base" commit might be non-trivial and you still have
> the resource usage issue) or when they appear on hydra. Thanks to git and
> Github reverting PRs that cause breakage is literally two clicks so
> anything that causes a merge conflict can be reverted and the submitter
> notified so that it can be fixed up to be re-merged.
>
> I think that the overall value proposition is a benefit as it will make
> merging PRs much less painful because build times will be down which
> prevents travis killing builds and enables faster feedback and iteration.
>
> TL;DR building PRs based off of latest master often takes too many
> resources for Travis so we should base CI builds off of the PR branch.
> *_______________________________________________*
> nix-dev mailing list
> nix-dev at lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
> Email had 2 attachments:
>
>    - signature.asc
>      1k (application/pgp-signature)
>    - smime.p7s
>      8k (application/x-pkcs7-signature)
>
>
> _______________________________________________
> nix-dev mailing list
> nix-dev at lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.science.uu.nl/pipermail/nix-dev/attachments/20160213/481ae280/attachment.html