[Nix-dev] Travis Testing Needs Rethinking

[Jakob Gillich]

I assume PR branch refers to unstable? Or even a stable branch? I don't
really see how testing against non-master is better when you're
submitting something for master.

On Sat, Feb 13, 2016, at 02:41 PM, Kevin Cox wrote:
> Hello,
>
> I have been trying to submit a pull request for quite some time but
> keep getting bitten by the Travis testing. The problem boils down to a
> couple of things. Firstly travis has build limits both in terms of log
> size and build time. These are expected and while there is possibility
> of getting them raised (at least for log size) I'm going to assume
> that these are the bounds we have to work within.
>
> The reason that this is a problem is because of the way nox (the
> review tool) works. Nox merges your new changes into the master branch
> then attempts to build your PR. The problem with this is that anything
> you depend on that has recently been changed will have to be rebuilt
> from source as they aren't yet in a binary cache. When you have a
> sufficiently complex package you almost always have dependencies that
> have recently changed so you end up rebuilding a lot of unchanged
> packages which will often send you over your resource quotas. This is
> especially true as these complex packages are often close to the
> limits on their own.
>
> To combat this problem I propose that packages are tested on their
> current branch. To determine what has changed simple identify the
> changes since the latest common ancestor of the pull request and the
> master branch. This means that you can allow your PR to get slightly
> behind master and now all of your dependencies are in the binary cache
> and don't eat up your resources.
>
> The apparent downside is that you aren't testing on the latest code so
> there may have been incompatible changes in the mean time that appear
> on merge. However since there is often a couple of days delay between
> Travis passing and merging of PRs anyways so I don't see this as much
> added risk. If desired an error or warning can be added when a PR is
> "too far" behind master to limit this problem as well. In the rare
> case of conflict it will be caught on the master branch, either by
> testing master in a similar way (although finding the "base" commit
> might be non-trivial and you still have the resource usage issue) or
> when they appear on hydra. Thanks to git and Github reverting PRs that
> cause breakage is literally two clicks so anything that causes a merge
> conflict can be reverted and the submitter notified so that it can be
> fixed up to be re-merged.
>
> I think that the overall value proposition is a benefit as it will
> make merging PRs much less painful because build times will be down
> which prevents travis killing builds and enables faster feedback and
> iteration.
>
> TL;DR building PRs based off of latest master often takes too many
> resources for Travis so we should base CI builds off of the PR branch.
> _________________________________________________
> nix-dev mailing list nix-dev at lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev Email had 2
> attachments:


>  * signature.asc  1k (application/pgp-signature)
>  * smime.p7s  8k (application/x-pkcs7-signature)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.science.uu.nl/pipermail/nix-dev/attachments/20160213/e2077c9a/attachment.html