[Nix-dev] Travis Testing Needs Rethinking

[Kevin Cox]

Hello,

I have been trying to submit a pull request for quite some time but
keep getting bitten by the Travis testing. The problem boils down to a
couple of things. Firstly travis has build limits both in terms of log
size and build time. These are expected and while there is possibility
of getting them raised (at least for log size) I'm going to assume that
these are the bounds we have to work within.

The reason that this is a problem is because of the way nox (the review
tool) works. Nox merges your new changes into the master branch then
attempts to build your PR. The problem with this is that anything you
depend on that has recently been changed will have to be rebuilt from
source as they aren't yet in a binary cache. When you have a
sufficiently complex package you almost always have dependencies that
have recently changed so you end up rebuilding a lot of unchanged
packages which will often send you over your resource quotas. This is
especially true as these complex packages are often close to the limits
on their own.

To combat this problem I propose that packages are tested on their
current branch. To determine what has changed simple identify the
changes since the latest common ancestor of the pull request and the
master branch. This means that you can allow your PR to get slightly
behind master and now all of your dependencies are in the binary cache
and don't eat up your resources.

The apparent downside is that you aren't testing on the latest code so
there may have been incompatible changes in the mean time that appear
on merge. However since there is often a couple of days delay between
Travis passing and merging of PRs anyways so I don't see this as much
added risk. If desired an error or warning can be added when a PR is
"too far" behind master to limit this problem as well. In the rare case
of conflict it will be caught on the master branch, either by testing
master in a similar way (although finding the "base" commit might be
non-trivial and you still have the resource usage issue) or when they
appear on hydra. Thanks to git and Github reverting PRs that cause
breakage is literally two clicks so anything that causes a merge
conflict can be reverted and the submitter notified so that it can be
fixed up to be re-merged.

I think that the overall value proposition is a benefit as it will make
merging PRs much less painful because build times will be down which
prevents travis killing builds and enables faster feedback and
iteration.

TL;DR building PRs based off of latest master often takes too many
resources for Travis so we should base CI builds off of the PR branch.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.science.uu.nl/pipermail/nix-dev/attachments/20160213/a80e47a1/attachment.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 213 bytes
Desc: This is a digitally signed message part
Url : http://lists.science.uu.nl/pipermail/nix-dev/attachments/20160213/a80e47a1/attachment.bin 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/x-pkcs7-signature
Size: 5658 bytes
Desc: not available
Url : http://lists.science.uu.nl/pipermail/nix-dev/attachments/20160213/a80e47a1/attachment-0001.bin