[Nix-dev] Travis Testing Needs Rethinking

[Tuomas Tynkkynen]

On 02/13/2016 03:41 PM, Kevin Cox wrote:

>
> TL;DR building PRs based off of latest master often takes too many
> resources for Travis so we should base CI builds off of the PR branch.
>

100% agreed. While we're on the topic of Travis and pull requests, our 
contribution guide currently contains this as the last step:

 > Rebase you branch against current master.

IMHO this is actively harmful and shouldn't be done, because:

- Sending a pull request for a commit hash X should imply a semantic 
meaning of 'I have personally tested that the exact commit X works'. 
Currently we are essentially encouraging contributors to send us 
untested stuff.
- If Kevin's suggestion wrt. building only against the PR branch is 
implemented, rebasing against master means that the likelihood of 
dependencies of the current Travis build not being yet built by Hydra 
increases, leading to Travis timeouts yet again.
- If master has been broken (e.g. tarball evaluation fails) by someone 
else's change in the meantime, this totally unrelated breakage causes 
Travis failures for all those pull requests where the rebase-to-master 
dance was done.

So, any objections for removing that part from the contribution guide 
and perhaps suggesting rebase only in case of GitHub reporting conflicts?