[Nix-dev] Continuous Integration

[Ericson, John]

As everyone knows, while updates to nixpkgs roll in 'round the clock,
the unstable channels are usually stuck for days or weeks before
lurching forward. In other words, we don't do continuous integration.
This does not seem sustainable as nixpkgs grows larger and changes
more rapidly.

When I asked about this on IRC previously, I was told that the problem
was intractable while we only had one mac mini, but now we have more
Darwin build machines, and I believe the Darwin nixpkgs infrastructure
has also matured leading to less mass-rebuilds. Thus, I hope there is
no longer a fatal imbalance between our Linux and Darwin needs and
capabilities.

So perhaps now is good time to consider: can we commit to ensuring
that nixos-unsable == nixpkgs-unstable == master? In particular this
means ensuring that every PR passes before merging, and rebuilding
previously-passing PRs whenever master changes. With most build
systems, this would unleash a thundering heard of rebuilding PRs every
time master changes, but since we cache very well and most PRs
commute, I believe this won't be a problem. Mass-rebuilds of course
violate this, but they are a bottleneck no matter what we do.

Note that Travis doesn't rebuild previously-passing PRs. Also, we'd
probably want some way to bless a bunch of PRs so they are
automatically merged as soon as their builds pass. I'd love to
integrate Hydra with github directly, but in the meantime the Rust
community wrote https://github.com/barosl/homu to address these
issues, and it might be worth looking at. [I follow Rust development
so I've seen it in action. But I haven't used it myself.]

What I think this boils down to is the difference between between
master and the *-unstable channels branches dictating hydra's queue,
and the set of open PRs doing so. The order in which PRs get merged is
mostly-arbitrary, and thus the linearity that imposes is mostly
worthless: Hydra must sign off on one commit before moving on to the
next even if the failing commit doesn't interact with the succeeding
commits at all.

Finally, there is more that I don't know than I do know about hydra
and the status quo, so I hope above all else this kicks off a good
discussion.

John