[Nix-dev] Set up a Sufficiently Powerful Build Farm

[Eelco Dolstra]

Hi Peter,

On 29/10/15 13:38, Peter Simons wrote:

<snip>

Regarding hydra.nixos.org disk space, I hope to address this by having
hydra-queue-runner store build results directly in an S3 bucket (i.e.
cache.nixos.org), rather than in the local Nix store. (That would be part 2 of
the queue runner overhaul - part 1 was having a single-process queue runner that
loads the dependency graphs of all builds in the queue and can thus schedule
builds more efficiently.)

This would have a few advantages:

* Unlimited disk space.

* Remove the Nix store on the central machine as a bottleneck. Currently copying
store paths to/from the build slaves takes significantly more time than the
actual builds.

* Shorter latency - packages would appear in cache.nixos.org as soon as they
have been built, rather than when the channel mirror script runs.

The Nix store on the central machine would then only be used to communicate .drv
files between hydra-evaluator and hydra-queue-runner. But this shouldn't be much
data, so it would enable using a fast SSD drive for the Nix store.

Another recent improvement is hydra-provisioner, which is a script that fires up
EC2 spot instances dynamically based on the Hydra queue. This allows us to burn
through Linux builds pretty quickly. Thus the main problem right now is Darwin
builds, which are done on a single Mac mini...

> It's unclear whether the Hydra software would cope with 66 package sets with
> some 111,000 derivations in them that need to be evaluated, say, once an hour.
> Hydra has undergone some architectural changes recently that might make such a
> load possible -- i.e. "hydra-evaluator" is more efficient than it used to be
> --, but I don't have any reliable data concerning the performance of the
> process, so I cannot say what is possible and what is not.

Currently the main cause of evaluator slowness on hydra.nixos.org is due to the
high I/O load and the disk not being very fast in general. If the store, Nix
database and PostgreSQL database were on an SSD drive, I expect the evaluator to
be much faster.

-- 
Eelco Dolstra | LogicBlox, Inc. | http://nixos.org/~eelco/