[Nix-dev] Set up a Sufficiently Powerful Build Farm

[Peter Simons]

Hi Eelco,

 > Regarding hydra.nixos.org disk space, I hope to address this by
 > having hydra-queue-runner store build results directly in an S3
 > bucket (i.e.cache.nixos.org), rather than in the local Nix store.

yes, that sounds like a good solution. It would be great to have that
feature available in hydra.nixos.org. Especially, if the change also
increases the performance of the evaluator.


 > Packages would appear in cache.nixos.org as soon as they have been
 > built, rather than when the channel mirror script runs.

Do you have any idea how to deal with the fact that some jobsets aren't
necessarily supposed to be available from cache.nixos.org, like
nixpkgs/multiple-outputs, because they're used for CI-testing work in
progress rather than for producing release binaries of Nixpkgs? There
seems to be an interesting problem insofar as that we don't know whether
a build run as part of "staging" is going to end up in the "unstable"
channel or not at the time when we run it. If the evaluation as a whole
is good, we merge "staging" to "master" and the build becomes a part of
Nixpkgs, but it might also happen that we have another stdenv change in
"staging" before such a merge occurs, and then the binaries produced by
that builds turn out to be irrelevant.


 > Do you have data on how well those packages compress? Cache.nixos.org
 > uses xz compression, so if hydra-queue-runner compressed build
 > results before uploading to S3, it would presumably need a lot less
 > than 1 TB.

I've nix-push'ed some 1,500 packages for testing purposes, and the tool
reports a total compression ratio of ~8%. That number is somewhat
inaccurate because nix-push publishes the *closure* of each store path I
give it -- so the compression ratio is computed over more than just the
Haskell packages, but still it feels like a reasonable result.

Generally speaking, the 1TB estimate is probably on the high end of the
scale, because it's based on the assumption that the size of the Haskell
package set for all platforms equals 3 times the size of that from
Linux/x86_64. In practice, however, i686 binaries will be smaller than
those from 86_64, and we also run far less builds on Darwin than on
Linux.

Anyway, I've also taken a moment to look at the build times of the
Haskell package set. On a moderately recent Linux/x86_64 server, the
build run-times (in minutes) are distributed as follows (based on
259,455 samples, not including "ghc" builds):

       Min.  1st Qu.   Median     Mean  3rd Qu.     Max.
    0.09744  0.22790  0.37780  0.92970  0.75990 81.83000

The most expensive builds in terms of average run-time are:

                          job builds   runtime     median    mean         sd
 1:                  metadata     14 1145.6167 74.766667 81.82976  23.462635
 2:                poker-eval     35 2136.6000 48.500000 61.04571  29.052927
 3:        unicode-properties     13  458.9667 34.050000 35.30513   8.226384
 4:                   FpMLv53     19  652.8000 30.416667 34.35789  15.851984
 5:                   texmath     91 3106.6000 21.550000 34.13846  50.151076
 6:          scholdoc-texmath     77 2437.1500 32.000000 31.65130   6.629542
 7:              amazonka-ec2    109 3423.9500 28.633333 31.41239  15.110833
 8: csound-expression-opcodes     27  846.7000  3.283333 31.35926 142.592864
 9:                      Agda     93 2860.5667 25.550000 30.75878  14.754584
10:                     idris    128 3269.7667 23.491667 25.54505   8.495109
11:        accelerate-fourier     15  373.9167 31.183333 24.92778  19.153399
12:                 uhc-light     46 1110.5333 22.950000 24.14203   8.405919
13:                       RSA     81 1940.1333  1.600000 23.95226  56.325163
14:            java-character     16  362.5500 23.700000 22.65938   3.750131
15:                  encoding     33  689.6333 17.833333 20.89798   9.091368
16:                      lens    117 2421.4667 18.700000 20.69630   8.762991
17:             unicode-names     12  238.3833 20.058333 19.86528   1.812477
18:                  hssqlppp     55 1019.6667 15.566667 18.53939   6.277109
19:                       uhc     21  369.0833 17.100000 17.57540   5.794108
20:                        gl     74 1287.9167 17.875000 17.40428   5.709247
21:            open-symbology     19  322.2667 16.983333 16.96140   2.655905
22:                  graphviz     53  894.6333 13.316667 16.87987  16.741832
23:                      riak     47  782.3667  0.800000 16.64610 108.513826
24:          haskell-src-exts     51  826.7000 14.533333 16.20980   7.695385
25:                       gtk     89 1408.8167 16.316667 15.82940   4.134229
26:                 git-annex    510 7850.3833 13.891667 15.39291   6.257803
27:                        yi    138 2101.7833 15.066667 15.23031   3.314749
28:                GenussFold     12  179.2333 14.941667 14.93611   4.535822
29:                      midi     39  579.5667  2.966667 14.86068  71.999306
30:                      gtk3     89 1260.9667 14.183333 14.16816   4.621893

The vast majority of builds is quick: approximately 82% of all builds
take less than 1 minute. 92% of all builds finish in less than 2
minutes.

The server I've been using runs up to 6 builds in parallel. So we'd
expect that machine to compile all of our 77,445 Haskell builds from
scratch within approximately 8 days. This means that a farm of ~10
servers should be able to compile all of our Haskell packages from
scratch once per day. :-)

Best regards,
Peter