[Nix-dev] Set up a Sufficiently Powerful Build Farm

[Eelco Dolstra]

Hi Peter,

On 04/11/15 14:52, Peter Simons wrote:

>  > Packages would appear in cache.nixos.org as soon as they have been
>  > built, rather than when the channel mirror script runs.
> 
> Do you have any idea how to deal with the fact that some jobsets aren't
> necessarily supposed to be available from cache.nixos.org, like
> nixpkgs/multiple-outputs, because they're used for CI-testing work in
> progress rather than for producing release binaries of Nixpkgs?

Everything would be stored in the binary cache right away, even packages that
don't end up in a NixOS/Nixpkgs channel. The main consequence is that we'd have
to run the binary cache garbage collection script [1] more frequently (currently
I manually run it once or twice a year).

[1]
https://github.com/NixOS/nixos-channel-scripts/blob/master/find-binary-cache-garbage.pl

>  > Do you have data on how well those packages compress? Cache.nixos.org
>  > uses xz compression, so if hydra-queue-runner compressed build
>  > results before uploading to S3, it would presumably need a lot less
>  > than 1 TB.
> 
> I've nix-push'ed some 1,500 packages for testing purposes, and the tool
> reports a total compression ratio of ~8%. 

Thanks!

Just to be clear: you mean it compresses *to* 8%, or *by* 8%?

-- 
Eelco Dolstra | LogicBlox, Inc. | http://nixos.org/~eelco/