[Nix-dev] Proposal for hydra with librsync

[Lluís Batlle i Rossell]

Hello,

I was thinking of the memory footprint of bsdiff, and I think it goes quite
quite big. So I think that the actual patch system in hydra requires quite a lot
of resources, due to the cost of bsdiff specially on big store paths (those
where bsdiff would add more benefit, I think).

So, dear Hydra programmers... don't you think there could be a way of using
rsync or librsync, and provide a non--simple-HTTP interface to pull storepaths
from the build farm to our nixos systems? Although they will not provide the
small patches from bsdiff, they will cost much less.

For example, I imagine that nixpkgs-unstable contains only, for each system, the
latest store paths resulting from nixpkgs evaluation. Nevertheless, hydra holds
not only those, but more closures, from previous nixpkgs-unstable manifests; I
don't know for how long time.

Using an algorithm to decide what common root store paths can be useful (let's
say, any shared package with the same name-version) between the farm and the
client computer, librsync could reduce a lot the amount of data transfered.

I have in mind the regular kde rebuilds we have, due to an update of it or some
of its dependencies, and how I would like not to redownload that much (without
paying for a faster internet access)

I think that such a system, additionally to the rest of the nix system, would
look astonishing compared to what other distros have.

Any volunteer to start with a hack to test this? Maybe this idea will look nice
even for Delft people to implement it. At least I sent this letter before I
forget.

Regards,
Lluís