[Nix-dev] Proposal for hydra with librsync

[Lluís Batlle i Rossell]

On Mon, May 10, 2010 at 06:48:52PM -0400, Jeff Johnson wrote:
> 
> On May 10, 2010, at 6:32 PM, Lluís Batlle i Rossell wrote:
> > 
> > I think that such a system, additionally to the rest of the nix system, would
> > look astonishing compared to what other distros have.
> > 
> 
> (aside)
> Distributing with rsync is hardly state of the art, /nix/store _IS_
> state of the art. I wouldn't bother with rsync "astonishment".

I did not know a softer English word. :) Nevertheless, I wanted to point the
rsync for binary distribution, not for source distribution as gentoo or MacOSX.
Additionally, in our case we have quite enough more updates, and a client may
have a quite similar file base; a dependency change may involve an update in
lots of packages, and by now we have to reload them all.

I did not do much tests with the rsync; I tried with kdelibs-4.4.3 yesterday
(with the difference caused by a change in the openssl dependency); the tar.bz2
of the store path amounted to 19MB, while rsync -zc could transfer the new
having the old with only 4MB. That, without closures - using directly the store path
directories. Maybe bsdiff would have done that with 200KB, but with much more
memory.

We can think of using Xdelta,
http://code.google.com/p/xdelta/wiki/TuningMemoryBudget, which can work with a
limited memory budget (not like bsdiff), and may provide even better reduction
of transfer.

> librsync is a zombie project. which is a shame, Martin Pool is a
> really smart guy, just that the rsync protocol is patent encumbered
> many years now.

I did not know about the patent problems.

> There's zsync which _IS_ active, and also has the benefit of
> burning client, not server, CPU cycles. But I haven't looked
> carefully at zsync for several years.
worth a try.

> For extra credit: Figger how to remove rsync's innate bias
> on file paths so that Nix instances could be rsync'd directly.
> Now _THAT_ would be spiffy and "astonishing".
'nix instances'? 'innate bias on file paths'? I don't know what do you talk
about. Can you explain a bit?
I don't mean, specially, to improve the rsync program. I'm interested in nix
closures distribution, specially by the build farm.

About implementing... I'd like to implement a solution at least for
'nix-copy-closure' - whenever I find any time, I'll go with it, with whatever
delta system I find best.

Thank you,
Lluís.