[Nix-dev] Please test Nix store auto-optimise

[Vladimír Čunát]

On 04/25/2015 03:56 PM, Ertugrul Söylemez wrote:
> Tl;dr:  The current linking strategy has a noticable space and time
> overhead that is large enough to be unacceptable for me, so I turned it
> off.

I turned it off due to time overhead on slow HDD (5.4 krpm), as it was
slowing down very much when downloading fast from binary cache; most of
those files I can GC very soon anyway, as I'm often just testing some
builds. (I believe space overhead is low enough on btrfs, as I posted in
this thread.)

IMO much of the time overhead is basically unavoidable, as you need to
do lots of random probes to a huge table (order of million entries, e.g.
1.5M in my case). Putting a single Bloom filter in front would shave I/O
of most *failed* probes, but otherwise we can't get much better, I
believe (being a PhD student in data structures).

Anyway, this deduplication work seems better delegated to some other
SW/library. Surely there are some already. If we wanted to deduplicate
*similar* (unequal) files, we would even need to get FS-dependent. In
the typical ext4 case, we don't even have reflinks AFAIK...


Vladimir


-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 3251 bytes
Desc: S/MIME Cryptographic Signature
Url : http://lists.science.uu.nl/pipermail/nix-dev/attachments/20150425/01ccaccb/attachment.bin