[Nix-dev] Please test Nix store auto-optimise

[James Cook]

Oops, yes, that's a good point. Filesystems with fancy things like
reflinks probably have their own tools for deduplication.

On 15 February 2015 at 01:48, Wout Mertens <wout.mertens at gmail.com> wrote:
> If you want to use reflinks, better to use the deduplication tools that come
> with the filesystem, e.g.
> https://btrfs.wiki.kernel.org/index.php/Deduplication (duperemove is in
> nixpkgs).
>
> Also, it would be nice if we could patch GNU cp so that it always tries to
> use reflinks (so that builds automatically have reflinks where possible).
> There's an option "--reflinks=auto" that does that and I don't understand
> why it's not the default (as discussed at
> http://unix.stackexchange.com/questions/80351/why-is-cp-reflink-auto-not-the-default-behaviour).
> Maybe we should make it a nixos option.
>
> Hardlinks are slightly impure and they do cause build failures in rare and
> easily-fixed cases, but they're cross-platform, robust and easy to inspect.
> Indeed the .links directory won't be able to be pruned due to reflinks being
> invisible.
>
> To implement reflinks properly you would have to keep a hash of each file in
> the nix-store in a DB. This has advantages, like being able to hash lazily
> (only when another file with the same size shows up), and being able to do
> queries, but it also means that you're duplicating that other DB, the file
> system.
>
>
> Wout.
>
> On Sun, Feb 15, 2015, 9:54 AM James Cook <james.cook at utoronto.ca> wrote:
>>
>> > Once I wondered if using reflinks instead of hardlinks might be better
>> > from
>> > some point of view, but it probably won't be a big difference.
>>
>> I would really like to see reflinks being used instead of hard links
>> on filesystems that support it. Hard linking is an impurity which can
>> cause bugs, as Wout pointed out at the start of the thread (e.g.
>> https://github.com/NixOS/nixpkgs/issues/4266).
>>
>> Implementation question: how would nix know when it can delete a file
>> in /nix/store/.links? I assume than now it just checks the number of
>> links, but I don't know if you can do that with reflinks.
>>
>> James
>> _______________________________________________
>> nix-dev mailing list
>> nix-dev at lists.science.uu.nl
>> http://lists.science.uu.nl/mailman/listinfo/nix-dev