[Nix-dev] Please test Nix store auto-optimise

[Wout Mertens]

If you want to use reflinks, better to use the deduplication tools that
come with the filesystem, e.g.
https://btrfs.wiki.kernel.org/index.php/Deduplication (duperemove is in
nixpkgs).

Also, it would be nice if we could patch GNU cp so that it always tries to
use reflinks (so that builds automatically have reflinks where possible).
There's an option "--reflinks=auto" that does that and I don't understand
why it's not the default (as discussed at
http://unix.stackexchange.com/questions/80351/why-is-cp-reflink-auto-not-the-default-behaviour).
Maybe we should make it a nixos option.

Hardlinks are slightly impure and they do cause build failures in rare and
easily-fixed cases, but they're cross-platform, robust and easy to inspect.
Indeed the .links directory won't be able to be pruned due to reflinks
being invisible.

To implement reflinks properly you would have to keep a hash of each file
in the nix-store in a DB. This has advantages, like being able to hash
lazily (only when another file with the same size shows up), and being able
to do queries, but it also means that you're duplicating that other DB, the
file system.


Wout.
On Sun, Feb 15, 2015, 9:54 AM James Cook <james.cook at utoronto.ca> wrote:

> > Once I wondered if using reflinks instead of hardlinks might be better
> from
> > some point of view, but it probably won't be a big difference.
>
> I would really like to see reflinks being used instead of hard links
> on filesystems that support it. Hard linking is an impurity which can
> cause bugs, as Wout pointed out at the start of the thread (e.g.
> https://github.com/NixOS/nixpkgs/issues/4266).
>
> Implementation question: how would nix know when it can delete a file
> in /nix/store/.links? I assume than now it just checks the number of
> links, but I don't know if you can do that with reflinks.
>
> James
> _______________________________________________
> nix-dev mailing list
> nix-dev at lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.science.uu.nl/pipermail/nix-dev/attachments/20150215/474b4d6a/attachment.html