[Nix-dev] Please test Nix store auto-optimise

[Raahul Kumar]

Can I know the best way to do this for BTRFS? Install duperemove, then what?

Aloha,
RK.

On Mon, Feb 16, 2015 at 8:32 AM, James Cook <james.cook at utoronto.ca> wrote:

> Oops, yes, that's a good point. Filesystems with fancy things like
> reflinks probably have their own tools for deduplication.
>
> On 15 February 2015 at 01:48, Wout Mertens <wout.mertens at gmail.com> wrote:
> > If you want to use reflinks, better to use the deduplication tools that
> come
> > with the filesystem, e.g.
> > https://btrfs.wiki.kernel.org/index.php/Deduplication (duperemove is in
> > nixpkgs).
> >
> > Also, it would be nice if we could patch GNU cp so that it always tries
> to
> > use reflinks (so that builds automatically have reflinks where possible).
> > There's an option "--reflinks=auto" that does that and I don't understand
> > why it's not the default (as discussed at
> >
> http://unix.stackexchange.com/questions/80351/why-is-cp-reflink-auto-not-the-default-behaviour
> ).
> > Maybe we should make it a nixos option.
> >
> > Hardlinks are slightly impure and they do cause build failures in rare
> and
> > easily-fixed cases, but they're cross-platform, robust and easy to
> inspect.
> > Indeed the .links directory won't be able to be pruned due to reflinks
> being
> > invisible.
> >
> > To implement reflinks properly you would have to keep a hash of each
> file in
> > the nix-store in a DB. This has advantages, like being able to hash
> lazily
> > (only when another file with the same size shows up), and being able to
> do
> > queries, but it also means that you're duplicating that other DB, the
> file
> > system.
> >
> >
> > Wout.
> >
> > On Sun, Feb 15, 2015, 9:54 AM James Cook <james.cook at utoronto.ca> wrote:
> >>
> >> > Once I wondered if using reflinks instead of hardlinks might be better
> >> > from
> >> > some point of view, but it probably won't be a big difference.
> >>
> >> I would really like to see reflinks being used instead of hard links
> >> on filesystems that support it. Hard linking is an impurity which can
> >> cause bugs, as Wout pointed out at the start of the thread (e.g.
> >> https://github.com/NixOS/nixpkgs/issues/4266).
> >>
> >> Implementation question: how would nix know when it can delete a file
> >> in /nix/store/.links? I assume than now it just checks the number of
> >> links, but I don't know if you can do that with reflinks.
> >>
> >> James
> >> _______________________________________________
> >> nix-dev mailing list
> >> nix-dev at lists.science.uu.nl
> >> http://lists.science.uu.nl/mailman/listinfo/nix-dev
> _______________________________________________
> nix-dev mailing list
> nix-dev at lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.science.uu.nl/pipermail/nix-dev/attachments/20150218/3640c6a6/attachment-0001.html