[Nix-dev] Please test Nix store auto-optimise

[Wout Mertens]

Since it looks for sub-blocks, first run the nix-store --optimise and then
run duperemove -dhr /nix/store.

Read the FAQ though:
https://github.com/markfasheh/duperemove/blob/master/FAQ.md

BTW, since bedup does files incrementally, it's nice to run that too. So
I'd do nix-store --optimise, bedup and then duperemove.

Of course if you don't want hardlinks (but there's really no reason on a
read-only store like /nix/store), you can leave the nix-store --optimise
out and only run bedup and duperemove.
If you want speed, only running nix-store --optimise and bedup should do
the trick, I wonder how much extra duperemove would save.

Wout.

On Wed Feb 18 2015 at 9:42:59 AM Raahul Kumar <raahul.kumar at gmail.com>
wrote:

> Can I know the best way to do this for BTRFS? Install duperemove, then
> what?
>
> Aloha,
> RK.
>
> On Mon, Feb 16, 2015 at 8:32 AM, James Cook <james.cook at utoronto.ca>
> wrote:
>
>> Oops, yes, that's a good point. Filesystems with fancy things like
>> reflinks probably have their own tools for deduplication.
>>
>> On 15 February 2015 at 01:48, Wout Mertens <wout.mertens at gmail.com>
>> wrote:
>> > If you want to use reflinks, better to use the deduplication tools that
>> come
>> > with the filesystem, e.g.
>> > https://btrfs.wiki.kernel.org/index.php/Deduplication (duperemove is in
>> > nixpkgs).
>> >
>> > Also, it would be nice if we could patch GNU cp so that it always tries
>> to
>> > use reflinks (so that builds automatically have reflinks where
>> possible).
>> > There's an option "--reflinks=auto" that does that and I don't
>> understand
>> > why it's not the default (as discussed at
>> >
>> http://unix.stackexchange.com/questions/80351/why-is-cp-reflink-auto-not-the-default-behaviour
>> ).
>> > Maybe we should make it a nixos option.
>> >
>> > Hardlinks are slightly impure and they do cause build failures in rare
>> and
>> > easily-fixed cases, but they're cross-platform, robust and easy to
>> inspect.
>> > Indeed the .links directory won't be able to be pruned due to reflinks
>> being
>> > invisible.
>> >
>> > To implement reflinks properly you would have to keep a hash of each
>> file in
>> > the nix-store in a DB. This has advantages, like being able to hash
>> lazily
>> > (only when another file with the same size shows up), and being able to
>> do
>> > queries, but it also means that you're duplicating that other DB, the
>> file
>> > system.
>> >
>> >
>> > Wout.
>> >
>> > On Sun, Feb 15, 2015, 9:54 AM James Cook <james.cook at utoronto.ca>
>> wrote:
>> >>
>> >> > Once I wondered if using reflinks instead of hardlinks might be
>> better
>> >> > from
>> >> > some point of view, but it probably won't be a big difference.
>> >>
>> >> I would really like to see reflinks being used instead of hard links
>> >> on filesystems that support it. Hard linking is an impurity which can
>> >> cause bugs, as Wout pointed out at the start of the thread (e.g.
>> >> https://github.com/NixOS/nixpkgs/issues/4266).
>> >>
>> >> Implementation question: how would nix know when it can delete a file
>> >> in /nix/store/.links? I assume than now it just checks the number of
>> >> links, but I don't know if you can do that with reflinks.
>> >>
>> >> James
>> >> _______________________________________________
>> >> nix-dev mailing list
>> >> nix-dev at lists.science.uu.nl
>> >> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>> _______________________________________________
>> nix-dev mailing list
>> nix-dev at lists.science.uu.nl
>> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.science.uu.nl/pipermail/nix-dev/attachments/20150218/d065f64d/attachment.html