[Nix-dev] Deduplication

[Mathijs Kwik]

Raahul Kumar <raahul.kumar at gmail.com> writes:

> Thanks Kirill,
>
> I'll go email the BTRFS devs, see if they have a fix for this hardlink
> issue. Anyone else got a workaround for the dedup issue with Btrfs and
> nixos?

I think you misunderstood the output.
"There are X files with equal contents out of Y files in total"
is just the statistic that gets reported on finish.
It does not mean there are now still X files that can be deduplicated.
When all goes well, it means X files have all now been deduped.
On ext4 you get this same message as well.

So you can't really get an indication of the still-duplicate files that
are left because of btrfs' limit. But I think it's safe to just ignore
it. Files that have 3000+ (current btrfs max hardlink limit is just
under 4k) copies are probably just empty files/dirs anyway.



>
> Aloha,
> RK.
>
>
> On Mon, Apr 14, 2014 at 4:30 PM, Kirill Elagin <kirelagin at gmail.com> wrote:
>
>> I've just googled a bit, and you are probably hitting this:
>> http://lists.science.uu.nl/pipermail/nix-dev/2012-October/010017.html.
>>
>> So, you don't get around it. Or consider changing your FS ;).
>>
>>
>> --
>> Кирилл Елагин
>>
>>
>> On Mon, Apr 14, 2014 at 10:04 AM, Raahul Kumar <raahul.kumar at gmail.com>wrote:
>>
>>> I followed Eelco's suggestion of running  nix-store --optimise.
>>>
>>>  Output:
>>>
>>> `/nix/store/.links/09d75zzfhid0v9azq4qpprhfpfyjb27b89qnarqwz6bqgjxl7pq6'
>>> has maximum number of links
>>> 172.20 MiB freed by hard-linking 34905 files; there are 351670 files with
>>> equal contents out of 365448 files in total
>>>
>>>
>>> There are a lot files still duplicated. How do I get around the file has
>>> maximum number of links error?
>>>
>>> Aloha,
>>> RK.
>>>
>>> _______________________________________________
>>> nix-dev mailing list
>>> nix-dev at lists.science.uu.nl
>>> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>>>
>>>
>>
> _______________________________________________
> nix-dev mailing list
> nix-dev at lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev