[Nix-dev] Please test Nix store auto-optimise

[Wout Mertens]

Well, it's in the nix-store library code so presumably that
nix-copy-closure passes through there. If you want to be sure, just run
nix-store --optimise -v and see if it finds anything to update.
Also, wow, 15GB saved :)

On Fri Feb 13 2015 at 6:43:05 PM Richard Wallace <
rwallace at thewallacepack.net> wrote:

> Thank you for pointing this option out! I had no idea it existed before
> and it should prove very useful!
>
> $ du -hs /nix/store/
> 38G /nix/store/
> $ sudo nix-store --optimise -v
> <snip>
> 12456.46 MiB freed by hard-linking 881561 files
> $ du -hs /nix/store/
> 23G /nix/store/
>
> Does the auto linking also happen if you use nix-copy-closure?
>
> Rich
>
> On Thu, Feb 12, 2015 at 3:47 AM, Wout Mertens <wout.mertens at gmail.com>
> wrote:
>
>> Hi all,
>>
>> (TL;DR: regain disk space, see at the bottom for instructions)
>>
>> the Nix store can be optimised by hard-linking files with the same
>> contents together, so they are only stored once. This is done by reading
>> all the files in your store, calculating their checksums and linking them
>> to /nix/store/.links/*[checksum]*. This is especially useful in Nix
>> where differing build inputs for a package end up with many of the same
>> files.
>>
>> Nix 1.8 has a few optimizations that make this operation very fast if you
>> run it multiple times. Nix also has the auto-optimise-store setting, which
>> will link new files added to the store. If your store was optimised, it
>> stays optimised.
>>
>> I feel this optimisation should be turned on by default but there were
>> some regressions in the past which is why it wasn't. Therefore I'd like to
>> ask you to turn on auto-optimise and run optimisation once. Your disk space
>> and memory footprint will thank you.
>>
>> Advantage:
>> - Use less disk space and memory cache
>> Disadvantages:
>> - Need to checksum all files written to the store
>>   * this is fast while writing the files because they still are in cache
>> - /nix/store/.links can get very large
>>   * a decent filesystem should handle it
>>   * E.g. ext3, ext4, btrfs, zfs etc should have no problems
>>
>> Hopefully this will prove that it's safe to use and Hydra will start
>> using it, which should give it tons more headroom for builds.
>>
>> *How to enable*:
>> - Run "du -hs /nix/store" to see how much you're using
>>
>> - If you use nixpkgs, put
>>     auto-optimise-store = true
>> in /etc/nix/nix.conf and run "nix-store --optimise -v".
>> - If you use NixOS, put
>>     nix.extraOptions = "auto-optimise-store = true";
>> in your configuration.nix and, as root, run
>> "nixos-rebuild switch" and "nix-store --optimise -v".
>>
>> - Run "du -hs /nix/store" to see how much you regained
>>
>> It is safe to interrupt the optimisation. It is also safe to clear out
>> /nix/store/.links but this will result in more work on the next
>> optimisation run.
>>
>> Please report any issues you encounter, and it would be nice if you reply
>> to this mail in a while to report that you have encountered no issues.
>>
>> Cheers,
>>
>> Wout.
>>
>> PS: Optimise is the British spelling, nix-store also has --optimize but
>> not auto-optimize-store :)
>>
>> _______________________________________________
>> nix-dev mailing list
>> nix-dev at lists.science.uu.nl
>> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.science.uu.nl/pipermail/nix-dev/attachments/20150213/78416ebe/attachment.html