[Nix-dev] Please test Nix store auto-optimise

[Richard Wallace]

Thank you for pointing this option out! I had no idea it existed before and
it should prove very useful!

$ du -hs /nix/store/
38G /nix/store/
$ sudo nix-store --optimise -v
<snip>
12456.46 MiB freed by hard-linking 881561 files
$ du -hs /nix/store/
23G /nix/store/

Does the auto linking also happen if you use nix-copy-closure?

Rich

On Thu, Feb 12, 2015 at 3:47 AM, Wout Mertens <wout.mertens at gmail.com>
wrote:

> Hi all,
>
> (TL;DR: regain disk space, see at the bottom for instructions)
>
> the Nix store can be optimised by hard-linking files with the same
> contents together, so they are only stored once. This is done by reading
> all the files in your store, calculating their checksums and linking them
> to /nix/store/.links/*[checksum]*. This is especially useful in Nix where
> differing build inputs for a package end up with many of the same files.
>
> Nix 1.8 has a few optimizations that make this operation very fast if you
> run it multiple times. Nix also has the auto-optimise-store setting, which
> will link new files added to the store. If your store was optimised, it
> stays optimised.
>
> I feel this optimisation should be turned on by default but there were
> some regressions in the past which is why it wasn't. Therefore I'd like to
> ask you to turn on auto-optimise and run optimisation once. Your disk space
> and memory footprint will thank you.
>
> Advantage:
> - Use less disk space and memory cache
> Disadvantages:
> - Need to checksum all files written to the store
>   * this is fast while writing the files because they still are in cache
> - /nix/store/.links can get very large
>   * a decent filesystem should handle it
>   * E.g. ext3, ext4, btrfs, zfs etc should have no problems
>
> Hopefully this will prove that it's safe to use and Hydra will start using
> it, which should give it tons more headroom for builds.
>
> *How to enable*:
> - Run "du -hs /nix/store" to see how much you're using
>
> - If you use nixpkgs, put
>     auto-optimise-store = true
> in /etc/nix/nix.conf and run "nix-store --optimise -v".
> - If you use NixOS, put
>     nix.extraOptions = "auto-optimise-store = true";
> in your configuration.nix and, as root, run
> "nixos-rebuild switch" and "nix-store --optimise -v".
>
> - Run "du -hs /nix/store" to see how much you regained
>
> It is safe to interrupt the optimisation. It is also safe to clear out
> /nix/store/.links but this will result in more work on the next
> optimisation run.
>
> Please report any issues you encounter, and it would be nice if you reply
> to this mail in a while to report that you have encountered no issues.
>
> Cheers,
>
> Wout.
>
> PS: Optimise is the British spelling, nix-store also has --optimize but
> not auto-optimize-store :)
>
> _______________________________________________
> nix-dev mailing list
> nix-dev at lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.science.uu.nl/pipermail/nix-dev/attachments/20150213/1ac9bb16/attachment-0001.html