[Nix-dev] Please test Nix store auto-optimise

[Wout Mertens]

Hi all,

(TL;DR: regain disk space, see at the bottom for instructions)

the Nix store can be optimised by hard-linking files with the same contents
together, so they are only stored once. This is done by reading all the
files in your store, calculating their checksums and linking them to
/nix/store/.links/*[checksum]*. This is especially useful in Nix where
differing build inputs for a package end up with many of the same files.

Nix 1.8 has a few optimizations that make this operation very fast if you
run it multiple times. Nix also has the auto-optimise-store setting, which
will link new files added to the store. If your store was optimised, it
stays optimised.

I feel this optimisation should be turned on by default but there were some
regressions in the past which is why it wasn't. Therefore I'd like to ask
you to turn on auto-optimise and run optimisation once. Your disk space and
memory footprint will thank you.

Advantage:
- Use less disk space and memory cache
Disadvantages:
- Need to checksum all files written to the store
  * this is fast while writing the files because they still are in cache
- /nix/store/.links can get very large
  * a decent filesystem should handle it
  * E.g. ext3, ext4, btrfs, zfs etc should have no problems

Hopefully this will prove that it's safe to use and Hydra will start using
it, which should give it tons more headroom for builds.

*How to enable*:
- Run "du -hs /nix/store" to see how much you're using

- If you use nixpkgs, put
    auto-optimise-store = true
in /etc/nix/nix.conf and run "nix-store --optimise -v".
- If you use NixOS, put
    nix.extraOptions = "auto-optimise-store = true";
in your configuration.nix and, as root, run
"nixos-rebuild switch" and "nix-store --optimise -v".

- Run "du -hs /nix/store" to see how much you regained

It is safe to interrupt the optimisation. It is also safe to clear out
/nix/store/.links but this will result in more work on the next
optimisation run.

Please report any issues you encounter, and it would be nice if you reply
to this mail in a while to report that you have encountered no issues.

Cheers,

Wout.

PS: Optimise is the British spelling, nix-store also has --optimize but not
auto-optimize-store :)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.science.uu.nl/pipermail/nix-dev/attachments/20150212/9cd45bbe/attachment.html