[Nix-dev] Please test Nix store auto-optimise

[Oliver Matthews]

Okay, initial run (small, relatively new box running ext4)

du gave 1.4gb before, 1.2gb after.

Oddly, it told me 80.05mb had been saved; even with rounding I am uncertain how 80mb~=0.2gb

Will report back if it falls over later.


Regards,
Oliver

---- On Thu, 12 Feb 2015 10:47:11 +0000 Wout Mertens<wout.mertens at gmail.com> wrote ---- 

Hi all,

(TL;DR: regain disk space, see at the bottom for instructions)


the Nix store can be optimised by hard-linking files with the same contents together, so they are only stored once. This is done by reading all the files in your store, calculating their checksums and linking them to /nix/store/.links/[checksum]. This is especially useful in Nix where differing build inputs for a package end up with many of the same files. 


Nix 1.8 has a few optimizations that make this operation very fast if you run it multiple times. Nix also has the auto-optimise-store setting, which will link new files added to the store. If your store was optimised, it stays optimised.


I feel this optimisation should be turned on by default but there were some regressions in the past which is why it wasn't. Therefore I'd like to ask you to turn on auto-optimise and run optimisation once. Your disk space and memory footprint will thank you.


Advantage:
- Use less disk space and memory cache
Disadvantages:
- Need to checksum all files written to the store
  * this is fast while writing the files because they still are in cache
- /nix/store/.links can get very large
  * a decent filesystem should handle it
  * E.g. ext3, ext4, btrfs, zfs etc should have no problems


Hopefully this will prove that it's safe to use and Hydra will start using it, which should give it tons more headroom for builds.


How to enable:
- Run "du -hs /nix/store" to see how much you're using


- If you use nixpkgs, put
    auto-optimise-store = true

in /etc/nix/nix.conf and run "nix-store --optimise -v".
- If you use NixOS, put 

    nix.extraOptions = "auto-optimise-store = true";
in your configuration.nix and, as root, run
"nixos-rebuild switch" and "nix-store --optimise -v".


- Run "du -hs /nix/store" to see how much you regained



It is safe to interrupt the optimisation. It is also safe to clear out /nix/store/.links but this will result in more work on the next optimisation run.


Please report any issues you encounter, and it would be nice if you reply to this mail in a while to report that you have encountered no issues.


Cheers,


Wout.



PS: Optimise is the British spelling, nix-store also has --optimize but not auto-optimize-store :)

 _______________________________________________ 
nix-dev mailing list 
nix-dev at lists.science.uu.nl 
http://lists.science.uu.nl/mailman/listinfo/nix-dev 





-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.science.uu.nl/pipermail/nix-dev/attachments/20150212/528d67d2/attachment.html