[Nix-dev] Binary cache substituter

[Eelco Dolstra]

Hi all,

The latest nixUnstable in Nixpkgs has an experimental new feature called the
"binary cache substituter" intended to eventually replace the current
manifest/nix-pull-based substituter.

The main problem with the nix-pull approach is that it requires the manifest you
nix-pulled to be in sync with your Nix expressions.  If these are not in sync,
then Nix will fall back to building from source, even though binaries may be
available somewhere.

The new binary cache substituter doesn't pre-download a manifest; rather, every
time Nix needs to build a new store path, it will ask a list of HTTP servers
(the "binary caches") if they have a binary of that store path available.

For instance, if you add the line

  binary-caches = http://nixos.org/binary-cache

to your nix.conf, then if Nix needs a store path /nix/store/<hash>-<name>, then
it will probe http://nixos.org/binary-cache/<hash>.narinfo.  That file, if it
exists, will contain a pointer to a NAR file containing the binary, and metadata
like the size and references of the store path.  Example:

  http://nixos.org/binary-cache/cj7a81wsm1ijwwpkks3725661h3263p5.narinfo

contains

  StorePath: /nix/store/cj7a81wsm1ijwwpkks3725661h3263p5-glibc-2.13
  URL: nar/0k6335lpbcmdgncqwkcry4dxr2m6qs77zia51684ynjgc7n5xx21.nar.bz2
  Compression: bzip2
  FileHash: sha256:0k6335lpbcmdgncqwkcry4dxr2m6qs77zia51684ynjgc7n5xx21
  FileSize: 10119014
  NarHash: sha256:120mhshkcgvn48xvxz0fjbd47k615v0viv7jy4gy7dwhmqapgd9d
  NarSize: 33702192
  References: ... jfcs9xnfbmiwqs224sb0qqsybbfl3sab-linux-headers-2.6.35.14
  Deriver: xs32zzf2d58f6l5iz5fgwxrds2q5pnvn-glibc-2.13.drv
  System: x86_64-linux

In other words, it contains the information that used to be in nix-pull manifest
files, but only for one particular store path.

As an example of how useful this is, here is how you would get Firefox 2.0.0.6
from the ancient Nixpkgs 0.11 release:

  $ cd /my/nixpkgs/checkout
  $ git checkout 0.11
  $ nix-build -A firefox --argstr system i686-linux

The nix-build will notice that binaries of the requested Firefox are available
at http://nixos.org/binary-cache and use them.  You don't need to make sure you
have the right manifest installed or anything like that.  So this makes the Nix
notion of binary deployment as a "transparent" optimisation of source deployment
much more reality.

Some random points:

* Instead of setting binary-caches in nix.conf, you can specify/override binary
caches on the command line, e.g. "nix-build -A firefox --option binary-caches
http://...".

* When building through the Nix daemon, you cannot use --option binary-caches to
configure arbitrary caches because that would be a security hole.  You can only
specify a subset of the binary-caches and trusted-binary-caches values
configured in nix.conf.  "trusted-binary-caches" is a list of caches that are
not used by default but can be specified by untrusted users.

* Subscribing to a Nix channel will auto-configure the use of a corresponding
binary cache.  For instance, the channel
http://nixos.org/releases/nixos/channels/nixos-unstable will automatically cause
http://nixos.org/binary-cache to be used.

* A binary cache can be created using nix-push, e.g. "nix-push --dest /tmp/cache
$(which firefox)".

* The binary cache substituter supports xz compression.  Xz compresses about 50%
better than bzip2 and decompresses about twice as fast (at the price of
significantly slower compression, but that only needs to be done once per path
on the server).  See commit 4911a10a4e51102a21a5d123a852c75d2ec92dbc for some stats.

* The main technical challenge in implementing the binary cache substituter is
dealing with the latency involved in checking lots of remote .narinfo files.
Nix uses persistent HTTP connections and lots of parallelism to look up .narinfo
files efficiently.  For instance, if you do "nix-env -qas \*" on Nixpkgs (which
involves checking the existence of thousands of .narinfo files), Nix will spam
the server with 150 parallel HTTP connections.  This will probably require more
tweaking.  Positive and negative .narinfo lookups are cached in
/nix/var/nix/binary-cache-v1.sqlite.  Negative lookups are expired after a
certain time.

* The main downside to the binary cache substituter is that it has no support
for binary patches (and it's not obvious how to support them).  If you really
need them, you can add "force-manifest = true" to nix.conf to force nix-channel
to nix-pull the channel's manifest.

-- 
Eelco Dolstra | LogicBlox, Inc. | http://nixos.org/~eelco/