[Nix-dev] Reproducibility testing in Hydra

[Eelco Dolstra]

Hi,

On the occasion of the second Reproducible Builds workshop currently taking
place in Berlin [1], I'm happy to announce that Hydra is now capable of
repeating builds in order to test for certain types of binary reproducibility
problems. Hydra can now be configured to run every build step in a jobset
multiple times, comparing the result. If the result differs between rounds, the
build step will still succeed, but the step will be marked as "Succeeded with
non-deterministic result". (Thus reverse dependencies can still proceed.) For
example:

  http://hydra.nixos.org/build/44628665#tabs-buildsteps

Furthermore, if there is a difference between rounds, diffoscope (a tool for
comparing arbitrary directory trees [2]) is run on the the resulting store
paths. For example, here is the build log for Samba:

  http://hydra.nixos.org/build/44628691/nixlog/2/raw

The diffoscope output is at the end (starting at "diff with previous round"),
showing that Samba stores a timestamp in lib/samba/libsmbd-base-samba4.so.

Similarly, here is the log for python-urllib3:

  http://hydra.nixos.org/build/44713633/nixlog/4/raw

This shows that the pip build tool uses a random temporary directory, which ends
up in .pyc files. (This affects almost 1800 packages.)

Currently, out of 36550 build steps in a full NixOS evaluation, only 4699
(12.8%) were detected as non-deterministic. (Note that Hydra can only detect the
presence of non-determinism, not its absence). Here is the full list:

  http://nixos.org/~eelco/tmp/nondeterministic-build-steps.txt

Currently only the nixos:reproducibility jobset runs build steps multiple times
(twice, to be precise).

[1] https://reproducible-builds.org/events/berlin2016/
[2] https://diffoscope.org/

-- 
Eelco Dolstra | LogicBlox, Inc. | http://nixos.org/~eelco/