[Nix-dev] NixOS hangs on boot after upgrading 15.09 -> 16.03; systemd update to blame?

[James Cook]

After updating my desktop from 15.09 to 16.03, it hangs on boot after
printing the message "regenerating udev hardware database". I do not
think the problem is directly related to that message; see below. Any
ideas for debugging this?

Here's what I've figured out so far:

* Using git-bisect, it seems that the problem starts at commit
ecbe04b4, "systemd: Update to 225". (I had to cherry-pick 3cc04956,
"libseccomp: Fix hash", to got it to build.)

* I don't think the script that prints "regenerating udev hardware
database" is the one that hangs. I added a debugging echo command to
the end of system.activationScripts.udevd (in
nixos/modules/services/hardware/udev.nix), and I saw the extra message
when I booted. So the hang happens after.

My next step was going to be to add more debugging statements to the
boot process, but I'm having trouble figuring out where any of that
stuff is. I can't find where system.activationScripts.udevd is called
from, and I was mystified to find that grep -R "egenerating udev"
/nix/store/(...)-nixos-16.0.3.git.3cc0495M produces no output (except
some errors, e.g. etc/zoneinfo/posix is a directoly loop).

James