[Nix-dev] RFC: Resolving impure Darwin dependencies in nix itself

[Jude]

Hi all! I'm back with another huge potentially-not-breaking change, this time in the nix codebase.

To give a short explanation: another critical problem that us Darwin folk have discovered is a difference in the impure link-time dependencies for certain system libraries between OSX Yosemite and OSX El Capitan.

A prime example of this issue is /System/Library/Frameworks/Versions/A/IOKit, which, during the upgrade from 10.10 to 10.11, incurred a dependency on /usr/lib/libScreenReader.dylib. This library doesn't exist on OSX 10.10.

We maintain a mapping of libraries to their impure dependencies here: https://github.com/NixOS/nixpkgs/blob/master/pkgs/os-specific/darwin/apple-sdk/impure-deps.nix, but they need to be statically known, and we can't add "false positives." That is, if /usr/lib/libScreenReader.dylib is added to the list of dependencies for IOKit, sandboxed builds will start failing on OSX 10.10.

There are three possible workarounds here:

1. Disable the sandbox completely for ld and dyld. This is actually pretty easy; the Apple sandbox supports a directive `(allow process-exec (subpath "...") (with no-sandbox))`, so neither ld or dyld will be sandboxed at all. This isn't an ideal situation; for example, cmake will again be able to find our system copy of libncurses.dylib. Installing Xcode results in some new libraries being installed on the system, and this could potentially cause runtime failures if a package is built on an Xcode-installed system and run on a non Xcode-installed system.
2. Add a conditional in the Darwin framework builder to add/remove impure deps from the list based on the current system version. This is a bit frail, and we'll need to continue to update it as new OSX versions are released.
3. Resolve the impure dependencies during nix-build. This is actually recently possible thanks to excellent work by Shea Levy in https://github.com/NixOS/nix/pull/510, which gives nix the capability to run a program before a build that can add a customized list of impure dependencies to the build environment.

Of these options, I found #3 to be the most user-friendly; if the pre-build hook is built in to nix, end users should never even have to know it's there. Developers, meanwhile, only need to know the absolute path to a library that they're attempting to link with, and not the entire dependency tree of that library.

My pull request on nix (#562, https://github.com/NixOS/nix/pull/562) sets a default pre-build-hook script on Darwin. See https://github.com/pikajude/nix/commit/ff6953cb031c0d2a625bbb2df985a7a3bb17bc46 for the relevant changes. In the PR I add a perl script called resolve-system-dependencies.pl. 

It's as simple as reasonably possible; it uses perl's Storable module to store a map of dependencies in ${stateDir}/dependency-maps. The map filenames are keyed on the current system architecture and OS version, thus OS upgrades will result in a map regeneration. Storable obtains an exclusive lock on the map file, so simultaneous attempts to fill out the dependency map won't trample over each other.

The script uses only the Perl standard library, since I wasn't sure where we stand on adding new dependencies.

The darwin "cctools" package will now be a runtime dependency of nix on Darwin systems; we can't rely on otool being present on the host system, since it's part of Xcode.

Finally, on Darwin only, "pre-build-hook" is defaulted to this script if not already set.

I'd love input from you all, especially nix maintainers, on what you all think, and whether my solution could be improved!

Thanks,
Jude
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.science.uu.nl/pipermail/nix-dev/attachments/20151021/713846b9/attachment.html