[Nix-dev] Making Nix use pivot_root in addition to chroot

[Harald van Dijk]

Hello,

I've recently been trying to use Nix in a somewhat different way than 
NixOS, and the builders I've got, it would be useful if they were able 
to run unshare -r. Unfortunately, while this command is normally 
available to unprivileged users, it is for security reasons not 
available to unprivileged users in a chroot environment.

Reading up on how this works, I've come to understand that as one effect 
of unshare -r is enabling chroot for unprivileged users, and the chroot 
capability is well-known as being sufficient to break out of a chroot 
directory, those security concerns are correct, and it is appropriate 
for unshare -r to report an error if Nix is configured to use chroot.

However, if chroot is combined with pivot_root, then the mount namespace 
root and the process root are the same directory again. Since there is 
no longer anything nothing to break out of, there is no risk of a user 
breaking out of anything, and then, unshare -r _does_ work.

A possible problem with pivot_root could have been that it lets the old 
root remain available inside the new root, but that is easily prevented 
by unmounting it afterwards.

The attached patch (to Nix 1.7) to use pivot_root is mainly a 
proof-of-concept, I know it is not appropriate in its current form. At 
the very least, it should properly handle systems that lack a pivot_root 
syscall, and even on systems that do have it, it might be better to make 
the use of pivot_root optional (compile-time as well as run-time).

Nonetheless, in its current form it is already useful for testing, which 
has shown that it addresses the problem for me that I had hoped for it 
to address, and that it does not cause problems for NixOS packages, 
which I do continue to use alongside my own.

Is there any interest in getting something like this in Nix? I would be 
perfectly happy to clean this patch up, get it into better shape, but 
I'd like to avoid doing so if (for whatever reason) it is decided that 
Nix should not be using this.

Cheers,
Harald van Dijk
-------------- next part --------------
--- a/src/libstore/build.cc
+++ b/src/libstore/build.cc
@@ -51,6 +51,7 @@
 #define CHROOT_ENABLED HAVE_CHROOT && HAVE_UNSHARE && HAVE_SYS_MOUNT_H && defined(MS_BIND) && defined(MS_PRIVATE) && defined(CLONE_NEWNS)
 
 #if CHROOT_ENABLED
+#include <sys/syscall.h>
 #include <sys/socket.h>
 #include <sys/ioctl.h>
 #include <net/if.h>
@@ -2021,6 +2022,11 @@ void DerivationGoal::initChild()
                     throw SysError(format("unable to make filesystem `%1%' private") % fs);
             }
 
+            /* Bind-mount chroot directory to itself, to treat it as a
+               different filesystem from /, as needed for pivot_root. */
+            if (mount(chrootRootDir.c_str(), chrootRootDir.c_str(), 0, MS_BIND, 0) == -1)
+                throw SysError(format("unable to bind mount `%1'") % chrootRootDir);
+
             /* Set up a nearly empty /dev, unless the user asked to
                bind-mount the host /dev. */
             if (dirsInChroot.find("/dev") == dirsInChroot.end()) {
@@ -2093,13 +2099,26 @@ void DerivationGoal::initChild()
 		chmod_(chrootRootDir + "/dev/pts/ptmx", 0666);
             }
 
-            /* Do the chroot().  Below we do a chdir() to the
-               temporary build directory to make sure the current
-               directory is in the chroot.  (Actually the order
-               doesn't matter, since due to the bind mount tmpDir and
-               tmpRootDit/tmpDir are the same directories.) */
-            if (chroot(chrootRootDir.c_str()) == -1)
+            /* Do the chroot(). */
+            if (chdir(chrootRootDir.c_str()) == -1)
+                throw SysError(format("cannot change directory to `%1%'") % chrootRootDir);
+
+            if (mkdir("real-root", 0) == -1)
+                throw SysError("cannot create real-root directory");
+
+#define pivot_root(new_root, put_old) (syscall(SYS_pivot_root, new_root, put_old))
+            if (pivot_root(".", "real-root") == -1)
+                throw SysError(format("cannot pivot old root directory onto `%1%'") % (chrootRootDir + "/real-root"));
+#undef pivot_root
+
+            if (chroot(".") == -1)
                 throw SysError(format("cannot change root directory to `%1%'") % chrootRootDir);
+
+            if (umount2("real-root", MNT_DETACH) == -1)
+                throw SysError("cannot unmount real root filesystem");
+
+            if (rmdir("real-root") == -1)
+                throw SysError("cannot remove real-root directory");
         }
 #endif