[Nix-dev] Using a remote machine for builds with two NixOS machines.

[roconnor at theorem.ca]

I have a slow laptop that needs a custom kernel.  It usually takes 
overnight to recompile a kernel, so to combat this, I've recently set up 
my laptop to use my desktop to perform builds.

Since this process isn't so well documented, I though I would try to 
make a record of what I did, or rather what I would do if I were doing 
this again.  I'll replace the contents of 
https://nixos.org/wiki/Distributed_build with this, if there are no 
complaints.


Step 1.  Create and exchange signing keys.

To generate signing keys I vaguely follow the directions in 
https://github.com/NixOS/nix/blob/master/doc/signing.txt

As root I run

# (umask 277 && `nix-build --no-out-link "<nixpkgs>" -A libressl`/bin/openssl genrsa -out /etc/nix/signing-key.sec 4096)
# `nix-build --no-out-link "<nixpkgs>" -A libressl`/bin/Aopenssl rsa -in /etc/nix/signing-key.sec -pubout > /etc/nix/signing-key.pub

This generates /etc/nix/signing-key.sec and /etc/nix/signing-key.pub.
This step can be done on either the client or server.  The created files 
should be copied to the other machine.  Now both machines have copies of 
both the private and public signing keys.  These keys are used to share 
closures between the two machines, via nix-copy-closure --sign, without 
needing to necessarly use root accounts.


Step 2. Generate ssh keys

On the client (i.e. my laptop):

# ssh-keygen -t ed25519 -N "" -C nixBuild -f /root/.ssh/id_nixBuild -N ""

This will create both /root/.ssh/id_nixBuild and 
/root/.ssh/id_nixBuild.pub files.


Step 3. Create a nixBuild account

On the server (i.e. my desktop), update /etc/nixos/configuration.nix to 
add a new user

   users.extraUsers.nixBuild = {
     name = "nixBuild";
     useDefaultShell = true;
     openssh.authorizedKeys.keys = [ "ssh-ed25519 AAA... nixBuild" ];
   };

replacing the string in the openssh.authorizedKeys.keys with the contents 
of the /root/.ssh/id_nixBuild.pub file generated on the client machine.
The client will access the server via this account in order to run Nix 
commands.


Step 4. Enable distributed builds.

On the client (i.e. my laptop), update /etc/nixos/configuration.nix to 
enable distributed builds.

   nix = {
     distributedBuilds = true;
     buildMachines = [{
       hostName = "<insert your server machine's name here>";
       maxJobs = <fill in maxJobs>;
       sshKey = "/root/.ssh/id_nixBuild";
       sshUser = "nixBuild";
       system = "x86_64-linux";
     }];
   };

For the maxJobs field, run

$ nixos-option nix.maxJobs

on the server machine and use the "Value:" listed.

For the system field, run

$ nix-instantiate --eval -E 'builtins.currentSystem'

on both the client and server machines.  They should both print the same 
value, and you should use that value in the system field.  You cannot use 
a remote machine with a different system for remote builds.


And that is all.  For testing purposes, you can set the nix.maxJobs value 
in your client machines /etc/nixos/hardware-configuation.nix value 
temporarily to 0.  This will force all builds to be done remotely.  You 
can restore the value after you are done testing.

It goes without saying that after updating configuration.nix files, you 
will need to run nixos-rebuild to realize the changes.


Some general comments:  This setup means that both the client and server 
trust each other's binaries.  I'd prefer that only the client trust the 
server's binaries, and have the client only upload source files to the 
server to build; however I don't know how to force this.

Also it would be nice to restrict the nixBuild user on the server to only 
run nix-commands.

-- 
Russell O'Connor                                      <http://r6.ca/>
``All talk about `theft,''' the general counsel of the American Graphophone
Company wrote, ``is the merest claptrap, for there exists no property in
ideas musical, literary or artistic, except as defined by statute.''