[Nix-dev] Sharing the Nix store between systems

[Rickard Nilsson]

Hi,

I decided to solve the problem of store-sharing by
letting the nix-worker on localhost forward everything
to the nix-worker on another host. This way, I can
let a "client host" mount the store read-only, but
it is still possible to perform builds and installation,
since it is really the master host that does all
writing to the shared store. This is in my opinion
a very nice solution for thin disk-less clients.

One thing to note about this solution is that the
master of course don't add any gcroots for the new
builds, so the builds might be garbage collected
by the Nix master even though a client host uses
them. I intend to solve this by sharing some
subdirectory of /nix/var/nix/profiles on the master
to the hosts. That is orthogonal to the nix-worker
forwarding, though.

The forwarding was done like this:

   jobs.nixRemoteDaemon = {
     name = "nix-remote-daemon";
     startOn = "startup";
     script = ''
       export  
PATH=${pkgs.openssh}/bin:${pkgs.socat}/bin:${pkgs.coreutils}/bin
       NIX_SOCKET=/nix/var/nix/daemon-socket/socket
       NIX_REMOTE_SSHKEY=/etc/secrets/nix-remote-daemon.id_dsa
       NIX_REMOTE_USER=nixremote
       NIX_REMOTE_HOST=mynixmaster
       mkdir -p $(dirname "$NIX_SOCKET")
       rm -f "$NIX_SOCKET"
       exec socat UNIX-LISTEN:$NIX_SOCKET,fork EXEC:"ssh -i  
$NIX_REMOTE_SSHKEY $NIX_REMOTE_USER@$NIX_REMOTE_HOST socat -  
UNIX-CONNECT\:$NIX_SOCKET"
     '';
   };

I simply define a new job that listens on the nix-worker
socket and forwards everything via ssh to the nix-worker
socket on the master. This works flawlessly for me.

I want to ask for recommendations on how to package this
and submit it to Nix/NixOS. I my opinion, this should really
be a feature of nix-worker itself, maybe something like this:

   nix-worker --remote --remote-host ... --remote-user .. --remote-key ..

The simplest implementation of this is to just write a bash
wrapper around the existing nix-worker that uses socat and
ssh like above.

Is this a good idea, or should it rather be packaged as a separate
nix-remote-worker daemon?


Best regards,
   Rickard Nilsson


Den 2011-09-30 08:50:32 skrev Marc Weber <marco-oweber at gmx.de>:

> Excerpts from Rickard Nilsson's message of Thu Sep 29 23:30:31 +0200  
> 2011:
>> * How can I in NixOS specify which user that should do the builds? I  
>> would
>> like to use users and groups defined in a central LDAP catalog, and not
>> let NixOS create any local build users/groups.
> grep in nix source for nixbldX
>
>> * How would I stop Nix on one host from removing contents from the store
>> that is used by other hosts?
> patch nix or put references in /nix/var/nix/gcroots/..
>
>> * Is there a way to also throw a build server into the picture, so that
>> NixOS computers can ask the build server to build store paths, which  
>> would
>> then be put on the NFS share directly by the build server?
> If you have a client server setup there are remote builders. That's what
> currently works. See manual and nix.buildMachines.
> If you want to share the store so that you don't have to copy store
> paths first before building I don't know about existing solutions yet -
> but I'm interested in finding one as well.
>
> Marc Weber
> _______________________________________________
> nix-dev mailing list
> nix-dev at lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev