[Nix-dev] Sharing the Nix store between systems

[Rickard Nilsson]

Den 2011-12-13 01:20:00 skrev Marc Weber <marco-oweber at gmx.de>:

>> I had it figured out quite well, with one "master" machine that could
>> perform all writes to the store and to the Nix database, but then I
>> discovered that I could not get this master machine to cross-build ARM  
>> for
>> a complete NixOS system. So, then I tried to share both the store and  
>> the
>> Nix database with write permissions to two different systems (one ARM  
>> and
>> one x86_64). Unfortunately, the database got severly corrupted when both
>> machines was building things at the same time. I guess this is a feature
>> of sqlite+NFS+file locks.
> NFS can be used in different ways. If you use the kernel build in or
> such locking does not work.
>
> I'm interested in this topic as well - because it is the fastest way to
> distribute builds without copying store paths first.
>
>> So, now I'm looking for suggestions on what to try next. I guess the
>> sqlite problem could be solved if a PostgreSQL or MySQL backend was
> No, you're probably not going to solve the issue because there are not
> only sqlite locks but also locks on .drv files and such ? I'm not 100%
> sure about it - but you should double check it..
>
> Instead of using MySQL you could also create a binary protocol and run
> all sqlite operations on the host. AFAIK the nixpkgs code does separate
> the store implementation from rest - thus it should be easy to add
> another protocol based implementation. But this does not solve locking
> issues on .drv files and such as noted above.

And it does seem a bit overkill. But so is probably a standalone SQL  
server, given Nix's small DB schema.


> Thus the easiest way is to get the locking working ...
> AFAIK NFS does support locking if configured correctly.
> Eg see:
> http://nfs.sourceforge.net/
> http://www.centos.org/docs/5/html/Deployment_Guide-en-US/ch-nfs.html (->  
> rpc.lockd)
> Looks like since V4 those additional lock protocols are no longer
> required? So maybe enure that V4 is used?
> http://tldp.org/HOWTO/NFS-HOWTO/security.html (6.3.3 => quote: 'full file
> locking is supported')
>
> So I'm pretty sure that its your configuration.

Yes, NFS4 would probably help. I'm running v3 now. The problem is that at  
least the store needs to be mounted already in the initrd, and I have  
never managed to get NFS4 running so early, not in NixOS and not in other  
distributions. It should not be any problems using v4 for the database  
though, I will try that. But if you say there's also locks in the store  
then maybe it isn't sufficient. I will try to look around for the store  
locks.


> Keep us up to date about your progress, please.

I will.


   / Rickard