[Nix-dev] nix on compute cluster?

[Wout Mertens]

On Fri, Oct 10, 2014 at 4:34 PM, Andreas Herrmann <andreash87 at gmx.ch> wrote:

> Thank you for the detailed answer.
>
> On Friday 10 October 2014 15:32:52 Wout Mertens wrote:
> > I think you could do this. You would set it up so the nix server does the
> > compiles and the grid runs distcc. See the wiki, the raspberry pi page
> has
> > explanations about distcc.
> Oh, I didn't know that this worked outside of NixOS. I just can't find any
> details on how to integrate distcc with sge. Do you have any experience
> with that?
>

No sorry, but I'd just install distcc as a daemon on all nodes or else use
sge for the distribution somehow... pretend that they're super long running
batch jobs...


> > Note that only one node can write to nix store at the same time due to
> the
> > db.
> That's interesting. The multi-user nix manual says that you can have
> multiple build users on one machine. Why does that work on one machine, but
> not spread over several machines? I would assume that you need to connect
> to some kind of db daemon anyway?
>

it's because of sqlite, it is safe on a single machine but not on NFS...

> Another option is to have private nix stores on all nodes but nfs mount
> all
> > of them under the remote stores directory. That way nix-store will fetch
> > missing packages from the remotes and store them locally. At least,
> that's
> > my understanding.
> I don't think I understand this. By private, do you mean one per user? And
> how do we mount all of them under one directory, do you mean some layered
> fs?
>

No I mean one per node, so on the local disk. 50GB should be ok I think, or
a few GB if that's hard but then do nix-collect-garbage often.


> > As for the Intel compiler, that could be a challenge, but right now we
> have
> > several gcc versions and clang, so it's not impossible. You can decide
> on a
> > per-package basis which compiler to use.
> Yes, my main concern are things like license keys, or - even worse -
> license servers. I don't know how to approach these things in nix. As long
> as you can mix your nix environment with an external environment this is
> not an issue. So, as a simple example, you could use nix to manage your
> version of boost, but use an intel compiler that is installed outside of
> nix for user code.
>

so that's a bit harder. If you need a license connection and for the rest
the build is stateless, you can make that work I think. You'd probably need
to cheat with a userhook for stdenv to get the license connection info in
there though.

> Not sure how mpi would influence state, can you elaborate?
> I meant this on a per job basis. In the case of mpi jobs the sge will
> decide which nodes your job will be distributed over. This information will
> be passed by a mixture of config files and environment variables. Hence,
> such jobs would not work inside a pure nix-shell, because the environment
> needs to be passed through. I don't think it's a big issue. Just something
> to keep in mind.
>

more userhooking :) See
https://nixos.org/wiki/Raspberry_Pi#Implementing_distcc

Cheers,

Wout.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.science.uu.nl/pipermail/nix-dev/attachments/20141010/04afe2d6/attachment.html