[Nix-dev] nix on compute cluster?

[Andreas Herrmann]

Thank you for the detailed answer.

On Friday 10 October 2014 15:32:52 Wout Mertens wrote:
> I think you could do this. You would set it up so the nix server does the
> compiles and the grid runs distcc. See the wiki, the raspberry pi page has
> explanations about distcc.
Oh, I didn't know that this worked outside of NixOS. I just can't find any details on how to integrate distcc with sge. Do you have any experience with that?

> Note that only one node can write to nix store at the same time due to the
> db.
That's interesting. The multi-user nix manual says that you can have multiple build users on one machine. Why does that work on one machine, but not spread over several machines? I would assume that you need to connect to some kind of db daemon anyway?

> Another option is to have private nix stores on all nodes but nfs mount all
> of them under the remote stores directory. That way nix-store will fetch
> missing packages from the remotes and store them locally. At least, that's
> my understanding.
I don't think I understand this. By private, do you mean one per user? And how do we mount all of them under one directory, do you mean some layered fs?

> As for the Intel compiler, that could be a challenge, but right now we have
> several gcc versions and clang, so it's not impossible. You can decide on a
> per-package basis which compiler to use.
Yes, my main concern are things like license keys, or - even worse - license servers. I don't know how to approach these things in nix. As long as you can mix your nix environment with an external environment this is not an issue. So, as a simple example, you could use nix to manage your version of boost, but use an intel compiler that is installed outside of nix for user code.

> Not sure how mpi would influence state, can you elaborate?
I meant this on a per job basis. In the case of mpi jobs the sge will decide which nodes your job will be distributed over. This information will be passed by a mixture of config files and environment variables. Hence, such jobs would not work inside a pure nix-shell, because the environment needs to be passed through. I don't think it's a big issue. Just something to keep in mind.

Best,

Andreas