[Nix-dev] nix on compute cluster?

[Andreas Herrmann]

Thanks for your answer.

On Tuesday 14 October 2014 09:16:54 Alexander Kjeldaas wrote:
> On Mon, Oct 13, 2014 at 10:14 PM, Andreas Herrmann <andreash87 at gmx.ch>
> wrote:
> > On Saturday 11 October 2014 23:36:36 you wrote:
> > > For a compute grid, where no graphics, no desktop, no kde etc. are
> > > compiled, a full NixOS build is in the hours, not days.   I would simply
> > > compile this on a single server and rdist,  No need to over-engineer
> > this.
> > Well, the reason I'm worried is that if any, or at least a number of
> > users, could cause recompilation of a fairly large number of libraries,
> > then this could make the login node quite sluggish. And I remember the
> > compilation of libatlas alone taking a whole night on my machine.
>
> I guess it depends on how many changes to the nixpkgs tree will happen.  I
> have never used any pre-cooked binaries and I'm generally using low-end
> Xeons.  I'm only using a basic set of development tools though (gcc, ghc,
> bunch of haskell libraries, emacs ...).  I do track nixpkgs HEAD, so I
> recompile quite a lot, but it it just something I do in a terminal.
One pattern that I noticed for myself, is that I often need to add new features to existing libraries. E.g. SGE mode for MPI, or MPI mode for hdf5. If many packages depend on these, then this will cause a large number of rebuilds.

But maybe we should just try a simple multi-user install and see how it works. One could hope that most of the heavy compilation would be done in a few days, after which it's only small compile jobs that are already done regularly by the users.

> Just compare the scenarios between a normal RHEL installation and Nix.  A
> developer asks for libatlas version Y and version X is installed.  2 weeks
> of debate ensures.  Using Nix, you just start compiling.  Ok, so i takes
> the night, about the same time it would take to discuss it on the mailing
> list.
I'm afraid, things are far less organized than you might think. Also, please note that I'm not the admin (we have one) I'm just an enthusiastic user...
Unfortunately, the current state of affairs is that - in many cases - just everyone compiles their own version of a library they need in their home-directory. So, everyone has their own version of boost, and a number of people even have their own gcc. I think that's a bit of a waste of time and space. Fortunately, atlas scale libraries exist globally.

> I'm not sure why your users would trigger a recompilation.  If I were you I
> would have one blessed nipkgs tree, and if they wanted a particular version
> of some library, do it as a commit to that tree.   If you have every user
> have their own patched nixpkgs tree, that might cause problems, yes.
Currently, I have nix installed in single user mode. And I am using it to handle all sorts of weird little dependencies that can't be found in any distributions repos. These are quite domain specific to the work that we are doing, and different people depend on different packages. However, for me nix has been an incredible productivity booster because it became so easy to swap packages and their dependencies around. By now I do also write nix-packages for my own projects, it makes it far easier to share code among them. Finally, the myEnvFun feature is very useful to encapsulate dependencies, and run jobs in different environments simultaneously.

However, these benefits are only available if you can have your own `~/.nixpkgs/config.nix` with your own package overloads. But, once a user has that he can cause any arbitrary recompilation.

> > One question: Why do you thing rdist would be better than a (cached) NFS
> > mounted nix-store?
> NFS with a cache should work fine.  It's been many years since I used NFS
> and at the time only IRIX had a decent NFS cache.
That's a fair point. I have no idea how well the NFS cache works on CentOS. I would just hope that it shouldn't be so bad in a scenario where the only time-critical operations are reads from the nodes and writes are seldom.

Best, Andreas