[Nix-dev] Setup GlusterFS server on NixOS machines

[James]

rohit yadav <rohityadav7787 <at> gmail.com> writes:




> James,

> I am certainly interested. However, I am relatively new to OS world and
haven't dealt with lower level details, though I am a linux user for
sometime. Managing cluster with ubuntu/debian posed a significant challenge
to me. Any experienced user would probably have no difficulty with standard
distributions but I found it very difficult. 


Not true. Even the commercial 'big data centers' "reload" systems all
the time for a variety of reasons. Long running clusters get hacked
all the time, if they face the internet and have many VMs and containers.
There's also the "cruft" cleanup maintenance issues on big clusters. All
sorts of orphaned files just keep building up and up and up.... There
are lots of other problems with both build time and runtime dependancies
too. There are many more problems, that are out of scope here to, with
cluster maintenance.


> Then, I found NixOS which offers significant advantage over other system
(ex: create a single file to manage the whole operating system with atomic
rollback, therefore I could experiment several things without fear of
leaving system unstable). I do not have specific preference for init system,
I guess each have pros and cons.


Is top-posting the norm for this list? I usually  respond "inline" to keep
the thread readable and flow consistent. Surely this list has a convention
(format) for posting?




> I am mostly interested in building packages in deterministic way, where
NixOS shines (its actually the feature offered by Nix, which you can install
on Gentoo too I believe). Therefore, the packages you would build in nix
would still be usable on other platforms. You can get started with NixOS
very easily.

OK, I'll accept this premise and verify it later.


> - Download an ISO image from here: "https://nixos.org/nixos/download.html"
> - Follow these instruction:
"https://nixos.org/nixos/manual/sec-installation.html"

> Lets get started with building the cluster of our choice. My first
priority is to really build a gluster/ceph based cluster to totally migrate
to NixOS and experiment with other packages.

I'm not familiar with gluster. Are there some links to the disk formatting
options to follow for setting up the hard disk with gluster? The example you
pointed to was for ext4. Any caveats with glusterfs?


OK, but it's going to be a few days (weekend?) before I get around to
installing a Nixos system.  I'll start with a (3) node cluster to
get things working. Have you already built such a nixos system as you
describe below? I'm not interested in installing nixos on top of another
distro. That is a huge performance penalty. Bare metal (my previously stated
goal) is all about performance. It's pretty much accepted as a fundamental
premise that ultimate performance of a cluster cannot be achieved by using
VMs, particularly High Performance Computing (HPC).
If you want you can use "containers" on your cluster to run many different
frameworks.  VMs are kinda dead in clusters that are alive and moving
forward with robust development, imho.


> Thanks,
> Rohit



> On Mon, Sep 28, 2015 at 9:28 AM, James <wireless <at> tampabay.rr.com> wrote:
> Roger Qiu <roger.qiu <at> polycademy.com> writes:
> >         I have very similar goals. It would be great if there are more
> >           individuals.>             In fact the main packages I need,
seem to all be in nixos::
> >             mesos, spark, zookeeper, marathon, storm and others. In fact
> >             all I could
> >             find missing is 'tachyon'.I have most of these packages in
gentoo, except tachyon and spark. If we do
> not find a quorum in nixos, as I know little about nixos other than what
> I have recently read, then I'll just complete the gentoo ebuilds and
> at that point I'll need folks to test and provide feedback. I also use
> openrc (so it's a non-systemd) cluster effort on my path. But if
> folks want to use systemd to build a mesos cluster, on gentoo, I think
> that too is very possible, as gentoo supports both openrc and systemd.
> Many are flocking to gentoo right now, because we have the best alternative
> to systemd; openrc [1].
> The clusters I'm interested in building, are optimized for performance::
> not only in the kernel but direct control over cgroups [2] to opetimze
> performance with bare-metal granularity over components. It's impossible to
> get that with most distro oriented cluster offerings. This is necessary
> to optimize a cluster for singularly large/complex tasks, like computational
> chemistry, sub-surface modeling of fluid flows, complex graphics and video
> rendering and analytics, just to name a few.
> Furthermore, the 'bare metal' approach will all for optimizing all sorts
> of distributed processing, available now via Clang-3.7.x (openmp)[3] and
> gcc-5.2 (OpenAcc). [4,5.6]
> Still, if nixos has all of this going on, the I'd like to test drive nixos.
> Maybe one of the devs can whip together a lived CD 'howto' to see it run?
> Then we can run a hi performance linpack on it (sys-cluster/hpl in gentoo)
> to benchmark the nixos offering?
> If you are interested, let's share emails and get a 'cluster club' started?
> Curious?
> James
> [1] https://wiki.gentoo.org/wiki/OpenRC
> [2] https://wiki.gentoo.org/wiki/OpenRC/CGroups
> [3] http://blog.cafarelli.fr/2015/09/testing-clang-3-7-0-openmp-support/
> [4] https://gcc.gnu.org/wiki/OpenACC
> [5] https://gcc.gnu.org/wiki/Offloading
> [6] http://www.phoronix.com/scan.php?page=news_item&px=GCC-5-Offloading-How-To
> 
> _______________________________________________
> nix-dev mailing listnix-dev <at>
lists.science.uu.nlhttp://lists.science.uu.nl/mailman/listinfo/nix-dev
> 
> 
> 
> 
> 
> 
> 
> _______________________________________________
> nix-dev mailing list
> nix-dev <at> lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>