[Nix-dev] Setup GlusterFS server on NixOS machines
rohit yadav
rohityadav7787 at gmail.com
Tue Sep 29 18:07:53 CEST 2015
NixOS is a OS-distribution based on "nix", the package manager. You have
option of trying out nix on any standard distribution (
https://www.domenkozar.com/2014/01/02/getting-started-with-nix-package-manager/).
Once, you understand nix and build packages with it and fine it interesting
then you can move on to NixOS. However, I have found it to be problematic.
Once in a while, few packages points to native distribution binaries. So,
if you are experienced use then you can directly install NixOS on bare
metal. Use ext4, xfs etc formatted disk to install NixOS. Please provided
atleast 75-80 GB drive for "/nix/store", the place where it puts binaries.
Since, each package is standalone and does not share with other binaries,
the size can add up pretty quickly. However, there are methods to
internally optimize by creating hard-links between binaries.
Once you have NixOS. Further, to install gluster, you should format disk
with xfs (recommended) or btrfs and mount those volumes. GlusterFS is not
native filesystem. It is layer on top of the standard filesystem. Its keep
in sync multiple volumes distributed across multiple computers in a set of
trusted cluster. It does not have authorization system yet (or atleast I
haven't researched enough to find one). You can stripped volume or
replicated volume or combination of both. Usually, I keep atleast 2
replicated volume on one cluster. Create gluster bricks on the mounted
volumes dedicated to gluster and create volumes which has several number of
bricks in the configuration you like. Once volumes are created and started,
you can mount those volumes to any client machine in that cluster. Now you
have you a private NAS with commodity hardware.
I haven't yet been able to use the available gluster binary on NixOS
channels to setup a cluster with glusterFS (though I have one running one
ubuntu). I am experimenting with NixOS on my laptop and a VM, no real
hardware is involved yet.
Hope this helps.
Thanks,
Rohit
On Tue, Sep 29, 2015 at 10:27 AM, James <wireless at tampabay.rr.com> wrote:
> rohit yadav <rohityadav7787 <at> gmail.com> writes:
>
>
>
>
> > James,
>
> > I am certainly interested. However, I am relatively new to OS world and
> haven't dealt with lower level details, though I am a linux user for
> sometime. Managing cluster with ubuntu/debian posed a significant challenge
> to me. Any experienced user would probably have no difficulty with standard
> distributions but I found it very difficult.
>
>
> Not true. Even the commercial 'big data centers' "reload" systems all
> the time for a variety of reasons. Long running clusters get hacked
> all the time, if they face the internet and have many VMs and containers.
> There's also the "cruft" cleanup maintenance issues on big clusters. All
> sorts of orphaned files just keep building up and up and up.... There
> are lots of other problems with both build time and runtime dependancies
> too. There are many more problems, that are out of scope here to, with
> cluster maintenance.
>
>
> > Then, I found NixOS which offers significant advantage over other system
> (ex: create a single file to manage the whole operating system with atomic
> rollback, therefore I could experiment several things without fear of
> leaving system unstable). I do not have specific preference for init
> system,
> I guess each have pros and cons.
>
>
> Is top-posting the norm for this list? I usually respond "inline" to keep
> the thread readable and flow consistent. Surely this list has a convention
> (format) for posting?
>
>
>
>
> > I am mostly interested in building packages in deterministic way, where
> NixOS shines (its actually the feature offered by Nix, which you can
> install
> on Gentoo too I believe). Therefore, the packages you would build in nix
> would still be usable on other platforms. You can get started with NixOS
> very easily.
>
> OK, I'll accept this premise and verify it later.
>
>
> > - Download an ISO image from here: "
> https://nixos.org/nixos/download.html"
> > - Follow these instruction:
> "https://nixos.org/nixos/manual/sec-installation.html"
>
> > Lets get started with building the cluster of our choice. My first
> priority is to really build a gluster/ceph based cluster to totally migrate
> to NixOS and experiment with other packages.
>
> I'm not familiar with gluster. Are there some links to the disk formatting
> options to follow for setting up the hard disk with gluster? The example
> you
> pointed to was for ext4. Any caveats with glusterfs?
>
>
> OK, but it's going to be a few days (weekend?) before I get around to
> installing a Nixos system. I'll start with a (3) node cluster to
> get things working. Have you already built such a nixos system as you
> describe below? I'm not interested in installing nixos on top of another
> distro. That is a huge performance penalty. Bare metal (my previously
> stated
> goal) is all about performance. It's pretty much accepted as a fundamental
> premise that ultimate performance of a cluster cannot be achieved by using
> VMs, particularly High Performance Computing (HPC).
> If you want you can use "containers" on your cluster to run many different
> frameworks. VMs are kinda dead in clusters that are alive and moving
> forward with robust development, imho.
>
>
> > Thanks,
> > Rohit
>
>
>
> > On Mon, Sep 28, 2015 at 9:28 AM, James <wireless <at> tampabay.rr.com>
> wrote:
> > Roger Qiu <roger.qiu <at> polycademy.com> writes:
> > > I have very similar goals. It would be great if there are more
> > > individuals.> In fact the main packages I need,
> seem to all be in nixos::
> > > mesos, spark, zookeeper, marathon, storm and others. In
> fact
> > > all I could
> > > find missing is 'tachyon'.I have most of these packages in
> gentoo, except tachyon and spark. If we do
> > not find a quorum in nixos, as I know little about nixos other than what
> > I have recently read, then I'll just complete the gentoo ebuilds and
> > at that point I'll need folks to test and provide feedback. I also use
> > openrc (so it's a non-systemd) cluster effort on my path. But if
> > folks want to use systemd to build a mesos cluster, on gentoo, I think
> > that too is very possible, as gentoo supports both openrc and systemd.
> > Many are flocking to gentoo right now, because we have the best
> alternative
> > to systemd; openrc [1].
> > The clusters I'm interested in building, are optimized for performance::
> > not only in the kernel but direct control over cgroups [2] to opetimze
> > performance with bare-metal granularity over components. It's impossible
> to
> > get that with most distro oriented cluster offerings. This is necessary
> > to optimize a cluster for singularly large/complex tasks, like
> computational
> > chemistry, sub-surface modeling of fluid flows, complex graphics and
> video
> > rendering and analytics, just to name a few.
> > Furthermore, the 'bare metal' approach will all for optimizing all sorts
> > of distributed processing, available now via Clang-3.7.x (openmp)[3] and
> > gcc-5.2 (OpenAcc). [4,5.6]
> > Still, if nixos has all of this going on, the I'd like to test drive
> nixos.
> > Maybe one of the devs can whip together a lived CD 'howto' to see it run?
> > Then we can run a hi performance linpack on it (sys-cluster/hpl in
> gentoo)
> > to benchmark the nixos offering?
> > If you are interested, let's share emails and get a 'cluster club'
> started?
> > Curious?
> > James
> > [1] https://wiki.gentoo.org/wiki/OpenRC
> > [2] https://wiki.gentoo.org/wiki/OpenRC/CGroups
> > [3] http://blog.cafarelli.fr/2015/09/testing-clang-3-7-0-openmp-support/
> > [4] https://gcc.gnu.org/wiki/OpenACC
> > [5] https://gcc.gnu.org/wiki/Offloading
> > [6]
> http://www.phoronix.com/scan.php?page=news_item&px=GCC-5-Offloading-How-To
> >
> > _______________________________________________
> > nix-dev mailing listnix-dev <at>
> lists.science.uu.nlhttp://lists.science.uu.nl/mailman/listinfo/nix-dev
> >
> >
> >
> >
> >
> >
> >
> > _______________________________________________
> > nix-dev mailing list
> > nix-dev <at> lists.science.uu.nl
> > http://lists.science.uu.nl/mailman/listinfo/nix-dev
> >
>
>
> _______________________________________________
> nix-dev mailing list
> nix-dev at lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.science.uu.nl/pipermail/nix-dev/attachments/20150929/4f3c9a31/attachment-0001.html
More information about the nix-dev
mailing list