[Nix-dev] Stripped down Linux distribution based on Nix/NixOS

[Nahum Shalman]

Hello everyone,

I'm hoping to get a little bit of guidance (skip down to "TL;DR" if you're
short on time/patience). I'm working on the Cerana project [1] which
contains at its core a customized Linux distribution (CeranaOS).

Before I joined the project we were using a different build system that I
am working on replacing with Nix/NixOS.

I began by opening #14538 [2] and hacking up the NixOS installer code as
both a proof-of-concept that Nix/NixOS would be a good substrate for Cerana
to build upon, and to provide some quick initial value to the NixOS
community (see [3] and [4]). The result of that initial work culminated in
#14740 [5] getting us one step closer to [3].

I've done some further hacky experiments [6] building an even more stripped
down image that includes some of the work that we're doing in the Cerana
project.

Sidebar -- Cool stuff being worked on in Cerana:
1. Adding a namespace to the Linux kernel that can be used by:
2. A modification to the SPL that can use the new namespace to leverage the
ZFS zone awareness to delegate ZFS datasets into that namespace and thus
into a "container".
3. Golang bindings for the upcoming stable libzfs API
4. Other cool cluster managment software being written in golang

This brings me to my current state and the next phases of work for Cerana:

1. We need to ship a kernel with patches that we will continue working on
(I'm curious to see how #15095 plays out) including the patch for our new
namespace
2. We need to ship a kernel with selinux rather than apparmor (I'm pretty
certain we considered grsecurity and have a good reason to be using
selinux; assume that's a hard requirement for the moment)
3. Ideally we'd like to be shipping a very very stripped down image that
doesn't even include Nix in it and puts the nix store directly into the
initrd instead of inside a squashfs inside it. I was excited to see the
Docker image code can generate such stripped down container images and that
is very much along the lines of what we want to be able to do.
4. We need to be able to set up continuous integration both for building
the latest version of our code, but also to do builds of pull request
branches. Simplifying the ability to patch the kernel based on dynamically
generated new versions of a patch will probably be important.

TL;DR

I think the two most critical areas for us to work on next are:
1. Shipping a kernel that enables selinux rather than apparmor. Any
suggestions about how to do this?
2. Do we continue in the direction of stripping down the NixOS installer
and trimming it down then adding additional packages to turn it into
CeranaOS, or do we start building CeranaOS from the ground up in a
different way?

Also relevant in the short term:
Should we maintain our variants of the Kernel, SPL, and ZFS packages
somehow within the nixpkgs tree making them available to other interested
parties or should we really keep them in a separate tree/branch/fork?

Suggestions, thoughts, impressions, etc. are most welcome.

-Nahum

[1] http://cerana.org/
[2] https://github.com/NixOS/nixpkgs/issues/14538
[3] https://github.com/NixOS/nixpkgs/issues/2100
[4] https://github.com/antonym/netboot.xyz/issues/37
[5] https://github.com/NixOS/nixpkgs/pull/14740
[6] https://github.com/NixOS/nixpkgs/compare/master...nshalman:cerana-test5
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.science.uu.nl/pipermail/nix-dev/attachments/20160503/71892eef/attachment.html