[Nix-dev] NixOS inside a 64-bit Xen Guest (x86_64)

[Jim Pick]

Nicolas Pierron wrote:

>>  http://github.com/jpick/nixos-xen64-nixos/tree/master
> 
> I have some remarks on your commits:

Thanks for looking at them!

> commit  317a2b236926e888cb81932a6528c45c8fb1bdc8:
> 
> You should not modify system/options.nix, instead you should put
> inside your /etc/nixos/configuration.nix file the following lines:
> 
> {
>   boot = {
>     kernelParams = [ "selinux=0" ];
>   };
> }
> 
> This configuration defines the default, which is currently the default
> use case in order to avoid pasting these lines into each user
> configuration. (even me)

Thanks for the info.  As with everything, that was a quick brute force 
hack - I was on a 'search-and-destroy' mission with 'grep -r' to get rid 
of the stuff which was preventing me from booting.  At this point, I 
haven't learned enough syntax to understand how the whole 
substitution/override mechanism works, so the easiest way for me to fix 
it was to hack it.

As a new user, it would be nice if there was some better documentation 
for configuration.nix.  It was a real struggle for me to try to figure 
out even the simplest things I needed to put in there.

I imagine that lack of configuration documentation is the #1 issue NixOS 
has for adoption - most people aren't going to be able to figure out how 
to do the simplest things (like assign a static IP address to an 
interface), because there is no documentation.

> commit  5aa4ab9f75fa002a58eefa672f02dd792c5277e6:
> 
> The root you have specified is hard coded and this may not be the same
> configuration for everybody.

Yes, I understand that.  I'm using pvgrub in Xen - it needed the extra 
"root (hd0,0)" for some reason.  I'm guessing that normal grub can talk 
to the BIOS to get that info, but pvgrub obviously can't, so I needed to 
be explicit.

> commit  95aa5ac0d2d54477f0ff0146cd26c1e582197992:
> commit  2109cb0c03ce8f050e1d10b80b85c9ae4b40c263:
> 
> Ok, your problem here (I am guessing) is that you have a bad name for
> you tty.  Wouldn't it be easier to just create a static link to the
> tty* at the boot stage ?
> 
> for i in /dev/tty*; do
>   ln $i ${i/tty/hvc};
> done

When I was running my older Xen kernel, I was able to use /dev/tty1 for 
my serial-style console.  At some point in the 3.x Xen series, they 
introduced "virtual framebuffer" support into the kernel.  I needed the 
old style consoles, that work with "xm console" admin tool found in Xen 
3.0.2.  That's what the new 'hvc' console devices are for.  I think the 
/dev/tty* devices still work with the virtual framebuffer devices if 
somebody has those configured and is using a newer admin tool with 
graphical support (eg. over VNC) to connect.

I'm not sure what the best way is to wire that in - I imagine that 
people might want to use one, or the other, or both.

> commit  42ce4cb8de8632afc810f9d0dfcca15535ee5706:
> 
> As well as the first comment, you should not modify system/option.nix,
> instead you should put inside your /etc/nixos/configuration.nix file
> the following lines:
> 
> {
>   services = {
>     mingetty = {
>       ttys = [0];
>     };
>   };
> }

I think it ought to work 'out of the box'.  Perhaps the startup scripts 
should detect that they are running on Xen, and in that case, start up 
an additional mingetty on /dev/hvc0 if the device exists.

>>  http://github.com/jpick/nixos-xen64-nixpkgs/tree/master
> 
> The kernel configuration is made to be portable, so as you have
> remarked, this is not a good choice to have a minimal kernel
> configuration.  There is an option which can be defined which is
> called "kernelPackages" and which takes your kernel installation in
> which you can specify your system configuration.

I initially tried to just add the additional Xen kernel configuration on 
top of the generic kernel.  However, I couldn't get it to work - the 
kernel would die with a "Machine Check Exception" very early in the boot 
process.  I suspect it might be because I am running on a very old 
version of Xen in my Dom0 (3.0.2 from Debian Stable).

As I had to iterate the kernel build many times to get a kernel 
configuration that would work, and as Xen only uses a few drivers, it 
saved me a lot of time by stripping down the kernel configuration.  The 
"portable" kernel configuration took several hours to build on my 
hardware.  With my minimal config, I could build and test a new kernel 
in about 15 minutes.

As somebody who has building kernels since Linux 1.2, I'm absolutely 
horrified at how long it takes these days...

My interest isn't in building a generic Linux distribution with NixOS - 
so I probably will never run a generic kernel.  Maybe somebody else who 
is interested in doing that can figure out the proper kernel config, and 
test it.  I estimate that might take many, many hours of work, and would 
need to be maintained over time.

>> It would be nice to get the modifications cleanly merged upstream.  The
>> hacks I currently have are more of the 'brute force' variety.
> 
> I think you should try to do the same configuration of NixOS without
> making any modifications inside nixos and nixpkgs repositories (except
> for the device name modification).  The other modifications you have
> made are good for your system but may not be good for others.
> 
> Feel free to ask if you have any questions.

My plans for Nix and NixOS mean I'm going to deviate substantially from 
what is found in the standard nixos and nixpkgs repositories -- almost 
immediately.  But, of course, I want to be able to share my changes back.

One of the first things I'm going to do is add a layer of abstraction on 
top.  I won't be using the Nix package expressions unmodified.  I'm 
going to parameterize the Nix expressions, and start factoring out 
'metadata'.  Practically speaking, that means that the software version 
numbers will be factored out (along with other things).  I'm going to 
use a templating framework to regenerate the Nix expressions, with a 
user-friendly web-based front-end.

Here's a sneak peek of where I'm going to collect the metadata:

   http://codeflow.freebase.com/

I also want to work on using Nix as part of a cross-compilation system. 
    Embedded systems are one of my targets.  I see there used to be 
support for that, but it's been removed.  I used to maintain a RPM-based 
distribution with 300+ packages that was fully cross-compiled, so I have 
some idea of the work involved.

I'm also want to investigate some tools for declaratively defining and 
building distributed systems.  Puppet + NixOS on EC2, anyone?

Cheers,

  - Jim