[Nix-dev] Please provide input on my attempts to rework how NixOS builds images
Daniel Peebles
pumpkingod at gmail.com
Thu Apr 13 18:22:40 CEST 2017
Hi all,
Eelco and I are having a slight disagreement on how to proceed on a PR of
mine and we figured we'd ask the community for opinions. If you've dealt
with images in Nix and/or virtual machines, please continue reading.
*tl;dr:* please give thumbs up or down on
https://github.com/NixOS/nixpkgs/pull/23026
Here's a quick summary of what's going on:
- NixOS/nixpkgs has a remarkable amount of image building code spread
throughout it, for building EC2/Google/Azure images, cd/dvd images,
initramfs, and probably several others.
- That code is generally copied and pasted a lot, but almost all of the
image builders currently run the filesystem build inside a Linux VM; the
usual process is:
1. start Linux VM (because writing filesystems requires root and is
mostly "impure")
2. set up partitions & filesystems
3. run nixos-install
4. shut down VM
5. save image to $out
- Starting a Linux VM inside a Nix build isn't always possible or
efficient. For one example, 95% of "the cloud" doesn't support nested
virtualization, so running an image build on EC2 is fully emulated and
takes a decimal order of magnitude or more longer than it does on physical
hardware, due to the lack of hardware-assisted virtualization. There's no
indication that any cloud providers will be be providing nested
virtualization support anytime soon. A second example is that nested
virtualization support in e.g., VMWare can be pretty flaky on Linux guests:
I used to run nested virtualization fine inside NixOS but some recent Linux
kernel changes now hang my system when I try it, and there doesn't seem to
be any indication that there's active work on remedying such flakiness.
- I build NixOS images *a lot* (mostly for EC2, on EC2) and don't have
access to any physical Linux hardware.
- I have devised a cute process
<https://github.com/NixOS/nixpkgs/pull/21943> (using the Linux Kernel
Library project) that can quickly and efficiently craft Linux filesystems
using the upstream Linux kernel libraries directly, without a VM.
Here's what I'm proposing:
1. Make the image builder code throughout NixOS (I'll also refactor and
share more code between the image generators over the next few weeks) use
LKL and do far less work in a VM, in order to build images efficiently on
EC2 (image build time goes from 30+ minutes to <1 minute) and to build them
at all on local virtualization. This means *the image builder code no
longer calls nixos-install*.
2. To avoid duplicating all the logic in nixos-install, factor out the
common elements in the VM-free image builder and nixos-install into a third
script, which I've been tentatively calling nixos-prepare-root, and have
nixos-install and image builders both call that.
Eelco is understandably concerned that by forking the image build and
making it take a slightly different path from nixos-install (even if they
share most of the core filesystem build logic), we'll make maintenance
harder and make it easier to introduce subtle bugs. Although I agree with
the gist of that and am striving to minimize divergence from nixos-install,
I still need it to work on EC2 and VMWare.
Eelco has said he won't veto the PR merge (and my subsequent work) if the
community supports it, but doesn't particularly like it himself. That's
where you come in :) please check out my PR and let us know what you think.
I'd obviously love to merge this, but if the community really doesn't want
this change I can maintain some forked logic here for my needs.
Thanks,
Dan
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mailman.science.uu.nl/pipermail/nix-dev/attachments/20170413/98f64a8a/attachment.html>
More information about the nix-dev
mailing list