[Nix-dev] Multiple outputs and hierarchical Nix store paths

[Bjørn Forsman]

Hi all,

I've been doing some thinking lately about multiple-outputs in Nix.
I've been involved in a project at work that use OpenEmbedded
(cross-compile embedded Linux), another build system that supports
"multiple-outputs". IMHO, OpenEmbedded has a rather nice approach to
multiple-outputs, something I now miss when working with Nix. But I
might have an idea how to improve it.

First, a bit about how the systems work today.

OpenEmbedded:
Adding a build dependency brings in *all* outputs. (So there is no
need to wonder which output should be used to build.) When the build
is finished, the build system looks at which files are really needed
(runtime deps) and trims down the (runtime) closure as needed, based
on the granularity of the package splits of each dependency.

Pro: Trivial to split packages in many sub-packages. There is only one
dependency to bring in, and runtime closure is automatically trimmed
as much as possible. For instance, gstreamer plugins each have their
own sub-package.
Con: Build time closure is bigger than needed. For instance, man pages
are not needed for a build.

In Nix:
I'm actually unsure about the details here, but I *think* the standard
builder simply takes the first output of each package in buildInputs.
The runtime closure is then either includes the build input or it does
not. (There is no sub-output garbage collection.)

Pro: Small build time closure.
Con: Developers must be careful how they split packages. Difficult to
create finely split packages (like separate package output for each
gstreamer plugin). (Or did I miss something?)

What I'd like is basically to bring the OpenEmbedded behaviour to Nix.
Does that make sense?

First I tried to figure out a way to union Nix outputs into one "all"
output. The idea is that we need more finely split packages, but that
isn't really feasible unless we can easily bundle them together first,
for builds. But I couldn't come up with anything that wouldn't end up
referencing the "all" output, which would prevent garbage collection.

Which brings me to Nix itself. Nix store paths are "flat", not
hierarchical. There is currently no way for a store path to "contain"
another store path. But can we change that?

$ nix-build -A pkgconfig
/nix/store/wndacaca5dq5cpyfa41v1fv4sfhr0y77-pkg-config-0.29
$ nix-build -A pkgconfig.share
/nix/store/wndacaca5dq5cpyfa41v1fv4sfhr0y77-pkg-config-0.29/share
$ nix-build -A pkgconfig.share.doc
/nix/store/wndacaca5dq5cpyfa41v1fv4sfhr0y77-pkg-config-0.29/share/doc

(Note that all outputs have the same store path prefix.)

I really have no idea how difficult it is to achieve this, if at all
possible. But I thought I'd throw the idea out there :-)

As for specifying the outputs, we could use something like this:

outputs = {
  out = "/";
  share = "/share";
  libfoo = "/lib/libfoo.*\.so.*";
}

A base "outputs" set can be defined in the standard builder, to
auto-split the common first-level directories (/bin, /include, ...).
Packages can extend/override the base as they please.

Now that I think of it, this can also be used to check a build
produces the expected files. With the following outputs specification,
the build would fail unless /bin/vim is produced:

outputs = {
  ...
  bin.vim = "/bin/vim";
}

Thoughts?

Best regards,
Bjørn Forsman