[Nix-dev] Introducing Nixpkgs Overlays

[Nicolas Pierron]

Hello Nixpkgs users,

One of the issues of Nixpkgs is the lack of modularity.  By
modularity, I mean that someone else can provide a complementary
source of packages which works side-by-side with Nixpkgs, and that as
a user, you can combine multiple of these sources.

# Extending Nixpkgs Today

Today, we have a lot of way to extend Nixpkgs:

1. We can import nixpkgs, and add packages on top of it.

```nix
let pkgs = import <nixpkgs> {}; in

pkgs // rec {
  foo = pkgs.foo.override { enableBar = true; };
  bar = import ./pkgs/bar { inherit (pkgs) stdenv fetchurl foo; };
}
```

2. We have `packageOverrides` from ~/.nixpkgs/config.nix.

```nix
{
  packageOverrides = pkgs: rec {
    foo = pkgs.foo.override { enableBar = true; };
    bar = import ./pkgs/bar { inherit (pkgs) stdenv fetchurl foo; };
  };
}
```

3. We have `pkgs.overridePackages`.

```nix
let pkgs = import <nixpkgs> {}; in
  pkgs.overridePackages (self: super: {
    foo = super.foo.override { enableBar = true; };
    bar = import ./pkgs/bar { inherit (self) stdenv fetchurl foo; };
  })
```

But none of these approaches can easily be composed with other sources
doing similar things.

# Extending Nixpkgs with Overlays

Fortunately, the recent improvements of Nixpkgs inner structure
[2,3,4,5], highlights that we have a list of layers which are stacked
one on top of the others.

These layers are literally what composes Nixpkgs, we could technically
rewrite the entire Nixpkgs collection into thousands of these layers,
where each one handles a single package at a time, but this would be
inefficient.

On the other side, we could expose this layering mechanism to external
sources of Nixpkgs, and gain the modularity that Nixpkgs never had.
This is what I did with the `overlays` pull-request [1].

An overlay is literally an additional layer added on top of Nixpkgs,
within the fix-point of Nixpkgs.
The same principles was already used by `pkgs.overridePackages`
without adding the composition ability of overlays.

An overlay would look something like [6]:

```nix
self: super:

{
  foo = super.foo.override { enableBar = true; };
  bar = import ./pkgs/bar { inherit (self) stdenv fetchurl foo; };
}
```

Which is essentially the minimal part of all the previous examples,
and more over the exact same way we represents the different layers
within Nixpkgs.

The modularity aspect of overlays comes from the fact that overlays
are listed, either in the directory, or as argument of nixpkgs.  By
default the ~.nixpkgs/overlays/ directory is considered unless
NIXPKGS_OVERLAYS specify otherwise.

This directory should only contain Nix expression files (or directory
with a default.nix) which are overlays, with the same layout as the
example above.  One can also use a symbolic link to where each of
these overlays are installed on your file system.

The directory is converted in a list of elements, which are ordered
based on the alphabetical order of the elements within the directory.

[1] https://github.com/NixOS/nixpkgs/pull/21243
[2] https://github.com/NixOS/nixpkgs/pull/9400
[3] https://github.com/NixOS/nixpkgs/pull/14000
[4] https://github.com/NixOS/nixpkgs/pull/15043
[5] With the help of Peter Simons' extend function.
[6] https://github.com/nbp/nixpkgs-mozilla/blob/nixpkgs-overlay/moz-overlay.nix

# Planning for the future.

As part of this modification, I intend to remove the
`pkgs.overridePackages` function, as this one can literally be
replaced by the following Nix expression:

```nix
let pkgs = import <nixpkgs> {}; in
  import pkgs.path { overlay = [ (self: super: {
    foo = super.foo.override { enableBar = true; };
    bar = import ./pkgs/bar { inherit (self) stdenv fetchurl foo; };
  }) ]; }
```

One additional reason to remove `pkgs.overridePackages` function, is
that the creation of this function also stands as the function which
adds Nixpkgs fix-point.  The `pkgs.overridePackages` function does not
play nicely with the `security-updates` branch, as for the applying
patches to packages, we have peels the fix-point out, as well as this
function.  Thus, there would be no way to benefit from the security
updates while using `pkgs.overridePackages`, without making it much
more complex.

On the other hand, the overlays system play nicely with the
`security-updates` branch, as it add layers within the existing
fix-point, and thus peeling the fix-point does not change the
overlays.

# If you are not yet convinced

A while ago, I suggested to add modularity to NixOS [7], and I
personally consider it as a great success for one reason.  This reason
is allow any user to extend NixOS without changing syntax, and without
changing NixOS sources.

Adding overlays would add the same benefits to Nixpkgs, such as giving
the opportunities to multiple companies to distribute software [6
(above)] without providing a full clone of Nixpkgs them-self.

[7] http://lists.science.uu.nl/pipermail/nix-dev/2008-November/001467.html

-- 
Nicolas Pierron
http://www.linkedin.com/in/nicolasbpierron - http://nbp.name/