[Nix-dev] Questions about the all-packages fixpoint

[Nicolas Pierron]

Hi Benno,

Before answering, let me give you a brief intro to the fix-extend
function combo.

The extend function is similar to the update operator of Nix `//`, the
main difference is that it give its left-hand side (`super`) as
argument to its right hand side.
But this extend function give an extra argument (`self`) to all its
arguments. When chaining these extend function, the `self` argument is
the same for the all layers.
When a fix point is applied to this chain of extend function, `self`
corresponds to the last set produced by the extend function chain.

Note that using `self` can easily cause infinite recursion, for
example if the addition of an attribute to the set depends on the
presence of an attribute in `self`.

On Fri, Mar 17, 2017 at 5:04 PM, Benno Fünfstück
<benno.fuenfstueck at gmail.com> wrote:
> 2. In `stage.nix`, why do we construct a *local* fixpoint just for
> all-packages? The current code is:
>
>     allPackages = self: super:
>       let res = import ./all-packages.nix
>         { inherit lib nixpkgsFun noSysDirs config; }
>         res self;
>       in res;
>
> Even more confusingly, if we look at `all-packages.nix`, it is a function
> defined like this:
>
>     { lib, nixpkgsFun, noSysDirs, config}:
>     self: pkgs:
>
> So the variable that is called `self` in `stage.nix` is bound to what is
> called `pkgs` in `all-packages.nix`, and `self` in `all-packages.nix`
> actually refers to what is called `res` in `stage.nix`!!!!

This is because this code is still legacy before the introduction of
the extend function.
We should not use this `self` argument under all-packages.nix, but
last time I tried to avoid changing any hashes.

Another reason why I did not address this, such as renaming it because
`self` is quite an overloaded name, and a naive search and replace
would not work as expected.
Note, there is also a package named `self`, for interpreting the Self language.

So `pkgs` of all-packages.nix is literaly the result of the fix-point,
given to all overlays under the name `self`.

> 1. An override is given the arguments `self` and `super`, as expected. But
> why does `super.callPackage` resolve dependencies from `self`?

`callPackage` take a function as argument, or a path to a function,
and list the arguments of it.  It then create a set with the requested
argument and a set given as argument.
In `top-level/splice.nix`, the callPackage function is given an
argument which is constructed from the result of the fix-point.
Thus `super.callPackage` will fill the arguments of the functions with
the value taken within the result of the fix-point.

> I would have
> expected it to not know about `self`, and resolve dependencies in the scope
> of `super`. If I wanted this behaviour, I would have used
> `self.callPackage`, but the current behaviour makes that unnecessary. Why is
> this implemented like that?

The history of all-packages.nix used to be that we had a "rec" keyword
on top of the attribute set.
Then we added the `self` reference that you complained about in (2) in
order to be able to override packages.

If `super.callPackage` were taking all its arguments from `super`,
then this would imply that you could not easily override the argument
of packages which are inside the dependencies of another one.
By taking the packages from `self`, we can easily replace it for all its uses.

> This is confusing to me. Also, why do we need `self` and `pkgs`? Wouldn't
> one of them be enough?

Yes, but again, renaming these variables is hard without the proper
tooling to guarantee that we are not making any mistake.


-- 
Nicolas Pierron
http://www.linkedin.com/in/nicolasbpierron - http://nbp.name/