[Nix-dev] Fixing module arguments (PROPOSAL MAY BREAK EXISTING NIXOS CONFIGS, PLEASE READ)

[Shea Levy]

Hi all,

The Problem
-----------

I've been doing a bit of work to make it possible to define nixops
networks modularly in the same way we do NixOS configs (with each
machine in a network being a submodule), and in doing so I've repeatedly
run into issues due to the fact that NixOS's modules take a bunch of
arguments beyond config, options, and lib, in particluar that they take
'pkgs'. The problem with these extra arguments in general is that they
are superfluous and non-modular: superfluous, because any internal
values we want to make accessible to all modules can be put into config;
non-modular, because the set of arguments has to be specified from one
central place and (without special hacks like we have for pkgs) modules
can't affect the value of the arguments. The additional problem with
pkgs in particular is that it is used to *define* nearly every module
(as originally lib wasn't passed, so modules take functions like
mkOption or mkIf from pkgs.lib instead), but its value also *depends* on
the configuration (via config.nixpkgs.config and config.nixpkgs.system).

The current practical effects of this issue are:

* The set of arguments to NixOS modules is maintained in one central
  place (<nixpkgs/nixos/lib/eval-config.nix>) and any configurability we
  may want to have for them must modify that file as well as the module
  that defines the option.
* The configuration must be partially evaluated with a default value for
  pkgs and then *re-evaluated* with the final value for pkgs. Due to how
  this partial evaluation works (only configuration.nix, its imports,
  and nixpkgs.nix are included in the evaluation), no modules in
  module-list.nix except nixpkgs.nix can affect pkgs.
* Inelegance. Opinions will vary here of course and it's hard to
  quantify the practical costs, but lacking a uniform interface, having
  a whole bunch of arguments (some of which I've never used in years of
  NixOS, like 'utils'), and having two ways to make data available to
  all modules (one of which is vastly inferior to the other) all adds up
  to cruft.

The additional practical effects this has on my implementation of
modular nixops networks are:

* I need a new type "submoduleWithExtraArgs", as the submodule type
  doesn't pass any extra arguments besides "name" to the submodules and
  that can't work with using NixOS configurations as submodules.
* I need a new type "dependentAttrsOf", where the type of the attr value
  can depend on the attr name, as different machines in one network
  might have different settings for nixpkgs.config and thus should be
  passed different values of pkgs, but as per point 1 the arguments
  passed to the submodule have to be part of the type
* I have to duplicate NixOS's two-phase eval for *each* machine in the
  network

The Easy Solution
-----------------

Accept the situation as it is, and that these arguments are part of how
modules work and aren't going away.

Pros: No end-user changes needed.
Cons: See above (most of which is shouldered by developers touching core
parts of the system)

The Conservative Solution
-------------------------

Promote a different style for modules, where they take at most config,
lib, and options and extract any other needed values from config. Slowly
migrate NixOS itself to this style, and eventually print a nix trace
whenever any of the other arguments are accessed, some day maybe turning
into a throw or finally just removing them. New modules should never use
the arguments and new uses of the module system should not have any.

Pros: Minimal change to end-user configs, with ample warning
Cons: The arguments must be maintained for quite some time, and must be
handled by all tools interacting with the NixOS module system.

The Moderate Solution
---------------------

Read arguments from some special config value, like
config._extraArguments. Eventually phase this out much like the
conservative solution.

Pros: Minimal change to end-user configs, with ample warning for most
such changes. Arguments can be handled modularly, and the
double-evaluation can be avoided. Since they are handled in the modules
themselves, only the core module system needs to bother itself with them
and consumers like <nixpkgs/nixos/lib/eval-config.nix> or
<nixops/eval-machine-info.nix> and types.submodule can act as if
arguments don't exist.
Cons: The module system must be able to fully evaluate
config._extraArguments.* from every module, which also implies needing to
evaluate config.nixpkgs, without needing to evaluate any of the
arguments. This means that any module which uses config = mkIf { ... }
where mkIf comes from pkgs.lib instead of lib will cause an infinite
recursion. I'm willing to fix this in nixpkgs, and most end-user modules
won't have to worry about this, but if you have externally-maintained
modules that use mkIf you will have to change your module to get it from
lib instead of pkgs.lib. Also, the core module system will need code to
handle the arguments.

The Radical Solution
--------------------

Eliminate all extra arguments.

Pros: A unified module interface, with no prolonged period of
maintaining the cruft.
Cons: Nearly every existing module will have to change. I'm willing to
do the ones in nixpkgs, but unless your configuration doesn't take any
arguments or only takes 'config' then evaluation will fail.


Because these pros and cons affect everyone differently, I'd like to put
these ideas out for discussion. Compatability breaks are scary and
should not be done without reason, and my threshold for what is a good
reason is probably different from yours. I lean toward the latter two
solutions, and think it makes sense to use the new NixOS release cycle
to batch up breaking changes like this one that on their own might not
warrant the break but together do.

Cheers,
Shea