[Nix-dev] Fixing module arguments (PROPOSAL MAY BREAK EXISTING NIXOS CONFIGS, PLEASE READ)

[Jan Malakhovski]

Hi.

Oh, I so understand you. I stumbled upon the same problem several days
ago while coming from the different angle.

I was trying to support building several unrelated systems with
different configurations at once. For instance, I want to be able to

* generate a live CD with a _set_ of configurations squashfsed
  together with a common grub loader
* generate a set of configurations that are to be booted through PXE.

This all is more than "somewhat untrivial" with current design of
NixOS. Let me mention some more problems that're bugging me here.

I want to be able to differentiate between NixOS outputs. Currently
"system.build.toplevel" is more or less the only output NixOS produces
directly (others are generated from release.nix and look very much
like hacks). And, frankly, it's a very strange one: a system that,
when activated, discovers other systems and generates bootloader
config for all of them.

I want a way to generate (I meas, by construction, "well-typedly", so that
configuration.nix has no meaningful way to get around this)

* a system without the control over the boot loader (think liveCD with
  a common loader and a set of systems on it, or a PXE-booted system)
* a system without the control over all the stuff that happens before
 activation, e.g. kernel version.

I actually think that merging modules/ with configuration.nix should
be one of the many ways NixOS can produce systems.

As to eval-config.nix I tried to a implement a different way to
evaluate stuff, a draft could be seen here (ignore the first patch
about closures, I cherry-picked it accidentally):

https://github.com/oxij/nixpkgs/tree/show-some-work

(it's far from being usable, look for the main idea)

The idea there is that one describes a configuration of a set of
configurations and the old eval-config is just a special case.

As a next step I was thinking about splitting NixOS into parts
parametrized by each other. Kind of this stuff in pseudo-code:

currentNixOS = let
    conf = evalSingleSystemConfig (merge [configuration.nix somewhatLikeCurrentNixOSConfigs])
  in collaborativeLoaderManagement (conf.boot.loader) (withKernel conf.kernel conf.system))

liveCD configs = let
    confs = evalASetOfConfigs configs
  in packIntoISO (grubConfigFor (map (s: (s.kernel, s.initrd, s.options) confs)))
                 (concatMap (s: s.system) confs) -- not though through well yet

pxeBootable boothost = let
    confs = evalASetOfConfigs (map (c: merge [c { fileSystems."/nix/store".device = "${boothost}:/nix/store" }]) configs)
  in generateClosureDirectory (staticiPXEConfigFor (map (s: (s.kernel, s.initrd, s.options) confs))))
                              (concatMap (s: s.system) confs) -- not though through well yet

ebookReader = let
    conf = evalSingleSystemConfig (merge [configuration.nix restrictedVersionOfCurrentNixOSWithout"boot.kernel"andRelated])
  in generateClosureDirectory (evalSingleSystemConfig [configuration.nix])

I hope you get the idea.

Cheers,
  Jan

On Wed, 19 Feb 2014 10:44:46 -0500
Shea Levy <shea at shealevy.com> wrote:

> Hi all,
> 
> The Problem
> -----------
> 
> I've been doing a bit of work to make it possible to define nixops
> networks modularly in the same way we do NixOS configs (with each
> machine in a network being a submodule), and in doing so I've repeatedly
> run into issues due to the fact that NixOS's modules take a bunch of
> arguments beyond config, options, and lib, in particluar that they take
> 'pkgs'. The problem with these extra arguments in general is that they
> are superfluous and non-modular: superfluous, because any internal
> values we want to make accessible to all modules can be put into config;
> non-modular, because the set of arguments has to be specified from one
> central place and (without special hacks like we have for pkgs) modules
> can't affect the value of the arguments. The additional problem with
> pkgs in particular is that it is used to *define* nearly every module
> (as originally lib wasn't passed, so modules take functions like
> mkOption or mkIf from pkgs.lib instead), but its value also *depends* on
> the configuration (via config.nixpkgs.config and config.nixpkgs.system).
> 
> The current practical effects of this issue are:
> 
> * The set of arguments to NixOS modules is maintained in one central
>   place (<nixpkgs/nixos/lib/eval-config.nix>) and any configurability we
>   may want to have for them must modify that file as well as the module
>   that defines the option.
> * The configuration must be partially evaluated with a default value for
>   pkgs and then *re-evaluated* with the final value for pkgs. Due to how
>   this partial evaluation works (only configuration.nix, its imports,
>   and nixpkgs.nix are included in the evaluation), no modules in
>   module-list.nix except nixpkgs.nix can affect pkgs.
> * Inelegance. Opinions will vary here of course and it's hard to
>   quantify the practical costs, but lacking a uniform interface, having
>   a whole bunch of arguments (some of which I've never used in years of
>   NixOS, like 'utils'), and having two ways to make data available to
>   all modules (one of which is vastly inferior to the other) all adds up
>   to cruft.
> 
> The additional practical effects this has on my implementation of
> modular nixops networks are:
> 
> * I need a new type "submoduleWithExtraArgs", as the submodule type
>   doesn't pass any extra arguments besides "name" to the submodules and
>   that can't work with using NixOS configurations as submodules.
> * I need a new type "dependentAttrsOf", where the type of the attr value
>   can depend on the attr name, as different machines in one network
>   might have different settings for nixpkgs.config and thus should be
>   passed different values of pkgs, but as per point 1 the arguments
>   passed to the submodule have to be part of the type
> * I have to duplicate NixOS's two-phase eval for *each* machine in the
>   network
> 
> The Easy Solution
> -----------------
> 
> Accept the situation as it is, and that these arguments are part of how
> modules work and aren't going away.
> 
> Pros: No end-user changes needed.
> Cons: See above (most of which is shouldered by developers touching core
> parts of the system)
> 
> The Conservative Solution
> -------------------------
> 
> Promote a different style for modules, where they take at most config,
> lib, and options and extract any other needed values from config. Slowly
> migrate NixOS itself to this style, and eventually print a nix trace
> whenever any of the other arguments are accessed, some day maybe turning
> into a throw or finally just removing them. New modules should never use
> the arguments and new uses of the module system should not have any.
> 
> Pros: Minimal change to end-user configs, with ample warning
> Cons: The arguments must be maintained for quite some time, and must be
> handled by all tools interacting with the NixOS module system.
> 
> The Moderate Solution
> ---------------------
> 
> Read arguments from some special config value, like
> config._extraArguments. Eventually phase this out much like the
> conservative solution.
> 
> Pros: Minimal change to end-user configs, with ample warning for most
> such changes. Arguments can be handled modularly, and the
> double-evaluation can be avoided. Since they are handled in the modules
> themselves, only the core module system needs to bother itself with them
> and consumers like <nixpkgs/nixos/lib/eval-config.nix> or
> <nixops/eval-machine-info.nix> and types.submodule can act as if
> arguments don't exist.
> Cons: The module system must be able to fully evaluate
> config._extraArguments.* from every module, which also implies needing to
> evaluate config.nixpkgs, without needing to evaluate any of the
> arguments. This means that any module which uses config = mkIf { ... }
> where mkIf comes from pkgs.lib instead of lib will cause an infinite
> recursion. I'm willing to fix this in nixpkgs, and most end-user modules
> won't have to worry about this, but if you have externally-maintained
> modules that use mkIf you will have to change your module to get it from
> lib instead of pkgs.lib. Also, the core module system will need code to
> handle the arguments.
> 
> The Radical Solution
> --------------------
> 
> Eliminate all extra arguments.
> 
> Pros: A unified module interface, with no prolonged period of
> maintaining the cruft.
> Cons: Nearly every existing module will have to change. I'm willing to
> do the ones in nixpkgs, but unless your configuration doesn't take any
> arguments or only takes 'config' then evaluation will fail.
> 
> 
> Because these pros and cons affect everyone differently, I'd like to put
> these ideas out for discussion. Compatability breaks are scary and
> should not be done without reason, and my threshold for what is a good
> reason is probably different from yours. I lean toward the latter two
> solutions, and think it makes sense to use the new NixOS release cycle
> to batch up breaking changes like this one that on their own might not
> warrant the break but together do.
> 
> Cheers,
> Shea
> _______________________________________________
> nix-dev mailing list
> nix-dev at lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev