[Nix-dev] The nixpkgs.config vs overrides (was: Documenting nixpkgs.config options)

[Jan Malakhovski]

Hi.

The problem is that people will say than "config is deprecated, use
overrides". I had a pull-request with a set of patches that allows one
to write

    nixpkgs.config = {
      allowUnfree = unfree;
      common.x11Support = graphical; # note this
      pulseaudio = false; # should be renamed to common.pulseAudioSupport
                          # but I didn't get around to rename it yet

      conky = {
        wirelessSupport = true;
        luaSupport = true;
      };

      mplayer = {
        xineramaSupport = graphical;
        xvSupport = graphical;
        screenSaverSupport = graphical;
        useUnfreeCodecs = enslaved;
      };

      sylpheed.gpgSupport = true;
    }

in their configuration.nix (with that `common` thing setting
`x11Support` to all the packages that have that option), but it was cut
down with "do not add new config options, they are deprecated, we have
overrides already".

Personally I have some problems with current practices:

* I don't like the idea of using `null`s in package arguments of a
  package expression as a way to specify features. I think this is ugly
  and error prone.
* Currently obsessive use of overrides generates more versions of
  packages than needed. Do

    egrep -A 3 -B 6 '   .*override' all-packages.nix

  every override of a package in an argument set for another package
  will likely to ask nix to compile another version. This is crazy. And
  I see that craziness in action every time I rebuild my configuration,
  'cause I use a shitton of packages in `environment.systemPackages` on
  my desktop machines. I haven't measured the CPU time and disk space
  losses to this craziness, but I don't like the fact that these losses
  exist at all.

  I think packages should assert that some options are set in their
  dependencies (e.g. the original xbmc[1] expression that used external
  ffmpeg checked that ffmpeg it was supplied was compiled with the
  required options using `passthru`; current `kodi` just builds another
  ffmpeg; it might have other valid reasons to do that, but still). And
  only then add another version of the package with those packages
  forcefully made to comply to those assertions. I.e. instead of the
  currently popular

    dependency = callPackage ({..., whateverSupport, }: ...) ...;

    somepackage = callPackage ({..., thisWhateverSupport ? true, dependency, ...}:
      assert thisWhateverSupport -> dependency.whateverSupport;
      mkDerivation { ... })
    {
      dependency = dependency.override { whateverSupport = true; };
    };

  that generates another version of dependency one would write

    dependency = callPackage ... {};

    somepackage = callPackage ({..., dependency, ...}:
      assert thisWhateverSupport -> dependency.whateverSupport;
      mkDerivation { ... }) {};

    somepackage_forced = somepackage.override {
      dependency = if config.somepackage.thisWhateverSupport
                   then dependency.override { whateverSupport = true; }
                   else dependency;
    };
 
  which doesn't. When one simply does

    nixpkgs.config.dependency.whateverSupport = true;
  
  or maybe even

    nixpkgs.config.common.whateverSupport = true;

* Overrides are not fixed points and this is a serious problem for some.
  See https://github.com/NixOS/nixpkgs/pull/7102.
* For some strange reason it seems that overrides sometimes evaluate
  stuff too early. I haven't been able debug this, but overriding all
  the X11, GTK and QT packages (exactly those packages, not the packages
  that use those packages) to `null`s in nixpkgs and then specifically
  overriding some of the packages that use them shows that some of the
  latter packages will try to evaluate X11, GTK or QT even when
  explicitly overridden not to so. I guess that's because something in
  the derivation get's stuck on the previous version of the argument,
  but, as I said, I haven't been able to debug this yet.

And so now I maintain my own set of patches I rebase on top of master
once and then, that goes against the grain and simplifies all that
`nixpkgs.config`-style configuration stuff fairly significantly by
adding another version of callConfPackages (also removing a ton of
copy-paste from `all-packages.nix` as a side effect)

https://github.com/NixOS/nixpkgs/compare/master...oxij:call-conf-package

and weep every time someone rewrites and expression for a package
dropping the support for `<feature>Support` options (e.g. most recently
`qemu` =/).

Said that, I agree that there are problems with this:

* All that `somepackage_forced` is somewhat ugly. I would prefer to have
  a way to describe and introspect assertions in the packages and
  generate those *_forced thingies automatically. Or maybe even write
  something like 

    environment.systemPackages = [ somepackage.force_dependencies ];
  
* Package options need documentation (which the original message by
  Herwig Hochleitner is about). This is totally doable even now with
  reusing NixOS modules in nixpkgs:

    { dependency1, dependency2, ... }:
    { config, ...} :
    { options = { supportWhatever = mkOption ...; };
      assertions = [ ... ];
      config = mkDerivation { ... };
    };

  and then fixing `callPackage` to grab only the `config` and move the
  `options` to `meta`. And then collecting the latter by grabbing all
  the stuff from the packages `nix-env -q` uses (one could even show the
  options in nix-env this way).
  
  But I agree that this is somewhat ugly. Especially because usually
  all `supportWhatever` does is adds a flag to `configureFlags` and
  another dependency to `buildInputs`.

  The proper solution is to implement a dual of `bulitins.functionArgs`
  that _generates_ a function with arguments and their defaults set from
  an attrset. And then implement a dual of `callPackageWith` (which I
  call `generatePackageWith` below) that generates package expressions
  with all the options, flags and requirements from expressions like:

    generatePackageWith (args: {
      x11 = { addToArgs = [ "libX11" ];
              configureFlags = [ "--with-x11" ];
              buildInputs = [ args.libX11; ];
            };
    }) { mkDerivation, other stuff, ...}:
    mkDerivation { ... };

  (with `generatePackageWith` simply overriding `mkDerivation` inside.)

  The type of a configuration stub shown above could then be generalized
  to something like

    cfbiFlag = name: f: { addToArgs = [ name ];
                          configureFlags = [ "--with-${f}" ]; 
                          buildInputs = [ args.${name} ];
                        };

  and put into lib.

All in all, all of this (and even more, but this message is long enough
already) is totally doable, but the generic lack of free time and a
continuous explicit and implicit discouraging of this by "we have
overrides, nixpkgs.config is deprecated, do not add new options there"
and the perverse use of those overrides

  (that, by the way, makes me either revert my changes a bit by bit or
  rewrite the whole expressions (which I actually did a couple of times
  for `samba` because I (and, I suppose, other sane people) DO NOT WANT
  `ceph` by default on the desktop!, but after another rewrite of that
  expression in upstream I dropped my changes altogether, simply
  overrode the stub in all-packages.nix and walked away)

holds most of those ideas at the "only the idea" stage of development.

If there is interest in all this, then maybe. Currently I have that set
of patches from above and another set which simply overrides another ton
of expressions directly in `all-packages.nix` so that this craziness is
limited to stuff I don't want to touch anymore. I'm not happy about
this, but I'm okay.

Cheers,
  Jan

[1] https://github.com/NixOS/nixpkgs/compare/master...oxij:xbmc-11
    this was merged into master awhile ago.

Herwig Hochleitner <hhochleitner at gmail.com> writes:

> Contrary to nixos *module* options, which are self-documenting at
> http://nixos.org/nixos/options.html, there is no equivalent documentation
> for options available in nixpkgs.config. ...