[Nix-dev] Fwd: The nixpkgs.config vs overrides (was: Documenting nixpkgs.config options)

[Herwig Hochleitner]

---------- Forwarded message ----------
From: Herwig Hochleitner <hhochleitner at gmail.com>
Date: 2015-06-01 15:04 GMT+02:00
Subject: Re: The nixpkgs.config vs overrides (was: Documenting
nixpkgs.config options)
To: Jan Malakhovski <oxij at oxij.org>


2015-05-28 14:30 GMT+02:00 Jan Malakhovski <oxij at oxij.org>:

> Hi.
>

Hi Jan,

thanks for your well thought-out response. You obviously have some
experience from attempts to reform that area of nixpkgs.

The problem is that people will say than "config is deprecated, use
> overrides".


How and why was config deprecated? Was there an ML thread?

I kind of get what it is about. I think, the best way to use config is to
have it determine the specific override of a package. See what I did with
the wine package, recently.

I don't think overrides are a proper replacement for config, because they
are clunkyer and a part of config anyway. Plus, they suffer from the same
shortcoming: Missing documentation, which can't easily be auto-generated.

it was cut down with "do not add new config options, they are deprecated,
> we have
> overrides already".
>

Interesting, do you have the ticket number handy?

Personally I have some problems with current practices:
>
> * I don't like the idea of using `null`s in package arguments of a
>   package expression as a way to specify features. I think this is ugly
>   and error prone.
>

Sure, to me, that is part of the problem with discoverability.


> * Currently obsessive use of overrides generates more versions of
>   packages than needed. Do
>
>     egrep -A 3 -B 6 '   .*override' all-packages.nix
>
>   every override of a package in an argument set for another package
>   will likely to ask nix to compile another version. This is crazy. And
>   I see that craziness in action every time I rebuild my configuration,
>   'cause I use a shitton of packages in `environment.systemPackages` on
>   my desktop machines. I haven't measured the CPU time and disk space
>   losses to this craziness, but I don't like the fact that these losses
>   exist at all.
>

I'm not sure I follow. In my understanding, only needed overrides will be
built. Also, changed config options will hopefully result in different
compilations, same as overrides. What effect should they otherwise have?

In fact, a better model of config, should also act as a guideline as to
which package-trees should be precompiled. Right now, an entry in
all-packages.nix needs to be created for every config (or override) that
should be built by hydra.

  I think packages should assert that some options are set in their
>   dependencies
>

agreed

  that generates another version of dependency one would write
>
>     somepackage_forced = somepackage.override {
>       dependency = if config.somepackage.thisWhateverSupport
>                    then dependency.override { whateverSupport = true; }
>                    else dependency;
>     };
>
>   which doesn't.


Are you talking about if a package `bar` uses a dependency `libfoo`,
configured differently from the rest of the system, while the rest of the
system could just aswell be run with with `bar`'s libfoo?
Such information could be generated by bidirectional inferrence, but I
believe that this shouldn't be done automatically, because it drives us
towards the non-determinism of other OSes. Instead the user should be able
to inform themselves of packages duplicated because of different
configurations, then they could consider using the most general
configuration as the system default.
In that light, I think `common.*` options are not sufficient, but if we had
a tool to find those overrides, maintainers would start to look for common
ground by themselves.
Of course for that, a package needs to know, which options it requires in
its dependecies. I believe gentoo use flags are a great source for that
information and a better model than asserts.

When one simply does
>
>     nixpkgs.config.dependency.whateverSupport = true;
>
>   or maybe even
>
>     nixpkgs.config.common.whateverSupport = true;
>
> * Overrides are not fixed points and this is a serious problem for some.
>   See https://github.com/NixOS/nixpkgs/pull/7102.
>

So if I understand correctly, this is about the `pkgs` parameter in
config.packageOverrides (pkgs: ...) being the version before it being
updated, while some users would expect it to iterate to a fixed point?
If so, I'm not sure how this could even theoretically be solved without
running the immediate risk of infinite override loops.
For what use case is this a serious problem?

* For some strange reason it seems that overrides sometimes evaluate
>   stuff too early. I haven't been able debug this, but overriding all
>   the X11, GTK and QT packages (exactly those packages, not the packages
>   that use those packages) to `null`s in nixpkgs and then specifically
>   overriding some of the packages that use them shows that some of the
>   latter packages will try to evaluate X11, GTK or QT even when
>   explicitly overridden not to so. I guess that's because something in
>   the derivation get's stuck on the previous version of the argument,
>   but, as I said, I haven't been able to debug this yet.
>

Can you post the overrides, that you used when discovering this?

And so now I maintain my own set of patches I rebase on top of master
> once and then, that goes against the grain and simplifies all that
> `nixpkgs.config`-style configuration stuff fairly significantly by
> adding another version of callConfPackages (also removing a ton of
> copy-paste from `all-packages.nix` as a side effect)
>
> https://github.com/NixOS/nixpkgs/compare/master...oxij:call-conf-package
>
> and weep every time someone rewrites and expression for a package
> dropping the support for `<feature>Support` options (e.g. most recently
> `qemu` =/).
>

I like your approach there. Maybe, defaulted parameters could even be used
as a fallback to discover package config options.

Said that, I agree that there are problems with this:
>
> * All that `somepackage_forced` is somewhat ugly. I would prefer to have
>   a way to describe and introspect assertions in the packages and
>   generate those *_forced thingies automatically. Or maybe even write
>   something like
>
>     environment.systemPackages = [ somepackage.force_dependencies ];
>

I created a couple of packages, that just generate their overrides
privately.
Do you use those _forced versions to get prebuilding by hydra?

* Package options need documentation (which the original message by
>   Herwig Hochleitner is about). This is totally doable even now with
>   reusing NixOS modules in nixpkgs:
>
>     { dependency1, dependency2, ... }:
>     { config, ...} :
>     { options = { supportWhatever = mkOption ...; };
>       assertions = [ ... ];
>       config = mkDerivation { ... };
>     };
>
>   and then fixing `callPackage` to grab only the `config` and move the
>   `options` to `meta`. And then collecting the latter by grabbing all
>   the stuff from the packages `nix-env -q` uses (one could even show the
>   options in nix-env this way).
>

That's what I meant by "use a schema similar to nixos modules". This could
be used to replace the fallback, if you want to document allowed values or
a description.

  But I agree that this is somewhat ugly. Especially because usually
>   all `supportWhatever` does is adds a flag to `configureFlags` and
>   another dependency to `buildInputs`.
>

Well, at the bottom needs to be something general, that can do it all.
Sugar for this default case can be added afterwards.

  The proper solution is to implement a dual of `bulitins.functionArgs`
>   that _generates_ a function with arguments and their defaults set from
>   an attrset.


+1, but that's not enough when you want docstrings and enum values.

So, I'm totally in favor of doing something to better the situation. And
I'd ask anybody, so inclined, to explain to me how overrides are enough,
with respect to the mentioned issues.

I think further work on this should be framed as "enumerating and
documenting pre-visioned overrides"

Also for the convenience of people starting with this thread, I'll
bottom-quote my original message here.

thanks

2015-05-26 13:11 GMT+02:00 Herwig Hochleitner <hhochleitner at gmail.com>:

> Contrary to nixos *module* options, which are self-documenting at
> http://nixos.org/nixos/options.html, there is no equivalent documentation
> for options available in nixpkgs.config. This is unfortunate, because a
> recent blog post (http://blog.lastlog.de/posts/useflags_in_nixos/) makes
> the very good point of framing them as a gentoo - useflag equivalent.
>
> Thus users wanting to customize packages, are left with the option of
> diving into the source, or hunting for wiki pages about their target
> package, e.g. https://nixos.org/wiki/Enable_Browser_Plugins
>
> https://github.com/NixOS/nixpkgs/issues/7446 is clearly relevant,
> however, I'm primarily concerned with documentation right now, since I want
> users to find the new wine options in the upcoming release.
>
> Manually compiling a wiki page might work for now, but that will
> eventually get outdated. Maintaining available flags in per-package
> metadata might be better, but still risks running out of sync with careless
> commits. Declaring a schema (a'la nixos-options), however, feels a little
> heavy handed for nixpkgs.config.
>
> Has there been work towards this? How should this be done?
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.science.uu.nl/pipermail/nix-dev/attachments/20150609/06ec7eac/attachment-0001.html