[Nix-dev] The nixpkgs.config vs overrides (was: Documenting nixpkgs.config options)
Herwig Hochleitner
hhochleitner at gmail.com
Tue Jun 9 02:29:58 CEST 2015
2015-06-09 2:20 GMT+02:00 Jan Malakhovski <oxij at oxij.org>:
> Hi.
>
> Was it by design that you didn't mention nix-dev in Cc? If not, then I
> suggest forwarding this conversation there too. Feel free to quote my
> comments when and if necessary.
>
No, wasn't intentional. I'm used to mailing lists, that set the reply-to
properly. Forwarded the previous messages.
>
> Herwig Hochleitner <hhochleitner at gmail.com> writes:
>
> > thanks for your well thought-out response. You obviously have some
> > experience from attempts to reform that area of nixpkgs.
> >
> > The problem is that people will say than "config is deprecated, use
> >> overrides".
> >
> > How and why was config deprecated? Was there an ML thread?
>
> No idea. I have not seen an official statement about this, yet there was
> this: https://github.com/NixOS/nixpkgs/pull/6968
>
> > I kind of get what it is about. I think, the best way to use config is to
> > have it determine the specific override of a package. See what I did with
> > the wine package, recently.
> >
> > I don't think overrides are a proper replacement for config, because they
> > are clunkyer and a part of config anyway. Plus, they suffer from the same
> > shortcoming: Missing documentation, which can't easily be auto-generated.
> >
> > it was cut down with "do not add new config options, they are deprecated,
> >> we have
> >> overrides already".
> >
> > Interesting, do you have the ticket number handy?
>
> https://github.com/NixOS/nixpkgs/pull/6968
>
> > Personally I have some problems with current practices:
> >>
> >> * I don't like the idea of using `null`s in package arguments of a
> >> package expression as a way to specify features. I think this is ugly
> >> and error prone.
> >>
> >
> > Sure, to me, that is part of the problem with discoverability.
> >
> >> * Currently obsessive use of overrides generates more versions of
> >> packages than needed. Do
> >>
> >> egrep -A 3 -B 6 ' .*override' all-packages.nix
> >>
> >> every override of a package in an argument set for another package
> >> will likely to ask nix to compile another version. This is crazy. And
> >> I see that craziness in action every time I rebuild my configuration,
> >> 'cause I use a shitton of packages in `environment.systemPackages` on
> >> my desktop machines. I haven't measured the CPU time and disk space
> >> losses to this craziness, but I don't like the fact that these losses
> >> exist at all.
> >
> > I'm not sure I follow. In my understanding, only needed overrides will be
> > built. Also, changed config options will hopefully result in different
> > compilations, same as overrides. What effect should they otherwise have?
> >
> > In fact, a better model of config, should also act as a guideline as to
> > which package-trees should be precompiled. Right now, an entry in
> > all-packages.nix needs to be created for every config (or override) that
> > should be built by hydra.
>
> Suppose you have a package `foo` with option `barSupport` by default set
> to `false` and a package `baz` depending on that `foo` with `barSupport`
> set to `true`. If you do `baz = callPackages { foo = foo.override {
> barSupport = true; }; }` then you'll produce two packages when you set
> `environment.systemPackages = [ foo baz ]` in your `configuration.nix`,
> but all you actually need to do is to set `barSupport` to `true` by
> default on your system.
>
> > I think packages should assert that some options are set in their
> >> dependencies
> >>
> >
> > agreed
>
> This sentence was about that stuff right above.
>
> > that generates another version of dependency one would write
> >>
> >> somepackage_forced = somepackage.override {
> >> dependency = if config.somepackage.thisWhateverSupport
> >> then dependency.override { whateverSupport = true; }
> >> else dependency;
> >> };
> >>
> >> which doesn't.
> >
> > Are you talking about if a package `bar` uses a dependency `libfoo`,
> > configured differently from the rest of the system, while the rest of the
> > system could just aswell be run with with `bar`'s libfoo?
>
> Yes.
>
> > Such information could be generated by bidirectional inferrence, but I
> > believe that this shouldn't be done automatically, because it drives us
> > towards the non-determinism of other OSes. Instead the user should be
> able
> > to inform themselves of packages duplicated because of different
> > configurations, then they could consider using the most general
> > configuration as the system default.
>
> Yep, that is a valid point.
>
> > In that light, I think `common.*` options are not sufficient, but if we
> had
> > a tool to find those overrides, maintainers would start to look for
> common
> > ground by themselves.
> > Of course for that, a package needs to know, which options it requires in
> > its dependecies. I believe gentoo use flags are a great source for that
> > information and a better model than asserts.
>
> Yep, but having at least the `common` thingy would be awesome, wouldn't
> it? :)
>
> > When one simply does
> >>
> >> nixpkgs.config.dependency.whateverSupport = true;
> >>
> >> or maybe even
> >>
> >> nixpkgs.config.common.whateverSupport = true;
> >>
> >> * Overrides are not fixed points and this is a serious problem for some.
> >> See https://github.com/NixOS/nixpkgs/pull/7102.
> >
> > So if I understand correctly, this is about the `pkgs` parameter in
> > config.packageOverrides (pkgs: ...) being the version before it being
> > updated, while some users would expect it to iterate to a fixed point?
> > If so, I'm not sure how this could even theoretically be solved without
> > running the immediate risk of infinite override loops.
> > For what use case is this a serious problem?
>
> I suppose one could just make `packageOverrides` to receive two parameters:
> `pkgs` before (aka `super`) and `pkgs` after (aka `self`) overrides.
>
> > * For some strange reason it seems that overrides sometimes evaluate
> >> stuff too early. I haven't been able debug this, but overriding all
> >> the X11, GTK and QT packages (exactly those packages, not the packages
> >> that use those packages) to `null`s in nixpkgs and then specifically
> >> overriding some of the packages that use them shows that some of the
> >> latter packages will try to evaluate X11, GTK or QT even when
> >> explicitly overridden not to so. I guess that's because something in
> >> the derivation get's stuck on the previous version of the argument,
> >> but, as I said, I haven't been able to debug this yet.
> >>
> >
> > Can you post the overrides, that you used when discovering this?
>
> Not now, but I plan to catch and document that the next time I play with
> this.
>
> > And so now I maintain my own set of patches I rebase on top of master
> >> once and then, that goes against the grain and simplifies all that
> >> `nixpkgs.config`-style configuration stuff fairly significantly by
> >> adding another version of callConfPackages (also removing a ton of
> >> copy-paste from `all-packages.nix` as a side effect)
> >>
> >>
> https://github.com/NixOS/nixpkgs/compare/master...oxij:call-conf-package
> >>
> >> and weep every time someone rewrites and expression for a package
> >> dropping the support for `<feature>Support` options (e.g. most recently
> >> `qemu` =/).
> >>
> >
> > I like your approach there. Maybe, defaulted parameters could even be
> used
> > as a fallback to discover package config options.
> >
> > Said that, I agree that there are problems with this:
> >>
> >> * All that `somepackage_forced` is somewhat ugly. I would prefer to have
> >> a way to describe and introspect assertions in the packages and
> >> generate those *_forced thingies automatically. Or maybe even write
> >> something like
> >>
> >> environment.systemPackages = [ somepackage.force_dependencies ];
> >>
> >
> > I created a couple of packages, that just generate their overrides
> > privately.
> > Do you use those _forced versions to get prebuilding by hydra?
>
> _forced and the related stuff from that section in my post is just a
> concept. I have not implemented it yet.
>
> > * Package options need documentation (which the original message by
> >> Herwig Hochleitner is about). This is totally doable even now with
> >> reusing NixOS modules in nixpkgs:
> >>
> >> { dependency1, dependency2, ... }:
> >> { config, ...} :
> >> { options = { supportWhatever = mkOption ...; };
> >> assertions = [ ... ];
> >> config = mkDerivation { ... };
> >> };
> >>
> >> and then fixing `callPackage` to grab only the `config` and move the
> >> `options` to `meta`. And then collecting the latter by grabbing all
> >> the stuff from the packages `nix-env -q` uses (one could even show the
> >> options in nix-env this way).
> >>
> >
> > That's what I meant by "use a schema similar to nixos modules". This
> could
> > be used to replace the fallback, if you want to document allowed values
> or
> > a description.
> >
> > But I agree that this is somewhat ugly. Especially because usually
> >> all `supportWhatever` does is adds a flag to `configureFlags` and
> >> another dependency to `buildInputs`.
> >>
> >
> > Well, at the bottom needs to be something general, that can do it all.
> > Sugar for this default case can be added afterwards.
> >
> > The proper solution is to implement a dual of `bulitins.functionArgs`
> >> that _generates_ a function with arguments and their defaults set from
> >> an attrset.
> >
> >
> > +1, but that's not enough when you want docstrings and enum values.
>
> It kinda is. If I could generate a function from an attrset description
> then, in principle, I could add any meta info to the `meta` of the
> package it produces using the same attrset description.
>
> > So, I'm totally in favor of doing something to better the situation. And
> > I'd ask anybody, so inclined, to explain to me how overrides are enough,
> > with respect to the mentioned issues.
> >
> > I think further work on this should be framed as "enumerating and
> > documenting pre-visioned overrides"
>
> > 2015-05-28 14:30 GMT+02:00 Jan Malakhovski <oxij at oxij.org>:
> >>
> >> And so now I maintain my own set of patches I rebase on top of master
> >> once and then, that goes against the grain and simplifies all that
> >> `nixpkgs.config`-style configuration stuff fairly significantly by
> >> adding another version of callConfPackages (also removing a ton of
> >> copy-paste from `all-packages.nix` as a side effect)
> >>
> >>
> https://github.com/NixOS/nixpkgs/compare/master...oxij:call-conf-package
> >>
> >> and weep every time someone rewrites and expression for a package
> >> dropping the support for `<feature>Support` options (e.g. most recently
> >> `qemu` =/).
> >>
> >
> > Are you talking about
> >
> https://github.com/NixOS/nixpkgs/commit/19c259161b987fcb5e1c3b7a87c746372ce017ee#commitcomment-11453831
> > per chance?
>
> Yep, that's the thing that made me weep that time.
>
> Cheers,
> Jan
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.science.uu.nl/pipermail/nix-dev/attachments/20150609/d22d4b1e/attachment-0001.html
More information about the nix-dev
mailing list