[Nix-dev] Generating nix expression vs. ad-hoc generation of derivations (was: Re: [Nix-commits] [NixOS/nixpkgs] c6a2a1: ffmpeg: Update to latest version 0.11.1.)

[Vladimír Čunát]

Just a line of warning, I'm a great dreamer... that's probably related
to why I do *theoretical* computer science :-)

On 24 July 2012 20:10, Florian Friesdorf <flo at chaoflow.net> wrote:
> If I understand correctly you want / are doing something like this:
>
> There is a nix expression that when called returns a nix derivation for
> a specific version of ffmpeg. I would call this ad-hoc generation of
> derivations using the nix language. I might be mistaken, but I think
> in a way this is similar to the approach Marc is using in his overlays
> [1].

No, I do nothing this complicated. I just used a classic technique in
functional programming: I *added a first parameter*, so by supplying
the package version you get the usual nix expression for that ffmpeg
version [video2]. This was just very simple modification and so it was
useful to implement directly in nix.

[video2] http://github.com/vcunat/nixpkgs/commits/vlada/video2


> A recurring discussion seems to be whether derivations should be
> generated on the fly like this or whether there should be nix
> expressions in nixpkgs that have a one-to-one correspondance to
> derivations.
>
> These expressions can also be generated, but that happens prior and
> independent to running nix, in whatever language desired.

I don't see much difference. The generating language can be anything
in both cases, if we have a good interface. The most important thing
IMO is that in git we have only the primary stuff and generators
(nothing generated).


> I see the following advantages:
>
> - generated expressions look very similar to manually created ones
>   making it much easier to understand what is going on

Not really needed, you should normally only read the generators and
leave the generated content only for nix utilities.

> - no knowledge needed of the tools used to create the expressions.
>   Instead it is easily possible to change an expression if needed and
>   those knowing the tools can integrate the changes into the tools
>   generating the expressions. AFAIK done like that by Peter and Andres
>   for haskellPackages.

You can always pretty-print the generated nix expression and modify
it. From experience there will always be cases when the generator
doesn't output exactly what we want (broken cabal files, etc.), so we
will want to patch some of the outputs.

> - magic, like dependency-resolution, is done outside of nix, keeping the
>   complexity of nix and nixpkgs low.

Well, this is a topic for a long discussion. I would personally rather
improve nix so it *could* handle dependency resolution. I'm not saying
we should normally use it for regular packages, as too much
variability hurts -- that's a problem in gentoo, two people are then
quite unlikely to build the same thing -- it has many unpleasant
implications.

> It feels that if we are able to generate derivations, we should do so,
> but put them into files (nixpkgs) and use these files as the interface
> between nix and the algorithms that help us to generate the derivations.

Certainly, we should generate nix expressions. The question is what
parameters will the generators accept -- how to choose the versions,
etc.

Also I again emphasize that distributing generated expressions is IMO
bad (or having them in git). We should only distribute generators and
patches for some broken to-be-generated expressions.

> 1. arbitrary source (hackage, pypi, marmalade, different ffmpeg versions)
>
> 2. magic tools (sharing code) to generate expression that declare
>    derivations, in contrast to generating them
>
> 3. nixpkgs
>
> 4. nix/nixos tools to install these package
>
> What do you not like about this approach and in what way do you think
> generating derivations on the fly is better?

I see storing the generated expressions *only as a caching mechanism*.
Nothing less, nothing more.


> On Tue, 24 Jul 2012 19:40:27 +0200, Vladimír Čunát <vcunat at gmail.com> wrote:
>> If we could generate dependencies and options from standard build
>> systems like autotools, cmake, ... that would really be great. Perhaps
>> with some amendments because of broken packages, but still it would
>> save so much code

I imagine it like this. Building:

- b1. Some *front-end* somehow generates the derivations to be built
(that is, nix expressions). That can be even a very complicated
constraint-solving gui tool which anyone configures according to his
needs. Various sources of expressions, substitutes, present packages
and other user demands can be considered here. A very complicated
process, in full strength perhaps not suitable for nix or similar
language, rather something that can use classic constraint-solving
libraries, full-blown gui etc.

- b2. The resulting expression is passed to nix as usual.

- b3. Some *environment-builder* is invoked to update the runtime
environment as required. In nixos case it would also restart services,
etc. Perhaps it could also do some configuration migration if versions
differ too much (in cases where the package don't do this well
automatically).


What can the generating look like?

- g1. Gather information:
   * what packages and substitutes are available (including hackage,
etc.). Also for other packages we would analyze configure scripts or
other build/run scripts to find out  (the results may have to be
patched or provided completely via nixpkgs). Also much of this work
could be cached, or even distributed by hydra (but in git we would
only have generators and patches).
   * which combinations aren't buildable according to attempts of
others (hydra), how much resources are the builds expected to take
   * what is the configuration and what the user wants from us: only
quickly apply security patches, have some packages on testing version
and others on some stable ones, have the most used packages use the
same libraries (to save CPU&RAM), do as little building as possible...

- g2. Compute "the best solution". That can be very complicated,
perhaps it can even be interactive, asking the user to confirm some
decisions. Like if updating glibc for most packages is a good idea at
the moment or we want to build the *new package* against the *old
library* (complicated to do now).



I hope I haven't completely confused or disgusted too many readers (if
they got this far anyway :-).

Vlada