[Nix-dev] Import from derivation (was: Google Summer of Code 2017)

[Peter Simons]

Profpatsch writes:

> On 17-01-04 09:42pm, Vladimír Čunát wrote:
>> On 01/04/2017 08:51 PM, Peter Simons wrote:
>>
>>> Another very important topic that needs to be addressed in Nix /
>>> Hydra is the question of how to deal with code that wants to import
>>> build products into the ongoing evaluation. [...]
>>
>> That feels rather vague topic ATM. My experience is that this kind of
>> "figure it out how to..." tasks isn't very suitable for similar
>> "project proposals" like for GSoC.
>
> Sounds more like a task for a master’s thesis (or adventurous
> bachelor’s thesis) to me.

I'm not sure it's that difficult. We've already seen a nice proposal how
to implement it -- although I cannot locate that comment anywhere in the
myriad of IFD-related tickets we have on Github.

IFD basically works today, but once you actually use it, Nix starts
compiling packages during *evaluation*. That is particularly bad for
Hydra, because Hydra has a queue runner and build slaves and all kinds of
facilities designed for compiling packages that it won't use if it has to
compile stuff just to evaluate the jobset.

An obvious solution someone (probably Eelco) suggested was to split use of
IFD into two steps: one builds the to-be-imported Nix files and the other
one imports those expressions into the current package set:

  myPackageExpr = callPackage ../build/nix/expression/for/haskell/packages {};
  myPackage = import myPackageExpr;

Now, Nix can evaluate "myPackageExpr" just fine -- it's a perfectly
ordinary build process that creates a store path containing a Nix
expression. Nix will refuse, however, to evaluate "myPackage" unless it
has already built "myPackageExpr" or unless it can substitute that store
path from a binary cache.

Whenever the inputs of myPackageExpr change, Hydra compiles that store
path and uploads it to cache.nixos.org, but it *won't* compile "myPackage"
in the same evaluation. In the next evaluation, however, it will.

IMHO, this is a pragmatic solution that should work well in practice. We
just need someone to implement it.

Best regards,
Peter