[Nix-dev] [***SPAM***] Dynamic Dependency Management by Calling External Programs from Nix

[Shea Levy]

Hi all,

For the most part, the nix expression language is extremely good at 
expressing the dependency relationship between derivations. 
Unfortunately, there are some cases where the requirement that the 
information required to resolve dependencies can only come from 
pre-existing text files (either through importing other nix expressions 
or builtins.readFile) leads to some real problems.

The obvious example is the language-specific package managers. The fact 
that a tool like cabal2nix can be used to do nearly all of the 
maintenance of Haskell packages means that we suddenly have a set of nix 
expressions that are more like "object files" than source code: They can 
be completely automatically generated, and if they need to be edited 
manually, it's considered a bug in cabal2nix and any changes will be 
overwritten on the next autogeneration (I believe Peter's policy is that 
if people need to change the generated files they should and then he 
will review and modify cabal2nix accordingly, but that is a VERY brittle 
maintenance procedure). There's a reason development repositories 
include configure.ac and Makefile.in but have configure and Makefile in 
their ignores, and it's the same reason they include .c files but have 
.o files in their ignores: The former can automatically generate the 
latter and will overwrite any changes when that occurs, and if there is 
something wrong with the latter the solution is to modify the former and 
regenerate, not change the latter.

There are some problems that are even more cumbersome to solve with 
today's limitations. For example, right now I'm working modularizing the 
kernel derivations by making separate derivations for the bzImage and 
each module. Most users will want to build the bzImage and all of the 
modules, which is easy enough, but some users might want to just build 
the modules they need to have a smaller system and shorter build for 
kernels that aren't available through substitutors. Suppose I want to 
build just ext4 for my kernel. How do I know which other modules I need 
on my system for ext4's dependencies to be satisfied? It's not too 
difficult to write a tool that, given a kernel build tree, generates a 
dependency tree for all of the in-tree modules that can be built, but 
that's a lot of bloat to add for each kernel version currently in 
nixpkgs, and of course anyone wanting to compile a custom kernel with 
different features built-in or as modules will need to generate the 
dependency tree themselves before they build their modules, which puts 
an explicit dependency on an external tool for end-users who are not nix 
devs to use nixpkgs, which is IMO a big problem.

So I'd really like to see one of two solutions implemented (I'll do the 
implementation myself if Eelco is willing to consider accepting these 
solutions) in nix (possibly with an explicit command-line flag or 
nix.config setting to enable them, like we have with enableParallelBuilds):

 1. builtins.exec. This builtin would take a string and would evaluate
    to a string. It would essentially call system() on its argument and
    evaluate to the output put to stdout by that command.
 2. Expand builtins.readFile to be able to read files in another
    output's path. Right now you can't do builtins.readFile
    "${derivation}/file", since that would require doing a nix-store
    realisation while instantiating. The advantage to this approach is
    that, unlike with builtins.exec, the nix-* commands would still be
    limited to affecting only files in the store. The disadvantage
    relative to exec is that dynamic downloads wouldn't (or shouldn't)
    be possible: you shouldn't need network access for non fixed-output
    derivations, so you need to know the hash ahead of time. Still, I
    think that disadvantage might actually be a good thing, and would
    turn the maintenance of, e.g., the Haskell packages into maintaining
    a single manifest file that contains only the info that cabal2nix
    would have pulled from the network (hashes+urls of .cabal files, and
    hashes of the sources if .cabal files don't contain that, most
    likely), then updating the set of Haskell packages would mean
    updating the url+hash of the manifest file, instead of every single
    nix expression that corresponds to an updated package.

Thoughts?

Cheers,
Shea Levy
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.science.uu.nl/pipermail/nix-dev/attachments/20111024/862d331c/attachment.html