[Nix-dev] Dynamic Dependency Management by Calling External Programs from Nix

[Marc Weber]

Excerpts from Shea Levy's message of Wed Oct 26 17:13:13 +0200 2011:
>      boot.extraModulePackages = config.boot.kernelPackages.kernelModule "ext4"

That's what I do for gimp:

  (pkgs.gimpGit.withPlugins [ "ufraw" "lqrPlugin" "fourier" "resynthesizer" "waveletSharpen" /*"gmic"*/ ])

and it works fine :)

But yes: I have a small list of gimp plugins only. And it brakes (due to
statesful $HOME/.gimp config files - but that's another story)

I see two main use cases:

a) 
  - use hydra
  - build everything possible (all modules)
  - create dependency list of modules which can be copied to /boot
    (necessary for startup)

  We know that the kernel build system does some kind of dependency
  analysis. Why do we need it? Why do we need it inside nix?
  AFAIK the only case is copying modules to /boot/* so that they are
  available in initrd. But we could do this at build time:
  /nix/store/xxxx-kernel/bin/copy-modules-and-deps-to-dir $out/modules-for-init-rd ext4 reiserfs ..

  I don't see why you really need readFile?
  even if you pass dependency information into the nix language you have
  to build the kernel first or unpack the sources  - taking much time

b)

  - always built custom kernel, never use hydra
  - only built what you require
  - then you can build modules required for startup into the kernel -
    don't use modules => you're fine. No need for readFile ?

  config could look like this:

  kernelPackages.withModules {
    "ext4" => "builtin"
    "my-dvb-t-stick-support" => "module"
  }


So tell me again in which way your case differs from a) or b).


Another option:
===============
Is it possible to do the same for kernel what Eelco
did for python: build modules individually as needed?
I don't have time..

> Something along these lines, yeah. I would want to limit the uses of the 
> extended features to as few points as possible, though.
Yeah - you can't protect against language abuse. Se we must assume that
users are smart :(

> This is another reason to prefer extending builtins.readFile.
What does it do exactly?
So do you want a readNixExprFromFile function limiting it allowing only
reading attr names (something like JSON) without functions?

> How does the proposed feature make this any worse then it is now? If 
> certain information needs to be saved in the name of efficiency, save it 
> in nixpkgs or do it once and save it in the store if it's global info.
Not really. I just tried illustrating what might happen if you start
using your proposal and allow network access..

> > If a mirror is unavailable evaluation of nix-env would break.
> > Thus doing syntax checks would become very hard.
> What do you mean by doing syntax checks here?
When merging upstream I always try to evaluate everything which catches
many failures such as "duplicated lex/bison names".
If such a check suddenly compiles a kernel .. :) Then I'd probably stop
doing it.

Note: consider dropping lines you no longer care about when replying.

The big question is: What should be done with cpan/haskell/.... why?
nix* was built to handle all cases: If you need older version you still
get a working system.

If you use dynamic languages updating to the newest library can be a
risk. That's why plone and its object database converge so slowly by
design.. There is a lot of code using old libraries - because you can't
afford updating and retesting.

>From that experience I conclude that its not enough to run tools like
  cpan2nix
  cabal2nix
  ....

eventually you have to run it multiple times:

cpan2nix 
    --target hydra  # will create hydra.nix
    --depends-on xml<2.0,lockfile>3.0,foo!=7

# my script does not use much, it may break. And its likely to work with all versions
cpan2nix 
  --target my-script  # will create my-script.nix
  --depends-on xml,lockfile,foo 


python2nix 
  --target my-backup-solution-please-dont-break-always-test  
  --depends-on rsync-2.0 # don't change without retesting!

Thus in the end you may have

  my-script.nix # cpan deps for this script
  hydra.nix     # cpan deps for hydra
  my-backup-[..]# python deps for this script
  [...]

when running nixos-rebuild .. ?

And yes - there is a strong reason why I talk about this: Rake files to
write down files telling about the complete set of dependencies being
used so that you don't break things. [1]
And they do it for reason I think.

Who wants to tell customers "Sorry: your database is messed up cause we had an
unexpected update?"


[1]: Gemfile.lock used by wagn (ruby web application) Michael Raskin
pointed me to in the past: Note that every dependency is hardcoded with fixed
version number!

GEM
  remote: http://rubygems.org/
  specs:
    RubyInline (3.8.6)
      ZenTest (~> 4.3)
    ZenTest (4.5.0)
    actionmailer (2.3.11)
      actionpack (= 2.3.11)
    [ .. more hardcoded versions omittet .. ]

PLATFORMS
  ruby

  [ .. the true dependencies used to find the specs above .. ]
DEPENDENCIES
  RubyInline (~> 3.8.4)
  aws-s3 (>= 0.6.2)
  hoptoad_notifier (>= 2.3.12)
  htmlentities (~> 4.2.1)
  image_science (~> 1.2.1)
  json (~> 1.4.6)
  macaddr (~> 1.0.0)
  mysql (~> 2.8.1)
  newrelic_rpm (>= 2.14.1)
  postgres (~> 0.7.9.2008.01.28)
  rack (>= 1.1.0)
  rails (= 2.3.11)
  rake (>= 0.8.7)
  gguuid (~> 2.2.0)