[Nix-dev] [SPAM] Dynamic Dependency Management by Calling External Programs from Nix

Wed Oct 26 17:13:13 CEST 2011

On 10/25/11 2:29 PM, Marc Weber wrote:
> First:
> What is your goal? Have a kernel derivation expose a list of modules it
> has built so that configuration.nix can fail because the kernel does not
> support nvidia?
> Thus you have to build the kernel before evaluation of nixos system
> can finish?
> What was your goal? Saving time ? :-) (Don't take this too serious)

My goal is to be able say something like

     boot.extraModulePackages = config.boot.kernelPackages.kernelModule 
"ext4"

And have my resulting system have a bzImage, the ext4 module, and the 
dependencies for the ext4 module, and nothing else relating to the 
kernel build. To do that now, we'd have to maintain a list of module 
dependencies for each kernel inside nixpkgs itself, or have the 
dependencies of the ext4 module be built as part of the same derivation 
as the ext4 module itself, breaking the modularity I want. If it were 
possible to use the contents of the output of a derivation in the nix 
expression itself, however, this wouldn't be an issue.

>>      a single manifest file that contains only the info that cabal2nix
>>      would have pulled from the network (hashes+urls of .cabal files, and
> You already have this: its called hack-nix. It works fine. The manifest
> is a huge .nix db used by the solver which was written in nix.
> All you have to do is install it.
> I did the same for Ruby (which is working fine but causes you to have
> 20.000 files about packages you never want to use.)

I will look more into hack-nix and respond to this later.

> So how would your proposal look like?
>
>
> == building darcs sample ==
>
>      hackageIndex = fetchurl { .. } # keep binary format
>
>      haskellDependencyTree = { hackageIndex, ghc_version, flags, target_packages}:
>        mkDerivation = {
>          name = "run cabal2nix dynamically";
>          hash_of_resulting_default_nix = "xx...";
>          buildPhase = ''
>            # (2) run cabal2nix on hackageindex creating dependency tree for target_packages
>            # (3) write result to $out/auto-nix-files/default.nix
>          '';
>        }
>
>      Then you can do
>
>          darcs = import (haskellDependencyTree { ghc-2.8, "+old-base", "darcs" }+"/auto-nix-files/default.nix" )

Something along these lines, yeah. I would want to limit the uses of the 
extended features to as few points as possible, though.

> ==
>
> and you have to make sure that nix-build and nix-env cannot exploit your system!
> Eg default.nix could contains strange loops causing segfaults ..
> Thus never ever run nix-build as root again ?

This is another reason to prefer extending builtins.readFile. This way 
nix-build can still only ever run derivations in the store, which is all 
it can do now so there's no added risk.

> Funnier: cross compiling: first you have to wait for compiling cabal2nix
> for arm, then you will have to build haskellDependencyTree on arm
> architecture copying back the dependency/package information you want to
> use to create .drv files for the haskell packages.
> Thus without "arm" architecture you can't evaluate anything anymore!
> Good thing?

This highlights the major cost, and why I think it would be a good idea 
for the feature to be used as infrequently as possible: When this 
feature is used, we lose the separation between evaluation and 
instantiation.

That being said, however, if haskellDependencyTree would need to be run 
on the arm architecture to generate proper arm dependencies, that means 
cabal2nix does too so we need to have the arm information already stored 
in the nixpkgs tree in order to evaluate arm haskell packages today, 
which obviously isn't the case. The nice thing about this feature is 
that it doesn't eliminate the old way of doing things: If the cost of 
separating evaluation and instantiation is too high, you can just 
precompile the information you need in some text format and save that in 
nixpkgs.

>
> ==>  future: ruby forge, cpan, eclipse packages, vim, emacs, haxe,
> java/scala libraries, python, [ .. endless list ]
>
> How to cope with them? Do you want to dump cpan? (maybe there is no
> other way, could those package databases be split into pieces? .. )

If we're not allowing network access for non-fixed-output derivations 
(which I think is a good policy), then we'd need some sort of manifest 
that contains all (and ONLY) the information a tool like cpan2nix would 
pull from the network (that could include a set of url/hash pairs for 
downloading individual package files, rather than all of the info in the 
package file, if they exist in cpan). Right now we have more than that, 
for cabal2nix packages: we have not only the info extracted from cabal 
files, but also info CALCULATED from cabal files. A manifest with 
url/hash pairs containing the paths to a bunch of cabal files and source 
tarballs should be all that's necessary with an expanded 
builtins.readFile, from my understanding of how cabal2nix works.

> python is a very funny case: pypi (the package index) exists, but its
> dependency information is not complete. You can get dependencies by
> ./setup.py --show-dependendencies or such.
>
> Thus you have to download and fetch a source of a package in order to
> know about its dependencies so that you can build and pass them.
>
> Thus there is no nice way to statically fetch this information in
> advance. (I still have to test whether you can run setup.py --show-deps
> or such on each package)
> Again nix-build would take ages (fetching all python sources)..

How does the proposed feature make this any worse then it is now? If 
certain information needs to be saved in the name of efficiency, save it 
in nixpkgs or do it once and save it in the store if it's global info.

> If a mirror is unavailable evaluation of nix-env would break.
> Thus doing syntax checks would become very hard.

Right, another cost of making instantiation dependent on realisation and 
a good reason why we'd want to keep this feature used as rarely as 
possible and probably to require a command-line flag/nix config option 
to enable it. Also don't use mirrors that go down ;)

What do you mean by doing syntax checks here?

> Let's hope that output does not depend on time or kernel version or
> you'll get different package sets using same nixpgks *shrug* :)

Well if builds depend on time or kernel version they might do so anyway, 
except we won't know it since the store hashes will be the same so we'll 
think everything's the same. But the "different package sets using the 
same nixpkgs" is a good reason to not allow builtins.exec, as that will 
depend heavily on what commands are available in the evaluating user's 
PATH, whereas extending readFile would mean that the command being run 
to generate nix expression information is buildable by nixpkgs.

> While it one step towards "managing scala, java, .. " libraries you also
> give up some control.
>
> But yes: I think its the only way to manage sbt, maven, ivy builds
> unless Java/Scala community join nixos (unlikely).
>
> It would be terrific if nixos could share eclipse plugins.
>
> but you'll also give up about "nix, tell me how long installation will
> take: show me how many bytes will be downloaded, estimate build time
> based on recorded build times found on hydra and my cpu power".
> It does not exist yet - but could be implemented. It still can,
> but the ETA will change the same way it does when running find on /.

Yes, but remember you give up on this conditionally. For a large class 
of builds, this feature will be unnecessary and so can be disabled. For 
those where it's either necessary or at least a big improvement, the 
benefit is a smaller, less-brittle, more customizable nixpkgs.

> I don't have the energy to think about all implications ..
>
> Sincerly
> Marc Weber
>
> The related mail I never sent. But maybe its time to do so:

This mail goes into some really good real-world detail on why this kind 
of feature would be useful. Thanks, Marc.

>
> From: Marc Weber<marco-oweber at gmx.de>
> To: nix-dev at cs.uu.nl
> Cc:
> Bcc:
> Subject: java/scala (maven,ivy,sbt); ruby (gem); python (easyinstall,pip); perl (cpan); eclipse (p2) =>  future?
> Date: Tue, 20 Sep 2011 03:14:19 +0200
> Message-Id:<1316476599-sup-1988 at nixos>
>
> There are many universes..
>
> 1) Do we want to support them in the future?
> 2) How do we want to support them in the future?
>
> They all have in common that there is kind of structure based on the
> individual build systems:
> - you have a package pool
> - you can query dependencies (sometimes this can be done after
>    installing a package only :-(, eg see all the python packgaes)
> - the package pool is that large - that you don't want to have
>    a full copy on your hard disk. Eg why have 20.000 package descriptions
>    (ruby) or 5.000 (haskell) or probably much more (cpan) if you're only
>    interested in 20 packages (and those may not have been packaged yet!)
>
> nixpkgs still has "simpleBuildTool" version 0.7.x. Current versions are
> 0.10 and 0.11: https://github.com/harrah/xsbt/wiki/Setup
>
> How to bootstrap?
> - get a .jar
> - run it
> What does it do?
> Its reusing existing maven/ivy/.. infrastructure: It fetches packages
> and puts the result into ~/.ivy2/... the way maven/.. tools do.
>
>
> We have three options:
> (1) adopt Nix (allow derivations to be build dynamically depending on
>    output of other packages)
> (2) package everything manually - only that what we need. (Nice, but a
>    lot of effort)
> (3) try to prefetch all info we can get from all packages. Then use
> server farms to keep things around - because in the long run nobody
> wants to have a dump of cpan/hackage/... on his/her disk.
>
> We can do (2) for packages we need only. Still its causing a lot of work
> to keep up to date. (Eg see how much time Peter had to spend on Haskell)
>
> proposal:
> What about modifying nix so that depending packages may depend on the
> outcome of a package?
>
> Example:
>
> python-lib-A is copied into $TMP (builddir).
> Then you can run pyhton setup.py --show-dependencies>>  $out/nix-support/deps
> Then you nix can read that special file and generate derivations for
> those deps on the fly also creating a new derivation for lib-A which
> depends on "dynamic dependencies".
>
> Yes - this will cause a lot of "please fix hash and store it somewhere".
> But that's all you have to do - and that could be automated.
>
> What does this mean?
> When generating derivations you do no longer know the total amount of
> derivations until some of the "dynamic" derivations have been built.
> Eg python-lib-A may depend on python-lib-B which depends on
> python-lib-C. However this all is known after running python setup.py
> --show-dependencies .. of python-lib-A and -B.
>
> I know that this suggestion is a huge step - and that we have to think
> carefully about the implications.. However I don't see how we should be
> able to use the power of nix for all those code with the current man
> power we have.
>
> I know what I'm talking about. Eg the zipped package descriptions of all
> available ruby packages is about 27MB of size! Its great. It works.
> I've been able to install 5 year old packages with little effort some
> time ago. But I don't think its a good idea to have 141MB of unzipped
> 27884 files laying around because you want to install 10-20 ruby
> libraries..
> Neither is it an option to spend 2 hours on package things manually
> if you can use existing ruby tools to install them (such as gem install,
> easyinstall, pip, cabal, cpan ..)
>
> How do you think about this?
>
> If we made that one little change we could package all those universes
> easily without causing mass downloads of package descriptions nobody
> wants to have on his/her disk. That's fine for devs. But not for end
> user or server systems (?)
>
> But they all use their own "stateful" installation directories.
> So I think that it would be best to patch them so that they use the Nix
> store as storage engine only. Maybe we can get what nobody else has:
> Shared storage of packages on multi user systems this way without
> spending additional days / weeks / month on writing nix expressions
> manually.
>
> I could extend the list of the subject by el-get (emacs install system),
> vim-addon-manager (vim install system), the haxe library installer
> (haxelib), pear (php), and even zsh ships its own install system ..
> And I'm pretty sure that when thinking about it you'll find more use
> cases - even firefox plugins could be installed in /nix/store/..
> (although I don't recommend this right now, talking about the future
> here)
>
> So why do the effort at all - why not using the individual package
> solutions?
>
> Because for admins its easy
> - to verify integrity of packages (using nix-store --verify --check-contents)
> - because you can duplicate working setups without caring too much about
>    the individual package systems
> - you can share builds among users.
> - you can still have the one /etc/nixos/configuration.nix file for
>    everything.
>
> Thoughts?
>
> Marc Weber
>
> Need a sample what I'm talking about? The new simple build tool
> bootstrap .jar dependency list:
>
> B
> C
> You package this once maybe two times. But at some time you want to code - not package.
> (Yes - I want to face the real world)
>
> downloading http://repo1.maven.org/maven2/net/java/dev/jna/jna/3.2.3/jna-3.2.3.jar ...
> downloading http://repo1.maven.org/maven2/org/scala-lang/scala-compiler/2.8.1/scala-compiler-2.8.1.jar ...
> downloading http://repo1.maven.org/maven2/org/scala-lang/scala-library/2.8.1/scala-library-2.8.1.jar ...
> downloading http://repo1.maven.org/maven2/jline/jline/0.9.91/jline-0.9.91.jar ...
> downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/sbt_2.8.1/0.10.1/jars/sbt_2.8.1.jar ...
> downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/main_2.8.1/0.10.1/jars/main_2.8.1.jar ...
> downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/compiler-interface/0.10.1/jars/compiler-interface-bin.jar ...
> downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/compiler-interface/0.10.1/jars/compiler-interface-src.jar ...
> downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/precompiled-2_9_0-1/0.10.1/jars/compiler-interface-bin.jar ...
> downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/actions_2.8.1/0.10.1/jars/actions_2.8.1.jar ...
> downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/interface/0.10.1/jars/interface.jar ...
> downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/io_2.8.1/0.10.1/jars/io_2.8.1.jar ...
> downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/ivy_2.8.1/0.10.1/jars/ivy_2.8.1.jar ...
> downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/launcher-interface_2.8.1/0.10.1/jars/launcher-interface_2.8.1.jar ...
> downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/logging_2.8.1/0.10.1/jars/logging_2.8.1.jar ...
> downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/process_2.8.1/0.10.1/jars/process_2.8.1.jar ...
> downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/run_2.8.1/0.10.1/jars/run_2.8.1.jar ...
> downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/classfile_2.8.1/0.10.1/jars/classfile_2.8.1.jar ...
> downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/classpath_2.8.1/0.10.1/jars/classpath_2.8.1.jar ...
> downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/incremental-compiler_2.8.1/0.10.1/jars/incremental-compiler_2.8.1.jar ...
> downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/persist_2.8.1/0.10.1/jars/persist_2.8.1.jar ...
> downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/compile_2.8.1/0.10.1/jars/compile_2.8.1.jar ...
> downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/completion_2.8.1/0.10.1/jars/completion_2.8.1.jar ...
> downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/discovery_2.8.1/0.10.1/jars/discovery_2.8.1.jar ...
> downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/task-system_2.8.1/0.10.1/jars/task-system_2.8.1.jar ...
> downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/tasks_2.8.1/0.10.1/jars/tasks_2.8.1.jar ...
> downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/tracking_2.8.1/0.10.1/jars/tracking_2.8.1.jar ...
> downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/testing_2.8.1/0.10.1/jars/testing_2.8.1.jar ...
> downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/control_2.8.1/0.10.1/jars/control_2.8.1.jar ...
> downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/collections_2.8.1/0.10.1/jars/collections_2.8.1.jar ...
> downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/api_2.8.1/0.10.1/jars/api_2.8.1.jar ...
> downloading http://repo1.maven.org/maven2/org/scala-tools/sbinary/sbinary_2.8.1/0.4.0/sbinary_2.8.1-0.4.0.jar ...
> downloading http://repo1.maven.org/maven2/org/apache/ivy/ivy/2.2.0/ivy-2.2.0.jar ...
> downloading http://repo1.maven.org/maven2/com/jcraft/jsch/0.1.31/jsch-0.1.31.jar ...
> downloading http://repo1.maven.org/maven2/commons-httpclient/commons-httpclient/3.1/commons-httpclient-3.1.jar ...
> downloading http://repo1.maven.org/maven2/commons-logging/commons-logging/1.0.4/commons-logging-1.0.4.jar ...
> downloading http://repo1.maven.org/maven2/commons-codec/commons-codec/1.2/commons-codec-1.2.jar ...
> downloading http://repo1.maven.org/maven2/jline/jline/0.9.94/jline-0.9.94.jar ...
> downloading http://repo.typesafe.com/typesafe/ivy-releases/org.scala-tools.sbt/cache_2.8.1/0.10.1/jars/cache_2.8.1.jar ...
> downloading http://repo1.maven.org/maven2/org/scala-tools/testing/test-interface/0.5/test-interface-0.5.jar ...
> _______________________________________________
> nix-dev mailing list
> nix-dev at lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev

~Shea

[Nix-dev] [***SPAM***] Dynamic Dependency Management by Calling External Programs from Nix

[Nix-dev] [SPAM] Dynamic Dependency Management by Calling External Programs from Nix