[Nix-dev] Dependency Semantics

Wed Jul 11 08:53:46 CEST 2012

Bryce L Nordgren <bnordgren at gmail.com> writes:

> Hi list,
>  
> I want to suggest something to see what you think.
>  
> Nix currently supports a single meaning for its dependency relation, which essentially can be summed up as "Depends on
> exact build of". As in, GDAL <depends on exact build of> GEOS(hash here). This is an enormously successful model, but it
> does force you to completely omit real dependencies (and to trick Nix into not detecting them). For instance, to avoid
> recompiling all Java source to bytecode simply because some change triggers the rebuilding of the java compiler (Note
> we're not even talking about upgrading Java here: we're saying, i.e., a sound library was upgraded so the same JDK was
> rebuilt.), it has been suggested to NOT depend on the jdk and to refer to the java interpreter only by an environment
> variable so that Nix won't pick up the dependency (e.g., $JAVA_HOME/bin/java). Likewise, "hash rewriting" has also been
> suggested as a manual workaround for those situations where the "Depends on exact build of" semantics are inappropriate.
> Certainly "Depends on exact build of" will produce a workable result given enough time and CPU cycles, but there are
> situations where it is extremely wasteful. Consider the following additional semantics:
>  
> * "requires environment" : Requires the presense of the interpreter software/VM (e.g. bash/python/perl/java/ruby) but
> does not force the dependent package to be rebuilt (and stored again) if the interpreter/VM is replaced. The target of
> the dependency does not represent a specific build, but an environment description.
> * "implements environment" : Declares that the package is designed to provide a specific environment. Two
> implementations of Ruby 1.8.7 could be the reference implementation of Ruby or Ruby-Enterprise. The target of the
> dependency represents an environment description.
> * "depends on executables from" : Refers to a Nix source expression instead of a specific build. The dependency is on a
> command line utility which has already been linked. The dependant package is not rebuilt when the target is. Also: any
> build instantiated from the target expression should be considered interchangable. For example: a dependency on the
> "dot" executable from GraphVis should not cause Doxygen to be rebuilt if a rebuild of GraphVis is triggered; conversely,
> any build of GraphVis (version X) present on the system should satisfy the requirement.
>  
> I understand that there is some resistance to these suggestions because they appear too similar to the semantics present
> in other package management systems (which completely lack "depends on exact build of"). This is not an effort to make
> Nix more "familiar" or to remove "depends on exact build of". It's intended to eliminate wasteful and needless
> rebuilding (while reducing the number of missed/masked dependencies) by allowing the packager to correctly express
> relationships between packages which are not "depends on exact build of". Additionally, this would allow "nix-env"
> "nix-build" and Hydra to correctly process these directives.
>  
> Thoughts?

I think it's hard to find a good definition for a "weak" dependency. The
thing I like about nix* is that it guarantees me I can get the exact
same setup/environment/dependencies on my production env as I have on a
dev box. Furthermore, it guarantees me that a certain configuration
builds successfully, which (in the case where you run tests during a
build) is also a very important thing. 
I'm planning to get rid of a dedicated CI server, which is basically
just doing these rebuild-checks every time some source code changes.
The current CI-server setup doesn't know anything about the runtime
environment, so if I don't change any sources, but upgrade some library,
I need to force the CI to rebuild the current sources.

I agree a lot of these rebuilds are needless/wasteful.
In the case of java for example, I'm pretty sure the VM provides a very
good abstraction layer, so any changes "below" it, won't change anything
for stuff that just needs a JVM. But the same thing cannot be said about
Ruby (in all fairness: Ruby's main goal never was to be a VM) and most
other environments. Also, some of these "language/platform environments"
have optional components. Let's say some  imaging library gets updated
and (for example) python is not able to use it anymore (upstream decided
to move the python-special-imaging-thingy to a separate package/egg). 
Currently, when python gets rebuilt, it won't be able to detect
image-thingy headers for a supported version, so it silently decides to
skip building just that small part. As a result, packages depending on
python get rebuilt, and (if the upstream configure/build script does at
least some small checks) fail, because there is something wrong.

This is exactly what I want, because something changed and broke that
package. Now, if "weak" dependencies get introduced, packages might just
say "I require some python env". And as special-image-thingy was
included (batteries included:) since day 1, no package ever bothered to
specify "I require python, and it has to be built with the image
thingy". So, as a result, all these packages will decide not to rebuild
after this python change (not upgrade) and I would think they still work
fine.

The way other distributions try to fix this is by introducing special
extra-version-number components which they can bump, to force signal all
depending packages to rebuild. And advanced flags like "built with
feature X", which depending packages then need to check/mandate.

In other words, I'm afraid that your suggestion would quickly lead to
hidden problems where stuff breaks without you knowing it. Also, it will
lead to complexities to work around breakage and to manually override
does-this-depend-and-need-a-rebuild decisions. Maintainers would need to
be far more familiar with the packages they maintain, so they can
intelligently apply "requires python env - no need to rebuild when lower
deps change", while currently, they can just count on the upstream
configure/build stuff to report problems.

So while I agree with what you are trying to achieve, I'm almost certain
it won't be possible to solve these issues elegantly without giving us
hidden breakage, or needing maintainers to really spec out a lot of
details about dependencies about their packages. The auto-detection
"magic" that really impressed me when I came to nix, will be a lot less.

I understand all this could be optional (--strict-rebuild-detection or
something), but having different build strategies probably leads to a
whole new other bunch of issues :)

But in all, if there is a way, I'm not against it.

Mathijs
>  
> Bryce
>  
>  
>
> _______________________________________________
> nix-dev mailing list
> nix-dev at lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
Url : http://lists.science.uu.nl/pipermail/nix-dev/attachments/20120711/b30f2145/attachment.bin 