[Nix-dev] Dependency Semantics

[Mathijs Kwik]

Bryce L Nordgren <bnordgren at gmail.com> writes:

> On Wed, Jul 11, 2012 at 12:53 AM, Mathijs Kwik <mathijs at bluescreen303.nl> wrote:
>
>     But the same thing cannot be said about
>     Ruby (in all fairness: Ruby's main goal never was to be a VM) and most
>     other environments. Also, some of these "language/platform environments"
>     have optional components. Let's say some  imaging library gets updated
>     and (for example) python is not able to use it anymore (upstream decided
>     to move the python-special-imaging-thingy to a separate package/egg).
>     Currently, when python gets rebuilt, it won't be able to detect
>     image-thingy headers for a supported version, so it silently decides to
>     skip building just that small part.
>
> There may be a couple of things I didn't articulate well.
>
> 1] I don't see these new semantics as "either/or". I fully expect you to be able to declare a weak dependency on
> the python environment and a "depends on exact build of" for the imaging thingy.
> 2] I would envision the semantics of java packages to other java packages to be "depends on exact build of".
> It's the dependency on the environment which needs to be weaker.

I understand that part, but even using a weak dependency on environments
can lead to problems. Essentially these "weak" dependencies are
like a flag to tell the resolver "do not rebuild, although you think you
should, I know better". I know that in some cases, this indeed is the
case, but in many subtle cases, it really isn't. 

I'm afraid (new) maintainers are tempted to put these flags on their
packages by default ("hey, it's a python package, it should probably
weak-dep on python), even when there's a hidden problem like I described
in my example (see below again). This will in the end lead to
"clean installs" being more trustworthy than upgraded systems. 

>
> Consider the example you gave though. Some imaging library triggers a python rebuild. The newly built python
> lacks some imaging functionality due to python bindings on the library being moved to a different namespace and
> python's build script can't autodetect it. So far, it's the same result whether you force all python packages to
> be rebuilt or not: both rebuilds are broken. 

Nope, that's not what I meant. and it's not the same result:
Python 2.7 can optionally use ImageThingy version 1.4.*. Python's 
(upstream) build scripts detect this library and use it (build bindings 
for it). If it's not detected, it doesn't lead to an error, but python's
build just decides not to build the bindings.

MyPackage just depends on python (not on ImageThingy, because that is
usable implicitly if python is installed). Its (upstream) build script
does check for the bindings and will fail when not found.

Now, ImageThingy 1.5 gets released and Python 2.7 is not able to
use/detect it. In the current situation, python rebuilds successfully,
but will of course now miss the bindings.
MyPackage gets rebuilt as well (because python changed) and report it is
now missing the ImageThingy headers from python. So the breakage is
detected.

In the "weak dep on python" case however, MyPackage doesn't rebuild
(because only python changed), so its build script will not run to
detect breakage. Only far later, when I try to run MyPackage on some
production server, I find out this problem.
To fix this, MyPackage's nix expression should capture the fact that it
depends on Python being built with ImageThingy support.

Of course, explicitly documenting dependencies isn't a bad thing on its
own, but MyPackage's maintainer never got really deep with inspecting
these internal workings. He just took the standard python-package
template, added some meta-info, and found it worked. We cannot blame him
for not diving into every little detail in the (upstream) build process
and watching it for future changes too.

> "depends on exact build of" does not prevent what it's supposed to
> prevent, it just means that after all the unrelated python packages are recompiled, the duplicates depend
> specifically on the broken python. In both cases, you fix it with a rollback and file a bug with upstream. You
> can rollback sooner if you're not waiting for all python packages to finish being built.
>
> Oh, and BTW: The new semantics are replacing the tricks currently used to fool Nix into believing there is no
> relationship at all between the two packages. They at least allow nix-env to check for the completeness and
> consistency of a set of installed software, whereas it can't do that now. The weaker semantics are not weakening
> anything. They are allowing currently missed dependencies to be
> specified.

As stated, I'm not against this, especially not if it's just replacing
dirty trickery. But my example, although contrived, is probably
something we can find in-the-wild. I would opt for the weak semantics to
be _never_ used by default (not in any templates for environments like
python), as they should _only_ be used when a maintainer fully
understands the (upstream) build process, and which checks it
performs. If this is fully the case (and this is re-checked if packages
upgrade), we can indeed save some time by specifying a certain
dependency is "weaker".

Mathijs

>
> Bryce
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
Url : http://lists.science.uu.nl/pipermail/nix-dev/attachments/20120711/73beb0d0/attachment.bin