[Nix-dev] Dependency Semantics

[Mathijs Kwik]

Bryce L Nordgren <bnordgren at gmail.com> writes:

> On Thu, Jul 12, 2012 at 12:40 PM, Mathijs Kwik <mathijs at bluescreen303.nl> wrote:
>
>     That's not what I was talking about, the python expression didn't change
>     in my example. Just a (made up) underlying graphics library.
>     That triggers a python rebuild, which can't use the new gfx version, so
>     it builds without bindings for it.
>     Then a package on top of that gets triggered to rebuild, and its
>     upstream configure/build script detects the missing bindings and fails.
>
> How often do you expect to run into this? I can see explicitly specifying something like --with-ImageThingy=/
> nix/store/{hash}-lib/blah.so on python's command line (or more prominently, PHP's. PHP has an awful lot of
> optional components). But I'd expect the python build to fail if you explicitly specify to use something and
> it can't find what it needs from that something.

I don't know how often these things happen, that's why I would like to
know when they do. Specifying every optional dep is a start (which does
complicate/grow the nix expression already). But there are probably more
subtle things at play every now and then too.

>  
> But for the sake of agreement, let's just say you don't use weaker dependencies for environments where it's
> likely to cause problems (TBD). If the environment is deemed unlikely to cause these issues (e.g., optional
> components are explicitly specified, not autodetected; build fails when specified component doesn't provide
> expected interface) give it enough rope to hang itself. Each of the environments will have to be evaluated on
> its own merits.
>  
>
>     I agree on java. I think java builds a solid abstraction which really
>     shields the host system off from packages using it.
>
> +1
>  
>
>     But I don't believe the same can be said about any other environment.
>
> Well, I suspect that each environment (even Java) is going to land somewhere between "perfectly isolated" and
> "fully integrated to the host". It will boil down to whether the savings in time, CPU cycles, bandwidth and
> disk space is worth the extra human effort to root out and handle cases where isolation was violated. Even if
> the default for an environment is "rebuild on version bump", any individual package could be given the
> stronger dependency (always rebuild). The CPU cycles are then focused on code which has proven to break
> isolation. I am not preaching a one size fits all solution here.
>  
>
>     So, this would mean adding an exception for glibc (force trigger
>     rebuilds for weak deps when glibc is involved), or having maintainers
>     specify that their package might use ipv6, and flag glibc versions with
>     "ipv6 capable" so to trigger just those rebuilds. Sounds like a lot of
>     work to me, and probably stuff will slip through.
>
>  
> Instead of tracking down every exception, what about being able to manually force rebuilds for weak deps on a
> specific environment? Again, each environment needs to be evaluated on its own (de)merits, but being able to
> instigate a full rebuild of all python code at a whim may give the maintainers confidence to "try it and see
> how much of a problem it is". Smell a problem? Hit the panic button.

Sounds good to me. A way to force individual parts of the system to
rebuild, or a setting to always use strong dependency semantics.

Ideally, we'd have a way to find out if the resulting package is
different after a forced rebuild. That way it can be used every now and
then just to point out the ones that need some extra work when using a
"weak" dep.

>  
> For that matter, whether or not all the dependent code needs to be rebuilt when the environment is could be a
> property of the environment. Ruby could be "rebuild always". Java could be "rebuild on version bump". Python
> can be whatever it turns out that python needs to be. It can be changed as time goes on, experience is
> gained, and the maintainers get a feel for how much caution is actually necessary.
>  
> There are many reasonable impediments which could have led Eelco never to try installing each package into a
> separate content-addressable directory with a hash in the name. But he did try it, and it seems to work so
> far. Still, there's no guarantees that tomorrow's library format/linkers will be compatible with such a
> scheme, as it's not very common. Even without such a guarantee, it
> seems like a worthwhile strategy to pursue.

I agree, but the difference here is that nix itself leads to a lot of
advantages that just haven't been reached before. Pursuing a strange
goal to get something special/new/powerful if you succeed, sounds like a
worthwhile thing to do.

Your quest seems to not really lead to special/new/powerful properties,
just saves a few cpu cycles, while (if not used wisely) undoing some of
the properties of nix. If you really think it is worth it, sure, go on.
As I've said before, I'm not _against_ it in general, because I know
there are some pointless rebuilds, so there is something to gain. I'm
just skeptical about the means to know/prove which rebuild are
pointless, and about the "just try it by default, and if problems are
found we can fix these" idea.

I would rather see this introduced the other way around.
A way to compare derivations to find out if a rebuild was useful.
It would be nice to know how big the problem actually is. If these
statistics can show us something like "hydra is producing pointless
rebuilds 20% of the time", it might make people more enthusiastic about
weak dependencies. Also, these statistics can probably help to point out
which packages would benefit most from a weak dep. For example, if the
statistics show that (up till now) rebuilding java packages after
upgrading some library that jre depends on (X11 for example), never lead
to different results, it's probably a good idea to weak-dep there.

Mathijs

>  
> Bryce
>
>  
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
Url : http://lists.science.uu.nl/pipermail/nix-dev/attachments/20120713/88db572d/attachment.bin