[Nix-dev] Dataflow

[Bardur Arantsson]

On 2014-12-25 17:13, stewart mackenzie wrote:
> Hello Moritz,
> 
> On Thu, Dec 25, 2014 at 10:39 PM, Moritz Ulrich <moritz at tarn-vedra.de> wrote:
>>
>> stewart mackenzie <setori88 at gmail.com> writes:
>>
>>> So I've packaged together a number of erlang dependencies together,
>>> then compiled them a number of times in a row.
>>>
>>> I noticed that the error outputs were different each time.
>>>
>>> To me this means nix isn't deterministic.
>>
>> Have you tried tracking down WHY the builds seem to be
>> non-deterministic? What does "different error output" mean? Are the
>> builds parallel (make -j etc.)?
> 
> Ah yes, rebar builds in parallel. That's it.
> 
>>> I'd like to discuss the inclusion of the dataflow language concept into nix.
>>>
>>> By including this concept the builds become deterministic, one could
>>> go a step further and introduce light weight language level threads.
>>> Should unbound variable Z in thread X become bound on thread Y then
>>> execution of thread X continues.
>>
>> I don't see how that would help with deterministic builds. Care to explain?
> 
> Declarative Concurrency is deterministic
> https://en.wikipedia.org/wiki/Oz_%28programming_language%29#Dataflow_variables_and_declarative_concurrency
> 
> One could spawn off many hundreds/thousands of build threads and
> building in parallel, if one thread has an un-built dependency waiting
> on another thread it politely waits. Then continues as per normal once
> the dependency is built.
> 
> In other words, the nix language has some decent concepts, but by
> including the declarative concurrent paradigm which consists of
> immutable dataflow variables and light weight threads quite possibly
> the implementation of the nix language could be simplified a lot.
> 

Maybe I'm wrong, but I don't think the Nix language itself has any idea
of concurrency as such -- at the manual says

   "The Nix expression language is a pure, lazy, functional language."

(https://nixos.org/nix/manual/#ch-expression-language)

Therefore, there's no value in adding concurrency unless your bottleneck
is the actual derivation of the build *plan* (not the actual build).

As you've discovered, your problem is that the underlying mechansism is
impure (and buggy). There's nothing Nix-the-language can do about that
-- unless you want to go about replacing the build systems used by
individual programs by more principled ones. I'm sure we'd all love
that, but it's not a trivial task and need buy-in from any potential
upstream.

Regards,