[Nix-dev] Dataflow

[stewart mackenzie]

Hello Moritz,

On Thu, Dec 25, 2014 at 10:39 PM, Moritz Ulrich <moritz at tarn-vedra.de> wrote:
>
> stewart mackenzie <setori88 at gmail.com> writes:
>
>> So I've packaged together a number of erlang dependencies together,
>> then compiled them a number of times in a row.
>>
>> I noticed that the error outputs were different each time.
>>
>> To me this means nix isn't deterministic.
>
> Have you tried tracking down WHY the builds seem to be
> non-deterministic? What does "different error output" mean? Are the
> builds parallel (make -j etc.)?

Ah yes, rebar builds in parallel. That's it.

>> I'd like to discuss the inclusion of the dataflow language concept into nix.
>>
>> By including this concept the builds become deterministic, one could
>> go a step further and introduce light weight language level threads.
>> Should unbound variable Z in thread X become bound on thread Y then
>> execution of thread X continues.
>
> I don't see how that would help with deterministic builds. Care to explain?

Declarative Concurrency is deterministic
https://en.wikipedia.org/wiki/Oz_%28programming_language%29#Dataflow_variables_and_declarative_concurrency

One could spawn off many hundreds/thousands of build threads and
building in parallel, if one thread has an un-built dependency waiting
on another thread it politely waits. Then continues as per normal once
the dependency is built.

In other words, the nix language has some decent concepts, but by
including the declarative concurrent paradigm which consists of
immutable dataflow variables and light weight threads quite possibly
the implementation of the nix language could be simplified a lot.

>> This makes nix deterministic.
>
> Why are you under the impression that nix isn't? (Also, are you talking
> about the language nix or the package manager nix?)

I meant the language. James gave me a good understanding.
Non-determinism is present at build time, then removed thereafter.
Thanks James.

>> Great efforts have been made to refining the nix way, ie reproducible
>> builds. This is achieved by downloading dependencies first then making
>> them immutable. This means we're half way there as the immutable
>> nature of these packages is critical for the implementation of
>> dataflow.
>
> The packages aren't just downloaded - this is just a substitute. You'll
> get the same results when building that specific package locally.

No, strangely it didn't for me. Maybe I'm doing something weird.
Okay I tried to build github.com/headcounter/deployment but failed
then went to https://headcounter.org/hydra/build/467310 and ran the
"reproduce locally" build :
$ bash <(curl https://headcounter.org/hydra/build/467310/reproduce)

Interestingly enough it built the manual locally with the above link
but failed when I tried the cloned version.

Anyway why I said nix isn't deterministic: if you run nix-build
against https://github.com/sjmackenzie/nix-ejabberd many times in a
row you'll get different error messages.

I'm putting this down to rebar's parallel build. So the above case
doesn't hold water proving that nix is not deterministic. Though there
seems to be a leaky abstraction in that error messages are not
reproducible. I suppose this is unavoidable with parallel build tools.

Having said that I think there might be something interesting w.r.t
the introduction of declarative concurrency into the nix language.