[Nix-dev] Ideas for a NixOS-related bachelors thesis?

[Ericson, John]

On Mon, Nov 2, 2015 at 12:06 AM, Vladimír Čunát <vcunat at gmail.com> wrote:

> On 11/01/2015 10:16 PM, Ericson, John wrote:
>
>> If I recall Eelco's thesis correctly, the derivation is built in a
>> temporary location, the build is hashed, and then the binary is mangled
>> so the temp path is replaced with the proper /path/to/store/hash/ path.
>> Well if we just stick the data in IPFS before the path replacement, and
>> let the other side do the mangling we dodge that problem.
>
> I don't think that would work, as hash rewriting *changes* the standard
> hash of the path (self-referencing problem).
>

In both my plan and the original, the hash for the data is calculated
before the mangling. In my plan the metadata object would contain the
hash-reference to the actual data, and the dummy path [unless we can fix
the dummy path for all builds using sand-boxing, so other side knows that a
priori]. Then other side both knows the dummy path and the hash they
replace it with. Of course after the mangling the hash no longer reflects
the data, but that's alright, as content-addressing is most important for
getting things across the network safely.

Source mirroring is a thing completely independent of nix(os), so it might
> have much larger chance of being successful, as P2P IMHO pays off only when
> you get many clients.


I thought it completely independent too. But then I released we could add
an option to re-host all fixed-output builds with IPFS. Still, even with
that enhancement I find this a less-interesting special case of the above.

Nitpicks: you all seem to use "intentional" instead of "intensional", but
> the name doesn't really matter, which is incidentally the only thing
> that would much change during ex->in switch. I don't really get why people
> call for it so much. The only useful (possible) consequence I can
> think of is that when we e.g. update some build tool, the builds using
> this tool would likely result into the same path, so the rebuild would
> *not* cascade like it does now. It would only cut non-runtime indirect
> dependencies, which isn't really so much, and we would have to deal
> somehow with the equivalence classes allowing such "sharing".


Oops, sorry about messing up the name.

I agree that better caching on transitive deps alone is not the most
interesting thing. I'm mainly interested in getting better distributed
building and distributed storage for Nix, and a distributed content
addressable store (I have no special interest in IPFS in particular) +
intensional store seems the best way to do this. Let my try to motivate
this:

As other have pointed out, we already use content-addressing for binary
substitutions. The biggest change that the intensional store therefore
introduces is keeping track of what *builds* of dependencies are used, and
multiple drv->build maps. But when substituting the builds of others, these
provide crucial information:

   - Correctness: our build vs their build of some dependency may
   (unfortunately) not be compatible.
   - Space usage: don't want many rebuilds of the same derivation lying
   around, so nice to audit what a build is depending on before downloading it.
   - Security: suppose one trusts another to build some tiny little
   utility, but would rather the critical library it uses (e.g. OpenSSL) come
   from cache.nixos.org
   - Security 2: suppose someone looses trust in another builder; now they
   need to know what builds to purge. Note that even local builds that only
   depend at compile time on the now-untrusted builds should be purged (think
   Ken Tompson compiler attack).

Keeping track of the information needed for these seems tantamount to
implementing the intentional store.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.science.uu.nl/pipermail/nix-dev/attachments/20151102/6ae992d3/attachment-0001.html