[Nix-dev] Incremental recompilation

[Ericson, John]

Nicolas is correct. One complication however one is that one has to be
careful when files contain relative paths. The cleanest treatment of the
recursive hashing would normalize any nix store path within the source
location to hash of just that subtree, but that breaks the relative paths.
Note that its not always possible to resolve all relative paths while
hashing either, as they may form cycles.

Back to Volkhov's original point, I think that compilers should work like
Nix itself -- explicit graph of computation all data is content-addressed
and all function application is cached on the hashes of the arguments (and
function itself). Its a lot of work to refactor compilers to work this way
of course, but practicalities aside I think that's the cleanest way to go.

To cache across derivations, let's say, the compiler and Nix should be able
to use the same content-address store---that's half the battle. The other
half is the compiler (also like Nix) needs to map function applications to
their evaluations. As I said above the arguments are already
content-addressed, but the functions need to be given some sort of abstract
names. Perhaps indices prefixed with the derivation path would do, and then
Nix and the compiler can share the same store for their evaluation maps too.

As it turns out, Rust is exploring incremental compilation
https://github.com/rust-lang/rfcs/blob/master/text/1298-incremental-compilation.md
and their plan is, well, kinda a first approximation of the above. I'd be
really interested in trying to put this plan into action with rustc---this
is our best shot given the feasibility of refactoring GCC, LLVM, GHC, etc
on our own.

John


On Sat, Apr 30, 2016 at 8:39 AM, Nicolas Pierron <
nicolas.b.pierron at gmail.com> wrote:

> On Fri, Apr 29, 2016 at 7:56 PM, stewart mackenzie <setori88 at gmail.com>
> wrote:
> > IMHO, incremental compilation is a strong boon for nix adoption. It is
> the
> > only serious issue I can see which prevents Nix being a Make replacement.
>
> I agree with you on this point, Nix has the potential of been a better
> ccache & distcc in one shot.
>
> Note, this is not the same issue as mention in the Volkhov email.  The
> email suggest caching artifacts of the compilation spawn by Nix and
> independently of the build system, while doing a make replacement
> involves replacing/wrapping existing build system and using Nix for
> caching artifacts of the compilation spawned in user-land.  But, I
> guess the second problem's solution could be a mean for solving the
> first problem.
>
> Also, I admit I thought on how to solve this issue in the first place,
> but the main problem I got is not even building, this is only about
> making a copy of the source directory into the nix store.
> When your project sources are larger than 1 GB (without the VCS), you
> don't want to be reading all the bytes before being able to start any
> compilation.
>
> Thus, none of the approaches proposed before are practical for larger
> projects.
> So, how can we make it such that it will scale?
>
> The only approach I can think of would be to:
>  - Upload each individual file to the Nix store, when they would be needed.
>  - Make a symlink fram to re-build fake source directories.
>  - Use hash-by-content to prevent recompilation.  (when using  "gcc
> -M"  to filter out directory content, you want to make sure you get
> the same set of headers, and only cause a recompilation if some of the
> header changed)
>
> --
> Nicolas Pierron
> http://www.linkedin.com/in/nicolasbpierron - http://nbp.name/
> _______________________________________________
> nix-dev mailing list
> nix-dev at lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.science.uu.nl/pipermail/nix-dev/attachments/20160430/cee46b2a/attachment.html