[Nix-dev] Incremental recompilation

[Mikhail Volkhov]

Hi,

I'm working at Serokell and I was suggested to solve the problem of
incremental recompilation generic framework absence as (a part of?) my
coursework in this term (university) that would be helpful for our
company as well.

Actually, it's not a real problem for most users, but for those who
build packages without hydra it might be helpful. It's also a great
thing for those who develop with nix and need to build the project
frequently (another approaches like stack-nix for haskell are of course
available, but pure nix solution has its own advantages and may be
preferable).

So while researching I discovered that:
1. Nobody has really succeeded in making stable generic framework for
this task -- all solutions are ad-hoc and language/build-tool specific
2. Ad-hoc solutions are mostly based on exploiting the nix itself --
writing a package that somehow captures its previous build states and
uses them.

My proposal is very raw and may appear completely nonviable. I'm writing
this to hear any comments about the idea, other approaches and get any
help that could be provided.

So, here the sketch:
1. Allow adding some extra field (let's call it `cache`) to derivation
that would be resemble out fields but with specific target -- storing a
folder in nixstore with builds (both succeeded and not) for this
derivation. This mechanism should be optional and so some hackery around
derivation is supposed to be done. Adding extra flags to store shouldn't
be hard too.
2. Write a smart framework in ~stdenv/generic/*~ for specifying behavior
on using last caches. Framework isn't supposed to know anything about
the state of builds in the folder generated by `cache` (from user point
of view), so one should just specify how to "reuse" caches from previous
build.
3. All failing and successful builds are going into `cache` folder in
nixstore. Some other piece of code (I'm not sure whether it should be in
nix itself or in nixpkgs) will, if requested, give those caches to
builder and he will reuse it somehow (with generic builder in most cases
obviously). Caches will have some kind of versions that would allow us
to build a tree from them -- given cache A and builder X it would
possibly modify A to state B. So we then put B as a child of B into
`cache` folder. So when tree of builds is constructed we can adopt the
following build strategy: choose some subset of leaves (based on some
heuristics I suppose), then just run builds, if any failed, put its
parent into set, and so do it until you reach a root -- the most raw
cache. Basically a leaves-first bfs-like traversal. If root fails to
build, build the derivation from scratch and put its result into `cache`
dir as another root.
4. We can also add some flag permitting to build project from source
directly when caches are available. Consider a situation when you're
building a project with estimated time 10h and you know that most of the
time it can fail because, for example, you applied some patch and not
sure if it will work. Then just fixing it after build fail and
running build from the previous cache would be a great idea.
5. Derivations in a particular cache tree could be stored by specifying
the diff with derivation's parent to save space. I'm not considering the
lack of space in nixstore as a problem though (for simplicity).

-- 
Volkhov Mikhail
M3#38 ITMO study group 17'
Computer Technologies Department