[Nix-dev] Security channel proposal

[Wout Mertens]

On Thu, Sep 25, 2014 at 6:33 PM, Michael Raskin <7c6f434c at mail.ru> wrote:

> >> I bet against our package set being buildable in 2 hours — because of
> >> time-critical path likely hitting some non-parallelizable package.
> >>
> >
> >I think most large projects can be compiled via distcc, which means that
> >all you need is parallel make.
>
> WebKitGTK… (there is a comment about failure to make it work with
> parallel build)
>

... https://trac.webkit.org/wiki/WebKitGTK/SpeedUpBuild#distcc

>Libreoffice build is inherently a single-machine task, so to speed it
> >> up you need something like two octocore CPUs in the box.
> >>
> >
> >Point in case:
> >
> https://wiki.documentfoundation.org/Development/BuildingOnLinux#distcc_.2F_Icecream
> >. Building with "icecream" defaults to 10 parallel builds.
> >
> >Also, with ccache the original build time of 1.5 hours (no java/epm) is
> >reduced to 10 minutes on subsequent runs.
>
> How would ccache cache be managed for that? How it would work with
> rented instances being network-distant from each other?
>

Perhaps with persistent block storage that gets re-attached when an
instance is spun up, or by using a central NFS server:
https://ccache.samba.org/manual.html#_sharing_a_cache

>> With such a goal, we would need to recheck all the dependency paths and
> >optimise the bottlenecks.
> >Sounds good :)
>
> We have too little manpower for timely processing of pull requests.
> I think that starting a huge project should be done with full knowledge
> that it can fail just because it needs too much energy.
>

I would start by autogenerating sensible metrics, which could then lead to
incremental improvements as people tackle packages one by one. For example,
perhaps the total number of dependencies (all depths) multiplied by the
compile time of the package.


> >Maybe making dependency replacement work reliably (symlinking into
> >> a special directory and referring to this directory?) is more feasible…
> >
> >Can you elaborate?
>
> One of the bruteforce ways is just to declare «we need reliable global
> dependency rewriting». In this case we could just have a symlink for
> every package ever used as dependency so a replacement would mean
> destructively changing this symlink.
>
> I.e. you depend on /nix/store/aaa-bash, there is a symlink
> /nix/store/aab-bash to /nix/store/aaa-bash, and the builder sees just
> /nix/store/aab-bash.


Perhaps this could be done by nix-store instead, just provide a list of
replacement packages and it will move the original packages away and
symlink the replacements in their place. We could also prototype this with
a pretty simple script.

So almost what you're saying, except that the /nix/store/aaa-bash gets
moved to /nix/store/OLD-aaa-bash and /nix/store/aaa-bash becomes a symlink
to aab-bash. No pre-determining of dependency rewrite targets.

Wout.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.science.uu.nl/pipermail/nix-dev/attachments/20140925/e8358407/attachment-0001.html