[Nix-dev] garbage collection and nix-push

[Mathijs Kwik]

On Wed, Oct 3, 2012 at 3:45 PM, Eelco Dolstra
<eelco.dolstra at logicblox.com> wrote:
> Hi,
>
> On 03/10/12 01:23, Mathijs Kwik wrote:
>
>> The hardware I use for building my channel is not very powerful. It
>> does a good job of building derivations, but packing nar files with xz
>> takes ages.
>>
>> My channel is nothing special, It just builds the configurations for a
>> few home machines and laptops and some packages that hydra doesn't
>> build with the optional dependencies I like.
>>
>> By registering the outputs as gc roots, I made sure no extensive
>> rebuilds are performed when I don't change much.
>> However, after a garbage collector run, all nar outputs get removed,
>> leading to long push times on the next run.
>>
>> So for now, I just lowered the garbage-collection cronjob to once a week.
>>
>> Does anyone have a suggestion on how to work around this?
>
> I think nix-push should just check whether a narinfo/nar for a given store path
> already exist in the output directory, and if so, refrain from rebuilding them.

Won't help me.

I start out with an (versioned) empty output dir every time so that
won't help me much.
I have a post-push script that then hard links the outputs into my
main binary-cache.
After removing old outputs, I delete every file in my main
binary-cache that has no more hard links to it.
That way, my main binary-cache stays cleaned up.

But that's not the issue now. My issue is that garbage collection
removes the produced nars (that are in store), and rebuilds most of
them on the next nix-push.

But, as suggested by Michael, I can just create a derivation that
links to these in-store nars and add that derivation as gcroot (like a
nix-push profile).

I would like nix-push to do this (optionally or as default).
Then, as an option, nix-push can decide to copy (useful for hydra) or
symlink nars to the output dir.

The big advantage is that nix-push's outputs and the intermediate
store paths can then be managed and garbage collected as any other
profile.

>
> --
> Eelco Dolstra | LogicBlox, Inc. | http://nixos.org/~eelco/
> _______________________________________________
> nix-dev mailing list
> nix-dev at lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev