[Nix-dev] extending release-lib and how to cache in nix

[Florian Friesdorf]

Hi,

yesterday Cillian and me continued work on the python-rework jobset to
limit hydra to only packages depending on python. We extended
nixpkgs/pkgs/top-level/release-lib.nix [1]:

- condPackagesWithMetaPlatform supports a list of valid platforms which
  is intersected with a package's meta.platforms

- condPackagesWithMetaPlatform supports a filter predicate

- packagesWithMetaPlatform is based on condPackagesWithMetaPlatform and
  shows the original behaviour

- dependsOn predicate recursively scans a package's *Inputs against a
  list of dependencies

- dependsOnOrIs checks whether a package is a dependency and delegates
  further to dependsOn if not


$ cd dev/nixos/nixpkgs


hydra's current python-rework only limits platform, no dependsOn yet as
the evaluation takes roughly 4mins on i7 with 2.13GHz. We limit the
platform to x86_64-linux until satisfied and then gradually extend to
i686-linux, darwin and others. Limiting via release.nix is quicker than
reconfiguring hydra.

$ nix-instantiate --eval-only pkgs/top-level/release-python.nix [2]


A very small jobset as proof-of-concept for dependsOn:

$ nix-instantiate --eval-only pkgs/top-level/release-python-test.nix [3]


The planned jobset for python-rework, in case we can clearly reduce the
evaluation time (caching):

$ nix-instantiate --eval-only pkgs/top-level/release-python-only.nix [4]


Caching
-------

Current idea, based on [5]: Instead of just returning a boolen
(depends/does not depend) we return an attr set:

{ depends = BOOLEAN; 
  cache = { yes = LIST_OF_PKGS_THAT_DO; 
            no = LIST_OF_PKGS_THAT_DONT; }}

And whenever information about a new package is gained we create a new
attr set with updated information.

The patch implementing this does not work so far and needs cleanup.

Before wasting time on this:

- Do you have a better idea?

- Is there some code implementing state I could look at?


@Eelco: You said on IRC that nix used to cache function calls with
identical arguments. What makes this concept unsuitable for nix?


regards
florian

[1] https://github.com/chaoflow/nixpkgs/blob/python/pkgs/top-level/release-lib.nix#L85
[2] https://github.com/chaoflow/nixpkgs/blob/python/pkgs/top-level/release-python.nix
[3] https://github.com/chaoflow/nixpkgs/blob/python/pkgs/top-level/release-python-test.nix
[4] https://github.com/chaoflow/nixpkgs/blob/python/pkgs/top-level/release-python-only.nix
[5] http://learnyouahaskell.com/for-a-few-monads-more#state

-- 
Florian Friesdorf <flo at chaoflow.net>
  GPG FPR: 7A13 5EEE 1421 9FC2 108D  BAAF 38F8 99A3 0C45 F083
Jabber/XMPP: flo at chaoflow.net
IRC: chaoflow on freenode,ircnet,blafasel,OFTC
-------------- next part --------------
A non-text attachment was scrubbed...
Name: not available
Type: application/pgp-signature
Size: 835 bytes
Desc: not available
Url : http://lists.science.uu.nl/pipermail/nix-dev/attachments/20120724/ba6171da/attachment.bin