[Nix-dev] Re: [Nix-commits] SVN commit: nix - 16296 [..] description of sourceByName, bleedingEdgeRepos ...

[Marc Weber]

> Please capitalise the description field.
> This description doesn't seem to match the package.
fixed, I'll commit this ASAP.
 
> What does "sourceByName" do?
> (And what are these "infos" anyway?)

short description:
==================

sourceByName, run-nix-repository-manager-with-config,
bleedingEdgeRepos make it easy to maintain a set snapshot sources
providing differential updates which is fastest.


long description:
==================

Sometimes you don't want official releases but current development code.
How can you put these sources into nixpks? There is fetchgit, fetchcvs,
fetchsvn, fetchdarcs fetchhg (?) .. However they are all bad. Why?
Because they start git clone, darcs pull, .. over and over again. What
a waste of time and network traffic only to get some small patches.

So I've written a system to automate these tasks:
  1) fetch sources
  2) create snapshot tarball
  3) distribute that tarball

HOWTO:

a) define a repository description in config.nix or in
  development/misc/bleeding-edge-repos/default.nix
  This looks like this:

       ghc_core_array = { type = "darcs"; url=http://darcs.haskell.org/packages/array; groups="ghc_group ghc_core_libs"; };
       ghc_core_base = { type = "darcs"; url=http://darcs.haskell.org/packages/base; groups="ghc_group ghc_core_libs"; };
       ghc_core_bytestring = { type = "darcs"; url=http://darcs.haskell.org/packages/bytestring; groups="ghc_group ghc_core_libs"; };
       ghc_core_Cabal = { type="darcs"; url=http://darcs.haskell.org/packages/Cabal;  groups="ghc_group ghc_core_libs"; };

b) update the nix-repository-manager tool so that it knows about the new
   repositories locations and ids (the id is the attr name)

   $ nix-env -A haskellPackages.nixRepositoryManager

c) fetch sources into a cache dir ~/managed-repos
   and create a dist file (which will be used by nix)

   $ nix-env $NIXPKGS --update ghc_core_array
   # get a group:
   $ nix-env $NIXPKGS --update ghc_group

  I used the distfile for these reasons:
  - I feared nix not recognizing changes in a directory
  - .svn .git _darcs files can be stripped of
    sanity checks such as git status can be run
    printing warnings about local modifications
  - it can be distributed (uploaded) so that others can just get the
    dist file and are done. Eg darcs needs a lot of time to get
    sources so darcs fetch --lazy is not an option for me.

d) tell nix to use your snapshot:

  stdenv.mkDerivation {
    [...]
    src = sourceByName "repo-id";
  ];


The pkgs/misc/bleeding-edge-fetch-infos.nix beeing updated by c) tells nix where to find 
a distributed snapshot. However the local dist file is preferred by default.

{ # Warning, this file is updated automatically by nix-repository-manager
  # which will add or replace exsiting attribute sets only
  # new items will be added before the last line, 4 lines will be
  # removed when
  # replacing always, matched by "name ="

  cabal = args: with args; fetchurl { # Thu Aug 21 09:07:56 UTC 2008
    url = http://mawercer.de/~nix/repos/cabal.tar.gz;
    sha256 = "2c370636ef30593325b1c01eed37eb0e993923acb6387724d97a5eed51b00887";
  };


use cases (2nd example)
=======================

A use case is mantaining ghc or xorg snapshots.
GHC requires updating about 30 packages to compile,
xorg requires much more. My list of repo descriptions is about 280
lines but not all of them were used.

So you're done with
  run-nix-repository-manager-with-config $NIXPKGS --update xorg
  # wait ..
  nixos-rebuild test

Now if everything is fine:
  # uload dist files
  run-nix-repository-manager-with-config --publish xorg
  # commit to nixpkgs (branch)
  svn commit -m 'new xorg snapshot' # (comitting new infos)

*plonk* snapshot failed 
=======================
If it doesn't work just rollback the info file (svn reset)
and continue using the distfiles which you uploaded to a server
previously.

The --update command does more: it also creates
~/managed-repos/*.log files telling you what exactly has changed:

You'll get log looking like this (darcs)
  nrmtag7 # Fri Apr  3 16:20:51 CEST 2009
  nrmtag7 # Fri Apr  3 16:42:58 CEST 2009

or this (git)
  4a4fe6997847a2cc22d6a4ccd1f2f2265d28520d # Sun Mar 15 13:05:25 CET 2009
  4a4fe6997847a2cc22d6a4ccd1f2f2265d28520d # Wed Mar 18 05:02:55 CET 2009

You could run gitk 4a4f..4a4f which will tell that everything was up
to date in this case. This is not necessary for git because it has a
reflog. But it's useful for darcs and other vcs systems.

If you still think you need more information.. add a wiki so that I can
explain this once and forever or reply :-)

Sincerly
Marc Weber