[Nix-dev] "live builds" proposal. The ultimate solution :)

[Evgeny Egorochkin]

On Sunday 01 August 2010 11:56:15 Eelco Dolstra wrote:
> Hi,
> 
> On 08/01/2010 01:20 AM, Yury G. Kudryashov wrote:
> > I propose the following extension to the nix language that will allow
> > native support for "live scm builds" etc.
> > 
> > If nix is called with --enable-exec, the new function builtins.exec is
> 
> > added. The proposed syntax is like the following:
> This is already kind of possible (if I understand correctly what you want).
>  For instance, to obtain the current head revision of a SVN repository:
> 
> rev = import (pkgs.runCommand "head-revision.nix"
>   { buildInputs = [ pkgs.subversion ];
>     dummy = builtins.currentTime;
>   }
>   ''
>     rev=$(echo p | svn ls -v --depth empty https://svn.nixos.org/repos/nix
> | awk '{ print $1 }')
>     echo "[ $rev ]" > $out
>   '');
> 
> So this imports a Nix expression generated by a derivation ("import
> (runCommand ...)").  The real trick is "dummy = builtins.currentTime;",
> i.e. passing the current time as an input to the derivation, to ensure
> that it gets rebuilt every time you evaluate the expression.
> 
> Needless to say that this is undocumented and unsupported ;-)

This was the missing bit I was looking for!

The patch below this email is an actual build that lets you override:

  * the whole source, eg if you want to build your local checkout 
with(uncommitted) changes

* repository, say if you want to build another developer's work

* revision, eg if it contains the fix you need or trunk doesn't build

* branch, which is one of the key features. If you don't specify revision, 
this build will track the branch you want.

Most importantly, all this is a pure nixy solution and it properly caches 
everything that can be cached.

If you provide a specific revision, it works like a regular build, caching the 
specific repository checkout.

if you use branch tracking, on every rebuild the remote repository is queried 
for the last revision available, and if it's the same as already checked out 
and built, nothing else happens, otherwise the package is rebuilt.

This functionality eclipses Gentoo's implementation, becase in gentoo you 
can't:
 * pick a branch
 * pick a revision
 * automatically rebuild when branch changes.

All gentoo does is rebuild the current trunk at your request.

Caveats:

Of couse different branches can have different dependencies and may require 
some hacks to compile. My sample package is a good example of this. Trunk 
doesn't compile because devs introduced an extra dep in a really stupid way( 
as a git submodule) instead of requiring it as a regular dep.

This is the reason why in the above list of advantages compared to gentoo I 
decided to not mention that gentoo requires a separate build for each branch 
if you want to support branches, because in nixos we might end up with branch-
specific hacks or several builds each good for similar branches. This i still 
better than being forced to create/edit a build for each branch(especially if 
you want to track some obscure feature branch), but not as good as a single 
live package which can't possibly exist until people start providing .nix 
files along with makefiles.

Implementation:

I expect that we wrap thehorrible expression for rev default in a convenience 
function similar to fetchgit... say fetchgitrevision. Of course I expect this 
list to come up with a better name and implementations for their favorite SCM, 
since right now we only have git and svn covered.

P.S. or Extra tweaks:

I just realized that it probably makes sense to write live builds like this:
{..., extra}:
stdenv.mkDerivation (rec {
} // extra)

This way you can also add some patches and buildInputs and config options 
quickly via overrides.

-- 
Evgeny


diff --git a/pkgs/development/tools/misc/openocd/live.nix 
b/pkgs/development/tools/misc/openocd/live.nix
new file mode 100644
index 0000000..830a9ce
--- /dev/null
+++ b/pkgs/development/tools/misc/openocd/live.nix
@@ -0,0 +1,48 @@
+{stdenv, fetchgit, git, runCommand
+, branch ? "heads/master"
+, repository ? "git://openocd.git.sourceforge.net/gitroot/openocd/openocd"
+, rev ? import (runCommand "head-revision"
+  { buildInputs = [ git ];
+    dummy = builtins.currentTime;
+  }
+  ''
+    rev=$(git ls-remote ${repository} | grep "refs/${branch}$" | awk '{ print 
$1 }')
+    echo "[ \"$rev\" ]" > $out
+    echo Latest revision in ${branch} is $rev
+  '')
+, src ? fetchgit {
+    url = repository;
+    rev = rev;
+  }
+, autoconf, automake, libtool, libftdi, texinfo }:
+
+stdenv.mkDerivation {
+  name = "openocd-live";
+
+  inherit src;
+
+  configureFlags = [ "--enable-ft2232_libftdi" "--disable-werror" "--enable-
maintainer-mode" ];
+
+  buildInputs = [ libftdi autoconf automake libtool texinfo ];
+
+#  preConfigure = "autoreconf -vfi";
+  preConfigure = "./bootstrap";
+
+  meta = {
+    homepage = http://openocd.berlios.de;
+    description = "OpenOCD, an on-chip debugger";
+
+    longDescription =
+      '' OpenOCD provides on-chip programming and debugging support with a
+         layered architecture of JTAG interface and TAP support, debug target
+         support (e.g. ARM, MIPS), and flash chip drivers (e.g. CFI, NAND,
+         etc.).  Several network interfaces are available for interactiving
+         with OpenOCD: HTTP, telnet, TCL, and GDB.  The GDB server enables
+         OpenOCD to function as a "remote target" for source-level debugging
+         of embedded systems using the GNU GDB program.
+      '';
+
+    license = "GPLv2+";
+    maintainers = with stdenv.lib.maintainers; [viric];
+  };
+}
diff --git a/pkgs/top-level/all-packages.nix b/pkgs/top-level/all-packages.nix
index 9dc4e28..05f05b5 100644
--- a/pkgs/top-level/all-packages.nix
+++ b/pkgs/top-level/all-packages.nix
@@ -2534,9 +2534,10 @@ let
 
   omake = callPackage ../development/tools/ocaml/omake { };
 
-
   openocd = callPackage ../development/tools/misc/openocd { };
 
+  openocd_live = callPackage ../development/tools/misc/openocd/live.nix { };
+
   oprofile = import ../development/tools/profiling/oprofile {
     inherit fetchurl stdenv binutils popt makeWrapper gawk which gnugrep;