[Nix-dev] fixed hash, fetchurl, should curl follow redirects, distribute diffs only?

[Marc Weber]

I'm working on adding rubyforge packages to nix.
To do so I've patched rubygems to add
gem nix package1 package2 ...

which generate

  # deps of package1, package2:

  # target packages 
  package1 = mkDerivation {
    name = ..
    buildInputs = [ .. ];
  }

  package2 = mkDerivation {
   ...
  }

Now it happens that the way to get the true source url of a gem is
querying http://gems.rubyforge.org/gems/#{spec.full_name}.gem
which returns a http 302 redirect which is not supported by
nix-prefetch-url / fetchurl

So I can make the gem nix command read the resulting url from that
redirect (actually I've done so) However this is bad because:
a) the same mirror will be used all the time
b) if the mirror goes down or stops beeing an "official mirror" we don't
  recognzie it.

So either gem nix should be run occasionally to update urls or the
fetchurl builder should support redirects.

However only because a mirror url changes this should not cause
rebuilds. (same if a fetch-url builder script changes).
Right now it does.

Eg put

  let pkgs = (import /etc/nixos/nixp) {};
  in pkgs.fetchurl {
    url = "http://mawercer.de?abc=5"; # dummy empty apache site which should not change
    sha256 = "8a8e1f7c28778246d25ede35ea17a8fafdb8e255b9df4f7b5cce0afad38e99e9";
  }

into test.nix and run nix-build test.nix
then change abc=5 by abc=6 and rerun. You'll get another store path.

Getting another store path in this case because the url or the builder
changes (eg by making curl follow redirects) is bad because *everything*
has to be recompiled which causes a lot of traffic. I don't care about
new store paths. But I do care about compilation time and internet
traffic cause that's slow here.

So what about either making nix set the out path to the sha256 value if
it's given as 'fixed' or adding kind of binary diff telling nix:
you can get /nix/store/aaaaaa-package by building
/nix/store/bbbb-package and replacing all occurences of
nix/store/A by /nix/storeB ?

If we added such kind of feature you could even remove spaces from
builders without causing much headache because only "substitute A by B"
would be transfered.

Marc Weber