[Nix-dev] How to download all sources for all of nixpkgs?

[Bjørn Forsman]

Hi all Nixers,

I would like to use Nix on an air gapped network. For that I need to
download all sources referred by nixpkgs.

All attempts so far have failed. You can read about it below my
signature. Any suggestions are welcome.

Wish list item N+1:
  * "nix fetch-sources --recursive $expr"

Best regards,
Bjørn Forsman

1.

Adapted the command line from
https://nixos.org/wiki/Download_all_sources. That page shows how to
download for a small subset of nixpkgs, I need "all" of it.

$ nix-store -r $(grep -l outputHash $(nix-store -qR $(nix-instantiate
'<nixpkgs>') | grep '.drv$'))
error: Package ‘Agda-Sheaves-8a06162a8f0f7df308458db91d720cf8f7345d69’
in ‘/nix/store/0dwbbd9cm5vnb0l2qj90i14r036f2c42-nixos-16.09.1821.0cb2838/nixos/pkgs/development/libraries/agda/Agda-Sheaves/default.nix:18’
is marked as broken, refusing to evaluate.
a) For `nixos-rebuild` you can set
  { nixpkgs.config.allowBroken = true; }
in configuration.nix to override this.

b) For `nix-env`, `nix-build` or any other Nix command you can add
  { allowBroken = true; }
to ~/.nixpkgs/config.nix.

At this point it just hangs. Running the only the inner command,
"nix-instantiate '<nixpkgs>'", it becomes clear that after the above
error is printed, this command exits. So the composed command line
will never produce what I want.

Trying harder:

$ NIXPKGS_ALLOW_BROKEN=1 NIXPKGS_ALLOW_UNFREE=1 nix-instantiate '<nixpkgs>'
error: assertion failed at
/nix/store/0dwbbd9cm5vnb0l2qj90i14r036f2c42-nixos-16.09.1821.0cb2838/nixos/pkgs/top-level/all-packages.nix:6177:19

This is the failing assert:

$ sed -n 6177p /nix/store/0dwbbd9cm5vnb0l2qj90i14r036f2c42-nixos-16.09.1821.0cb2838/nixos/pkgs/top-level/all-packages.nix
  binutilsCross = assert crossSystem != null; lowPrio (forceNativeDrv (

So yeah, that doesn't seem like a way forward...


2.

Using find-tarballs.nix:

$ nix-instantiate --eval --json --strict
./maintainers/scripts/find-tarballs.nix --arg expr "(import ./. {})"
--show-trace
error: while evaluating anonymous function at
/home/bfo/nixpkgs/maintainers/scripts/find-tarballs.nix:6:1, called
from undefined position:
while evaluating ‘derivationsIn'’ at
/home/bfo/nixpkgs/maintainers/scripts/find-tarballs.nix:30:20, called
from /home/bfo/nixpkgs/maintainers/scripts/find-tarballs.nix:26:28:
while evaluating anonymous function at
/home/bfo/nixpkgs/lib/attrsets.nix:224:10, called from undefined
position:
while evaluating anonymous function at
/home/bfo/nixpkgs/maintainers/scripts/find-tarballs.nix:34:60, called
from /home/bfo/nixpkgs/lib/attrsets.nix:224:16:
while evaluating ‘derivationsIn'’ at
/home/bfo/nixpkgs/maintainers/scripts/find-tarballs.nix:30:20, called
from /home/bfo/nixpkgs/maintainers/scripts/find-tarballs.nix:34:63:
while evaluating anonymous function at
/home/bfo/nixpkgs/lib/attrsets.nix:224:10, called from undefined
position:
while evaluating anonymous function at
/home/bfo/nixpkgs/maintainers/scripts/find-tarballs.nix:34:60, called
from /home/bfo/nixpkgs/lib/attrsets.nix:224:16:
while evaluating ‘derivationsIn'’ at
/home/bfo/nixpkgs/maintainers/scripts/find-tarballs.nix:30:20, called
from /home/bfo/nixpkgs/maintainers/scripts/find-tarballs.nix:34:63:
while evaluating ‘canEval’ at
/home/bfo/nixpkgs/maintainers/scripts/find-tarballs.nix:48:13, called
from /home/bfo/nixpkgs/maintainers/scripts/find-tarballs.nix:31:9:
while evaluating the attribute ‘alice’ at
/home/bfo/nixpkgs/pkgs/development/beam-modules/hex-packages.nix:1579:5:
while evaluating ‘callPackageWith’ at
/home/bfo/nixpkgs/lib/customisation.nix:95:35, called from
/home/bfo/nixpkgs/pkgs/development/beam-modules/hex-packages.nix:1548:19:
while evaluating ‘makeOverridable’ at
/home/bfo/nixpkgs/lib/customisation.nix:54:24, called from
/home/bfo/nixpkgs/lib/customisation.nix:99:8:
anonymous function at
/home/bfo/nixpkgs/pkgs/development/beam-modules/hex-packages.nix:1550:9
called without required argument ‘redix_0_3_6’, at
/home/bfo/nixpkgs/lib/customisation.nix:56:12

Aborts early on / doesn't finish.


3.

(Failed attempts at) using all-tarballs.nix:

$ nix-instantiate ./maintainers/scripts/all-tarballs.nix
error: value is a set while a Boolean was expected

$ (export NIXPKGS_ALLOW_UNFREE=1 NIXPKGS_ALLOW_INSECURE=1
NIXPKGS_ALLOW_BROKEN=1; nix-store -r $(grep -l outputHash $(nix-store
-qR $(nix-instantiate --eval --json --strict
./maintainers/scripts/all-tarballs.nix))))
error: assertion failed at
/home/bfo/nixpkgs/pkgs/os-specific/linux/dpdk/default.nix:3:1
(use ‘--show-trace’ to show detailed location information)
bash: /run/current-system/sw/bin/nix-store: Argument list too long


4.

I changed ./maintainers/scripts/copy-tarballs.pl slightly to remove
uploading to AWS/S3. That gave me all all but the git repos. First
issue is that copy-tarballs.pl excludes non HTTP/FTP/.. URLs. The
second issue is that the full list of URLs it iterates over comes from
find-tarballs.nix, and that expression seems to not include git URLs
anyway! I don't grok find-tarballs.nix enough to modify it :-/