[Nix-dev] nix-env and nixpkgs usability improvements

[Eelco Dolstra]

Hi,

I'm thinking about making a number of changes to Nixpkgs and nix-env
to make them more user- (and developer-) friendly.  There are
currently a number of problems:

- The situation in the pkgs/system directory is quite confusing: we
  have quite a few files there involved in defining the "top level"
  available packages (all-packages-generic.nix, all-packages.nix,
  i686-linux.nix and friends, populate-cache.nix).

- It's bad that we have two sets of "available" packages: those in
  i686-linux.nix (which is in turn an instantiation of
  all-packages-generic.nix), and populate-cache.nix (which contains a
  subset of the former, namely the pre-built binary packages in the
  channel).

- Even worse is that "nix-env -qa" doesn't work on (for instance)
  powerpc-darwin.nix, since many packages have assertions of the form
  'assert system == "i686-linux"'.  This causes evaluation to fail on
  all other platforms.  Right now i686-linux is the only platform
  where all-packages-generic.nix evaluates entirely.

- For people who use a channel, it is hard to install packages that
  are not pre-built by the Nixpkgs release.  They only get to see the
  packages available in populate-cache.nix.

- There are too many name conflicts in all-packages-generic.nix.  For
  instance, there are four packages named "gcc-4.0.2" (all but one are
  cross-compilers).

- "nix-env -qa" on all-packages-generic.nix prints out lots of
  garbage, in particular "packages" that shouldn't be in the list like
  tarballs.  For example:

    $ nix-env -f ~/Dev/nixpkgs/pkgs/system/i686-linux.nix -qa
    ...
    libgnomecanvas-2.10.0
    libgnomecanvas-2.10.0.tar.bz2
    ...

  This is because it currently recurses into attribute sets.

- It is hard to maintain local modifications of a Nixpkgs tree.  For
  instance, Firefox in the channel is built without the RealPlayer
  plugin enabled (for legal reasons).  If you have a Nixpkgs tree
  checked out, this is easy to change by editing the appropriate lines
  in all-packages-generic.nix.  But this makes it hard to upgrade
  later: "nix-channel --update; nix-env -u firefox" will just install
  a new Firefox *without* RealPlayer.

  Other examples are whether to build Subversion's Apache modules,
  whether to build GnuPG with (patent-encumbered) IDEA support, and
  configuration settings like the path of the MythTV database or the
  Quake 3 game files.

- It is too hard to easily build a package without installing it.
  That's what "nix-build" is for, but nix-build has a quite different
  interface than nix-env: e.g., it only works on top-level Nix
  expressions (i.e., you cannot select a package by symbolic name).

- A better default policy for resolving file collisions between
  packages (i.e., when two packages both have a file "bin/foo") is
  required.  Right now the user environment builder just bails out.  A
  more user-friendly default would be to let the most recently added
  package override older ones.

- Finally, there are some user interface warts in nix-env, such as
  https://bugs.cs.uu.nl/browse/NIX-17 and
  https://bugs.cs.uu.nl/browse/NIX-31.

===

So I propose a number of changes to nix-env and Nixpkgs:

- Rename all-packages-generic.nix to all-packages.nix, get rid of
  the old all-packages.nix and i686-linux and friends.  The latter are
  all redundant: default arguments can be used for that.  The new
  all-packages.nix would be function like this:

  {system ? __currentSystem}:

  rec {
    stdenvs = ...;
    stdenv =
      if system == "i686-linux" then stdenvs.stdenvLinuxPkgs
      else if system == "powerpc-darwin" then ...
    ... all packages ...
  }

- Rename populate-cache.nix to build-for-release.nix or something like
  that ("cache" is a misnomer).

  So the result is that we have only two files:

  * all-packages.nix
  * build-for-release.nix (which can be ignored by most people)

- Make all-packages.nix the default Nix expression for the channel
  instead of populate-cache.nix.  This makes all packages in Nixpkgs,
  including those not available in binary form, visible to channel
  users.

- Of course, this gives assertion failures on platforms other than
  i686-linux.  So nix-env should be modified to silently ignore
  derivations that evaluate to an assertion failure.

- Since all-packages.nix now shows everything, including source-only
  packages, it may be useful to have a flag "--prebuilt-only" (or
  something like that) to filter out packages for which there are no
  substitutes.

- Since all-packages.nix is now a function, nix-env should be modified
  to automatically call functions whose arguments all have defaults
  (nix-build already does this!).  So

    $ nix-env -f .../pkgs/system/all-packages.nix -i foo

  will call the function in all-packages.nix with "system" set to
  "__currentSystem" (like "i686-linux") and do the right thing.

- nix-env shouldn't recurse into attribute sets anymore.  Sometimes
  this is desirable though, so maybe an attribute like "nixenvRecurse"
  could be specified in an attribute set for which we do want this.

- Add a flag "nix-env -A" to select packages by attribute name, e.g.

    $ nix-env -i -A subversionWithJava
    $ nix-env -i -A xlibs.xorgserver

  (This is equivalent to "nix-env -i -E 'x: x.xlibs.xorgserver'", but
  rather easier.)

- Merge nix-build into nix-env, for instance by adding a flag "nix-env
  --build-only".

- Maintaining local configuration: all-packages.nix could read in a
  file like ~/.nixpkgs/config.nix containing stuff like

    { subversionApacheSupport = true;
      firefoxRealplayerSupport = true;
      quake3Paks = "/bla/quake3/baseq3";
      mythtvState = "/var/db/mythtv";
    }

  The only catch is that the Nix expression language needs a way to
  find this file (through $HOME) and only read it if exists.
  Something like this (in all-package.nix):

    rec {
      defaults = {
        subversionApacheSupport = false;
        firefoxRealplayerSupport = false;
        ...
      }

      config = loadWithDefault defaults
        ((getEnv "HOME") + "/.nixpkgs/config.nix");

      ...

      subversion = import .../subversion {
        inherit fetchurl stdenv ...;
        httpServer = subversionApacheSupport;
      };

      ...
    }

- We could even allow users to override entire packages.  For
  instance, we can wrap all-packages.nix like this:

    (rec {
      .. all packages ...
    })
    //
    (loadWithDefault {}
      ((getEnv "HOME") + "/.nixpkgs/overrides.nix"));

  where ~/.nixpkgs/overrides.nix can contain definitions like

    firefox = { ... };

- In Gentoo style, it should be possible to "mask" packages so they
  won't be installed or upgraded automatically.

- Eliminate the naming conflicts in all-packages.nix.  For instance, a
  gcc cross-compiler for Sparc might be called
  "gcc-x86-to-sparc-4.0.2" instead of "gcc-4.0.2".

Okay, this post was way too long.  Comments?

-- 
Eelco Dolstra | http://www.cs.uu.nl/~eelco