[Nix-dev] nix for reproducible and self-updating developer environments

[Alexander Schmolck]

Hi,

I've been using nix for providing developers on linux and os x with a
reproducible development environment that is automatically kept up to
date. I'd like to give a little bit of feedback about the experience
and ask for some advice how to improve certain aspects (I'm pretty
sure what I'm doing isn't optimal, so any help is greatly
appreciated).

The basic setup is that I have a file packages.nix in our main
development repo that looks roughly like this:

# packages.nix -----
let
     # override versions and options for a few system packages, e.g.
to unbreak php on darwin
     config = import ./config.nix;
     # pin to a particular version of the nix-channel
     pinnedNixpkgs = fetchTarball
https://github.com/NixOS/nixpkgs-channels/archive/SOMEHASH.tar.gz;
     pkgs = import pinnedNixpkgs {inherit config};
     callPackage = pkgs.lib.callPackageWith (pkgs // self);
     self =  {
       pep8 = pkgs.pythonPackages.pep8;
       yapf = callPackage ./yapf.nix { };
       # ... lots more packages, tools, languages etc.
     };
} in  self
# packages.nix ----

I then added a git hook for branch changes, merges etc. that does
nix-env -if packages.nix. That way, for all the development tools and
dependencies managed by nix, a developer should automatically always
have the right version when switching to a particular branch or
merging the latest master release into their branch.  On CI, I run the
build in a nix-shell with a default.nix that roughly looks like so:

# default.nix ---
let stdenv = (fetchTarball
https://github.com/NixOS/nixpkgs-channels/archive/SOMEHASH.tar.gz).stdenv;
in
{
   ourDevEnv = stdenv.mkDerivation rec {
      name = "our-dev-env";
      buildInputs = builtins.attrValues (import ./packages.nix {});
   };
}
# default.nix ---

which should hopefully provide isolation for parallel build agent
runs. After a successful build of master on the CI server I also do
nix-push --dest /tmp/cache $(nix-build packages.nix) and upload to S3,
to create a binary cache for our derivations.


The positives:

- a comprehensive cross-platform and cross-language solution! In
particular, it covers the most popular deployment (linux) and
development (os x, linux) platforms and can handle the whole stack,
unlike e.g. virtualenv/pip or rvm/bundler. I've been bitten badly and
frequently in the past by python and ruby problems caused by subtle
shared library differences between different machines even when all
the python or ruby dependencies where fully pinned.
- nix-env -if package.nix is extremely fast in the no-op or switching
back to a previous version case (under a sec), this is what makes it
possible to run it all the time in a git hook without slowing down git
use. That's really helpful for ensuring everyone has an up-to-date
environment.
- good coverage and recency of packages, especially given the
community size relative to ubuntu, centos etc.
- not only is nix(os) conceptually years ahead, I've been very
impressed by the design of the nix language. Normally when I encounter
a DSL I just wish someone had used scheme or similar but there are
very few things I don't like about the nix language, it's very well
thought out.
- in many cases, adding a package that's not in nix yet (e.g. yapf
above) or overriding the version is pretty simple and straightforward
and the majority of derivations in nixpkgs look quite nice (much more
so than e.g. debian package defs).
- well written reference documentation (although certain parts could
use a bit more coverage)
- it's pretty easy to setup a cache in S3

The not so positives:

- selecting and pinning, and later bumping a version of nixkpkgs seems
more work than it should be (maybe I'm doing it the wrong way)
- neither nix-env or nix-shell really feels quite right for managing
developer environments. Nix-shell is great for developing derivations
or CI but it's too high friction for normal development (and I believe
wouldn't support the git-hook-auto-update trick). Nix-env, on the
other hand, is a tad too stateful. I'd like a way to make sure that
*only* the packages currently in packages.nix (and maybe some optional
user defined personal.nix) are active, so that removing something from
our packages.nix really deletes it from the user environment. I tried
something along the lines of keeping a a reference to the /nix/store
path of the current nix-env binary, nix-env -e '*' and then use the
absolute path to install packages.nix. But that appears tricky to get
right and isn't super-elegant. Is there a better solution I'm missing?
- os x support felt pretty alpha a lot of the time. Several packages
are linux only and amongst those that are not important packages like
php have been consistently broken. The El Captain upgrade caused us
lots of woes and all sorts of fundamental clang compilation breakage.
Of course major changes between xcode and os x versions made that a
tough nut to crack and it seems the situation has improved quite a bit
now, probably thanks to the switch to pure darwin.
- The reproducible bit turned out harder in practice than I had hoped,
despite having our own cache and despite trying to pin to a particular
version of the channel (see above), I ran into at least two problems:
the above nix-push didn't seem to be sufficient to really put all the
required artefacts into our cache. One thing that's evidently missing
is the nixpkgs tarball. I'd like to include it (what's a good way of
doing so?), but a bigger problem is missing binaries, especially in os
x where . In one case I had to resort to nix-serve to get an os x
developer back to a working build environment, because his clang
compile on nix was hosed and although the nix-env -if got some
packages out of our cache, it tried to install some other stuff from
source. The second problem I had, is that try as I might, I couldn't
convince nix not to use cache.nixos.org, even if I removed it from
/etc/nix/nix.conf.
- Other devs experienced various nix-related breakage that needed my
help to debug in order to get them back to a functioning development
environment again (e.g. the nix install via curl sometimes appeared to
be broken, there was at least one instance of corrupted hydra
packages, nix1.9 became unable to read nixpkgs in a relatively short
timeframe, I had some instances were people had problems with stuff
that completely puzzled me, e.g. different conflicting versions of the
same package installed (despite our pinning attempts), ~/.nix-defexpr
going missing, apparent inconsistencies in the nix store for no
obvious reasons – "waiting for locks or build slots" etc.). Maybe some
of these were caused by attempts to fix things themselves that were
based on mistaken notions how nix works, but either way it was more of
a problem than I had anticipated.
- overriding stuff can be quite challenging in certain cases, for
example php uses an older and mostly undocumented override mechanism
- private repos are a headache (fetchGitPrivate is awkward enough that
I decided to  avoid it completely because I saw no way to robustly
make it work in an automated fashion. That necessitated some
additional complexity)
- ssl authentication has also been a cause of headaches; in the end I
had to remove curl and git from packages.nix because ssl validation
failures caused to many problems (this was compounded by some
particularities of our build process, such as enforcing a well
controlled set of environment variables for reproducibility, which
means SSL_CERT_FILE set by nix.sh is not preserved in builds). Also,
whilst SSL_CERT_FILE works in principle on linux, because there is an
authoritative file in the required format, I've had various issues on
OS X where no canonical such file exists.
- although nix is a nice language, it suffers a bit on the tooling
side (the combination of lack of type-checking and laziness can result
in confusing error messages, pretty printing and editor support could
be better, and as far as I'm aware there is no tooling that could )
- the quality of programming language infrastructure is a bit uneven
and documentation is generally sparse (e.g. there seem to be two
different and incompatible versions of npm2nix and for languages other
than Haskell documentation on how to do development with language X
and nix is pretty sparse). What I tried to do with python and ruby
worked pretty well, but I've had for example no luck with getting
node-based utilities with a plugin architecture to work (e.g eslint,
with more than one plugin at the same time).
- although there is no lack of terrible language packaging
infrastructure (python or go come to mind) outside of the Haskell camp
(cursed with cabal and blessed with an unusual appetite for obscure
tech) nix seems to have seen little uptake as a dev dependency
management solution by language communities.

One frustration I have with nix is that I sometimes get the impression
that after solving 95% of the hard problems in an elegant and original
way it fails to leverage this innovation to provide clearly outlined
go-to solutions for problems for which people struggle with.  Even
after investing a fair amount of time into learning it, it's often not
obvious how to use to accomplish particular tasks in a way that
represents a major win. Compare that to e.g. docker were I can find
and teach others in minutes how to solve concrete problems that would
otherwise be real pain-points with a minimum of fuzz (e.g. run a bunch
of integration tests in parallel with port and process isolation, test
a build with a different linux version, bring up a self contained
service like a code search engine for your repo with a single shell
line etc.). Docker has it's share of design problems and pitfalls, but
it's possible to get some real value out of it amazingly quickly. I
don't think that's true of nix yet to the same extent, although I find
it far more interesting than docker.

For example the nix homepage touts reproducibility and that "Nix makes
it trivial to set up and share build environments for your projects,
regardless of what programming languages and tools you’re using." It
certainly feel like that ought to be the case, because that would be a
natural killer feature for nix. However, based on my experiences, that
statement just doesn't strike me as accurate for any reasonable
definition of "trivial" (at least if one goes by what can easily found
in the official docs).

What would be great to see in this context are clear step to step
instructions for:

- bootstrapping a fixed version of nix, ideally from a self-contained archive
- a simple way to specify (and later bump) an exact known good version
of nixpkgs and also to downgrade or upgrade individual packages
relative to that.
- self-hosting all vital components, so that e.g. an outage (or
disappearance) of nixos.org or one of the upstream providers of src
tarballs does not mean that it's suddenly no longer possible to set up
a developer machine or do a deploy. It's also needed so that
non-public artifacts can be installed efficiently rather than
recompiled on every dev's machine. In practice I think that means an
easy way to set up a binary and, ideally, upstream src tar.gz cache in
S3 (other options are good as well, of course, but S3 seems to be the
winning solution for hosting a bunch of files easily, cost-efficiently
and with sufficient reliability, access controls and bandwidth).
- a robust way to set additional configuration options where required
in an automated fashion (think 'git config set'/install flags rather
than sed and pray)
- a good way to integrate private packages
- some easy steps and concrete examples how to integrate utilities and
library dependencies for the key programming languages. What the key
languages are is of course both open to debate and a moving target.
However, by any reckoning javascript,  python, ruby, go, c/c++ are
definitely up there (and java and php probably too). Mainly because of
the sheer amount of tooling they underpin, in addition to their
popularity as languages. Also, since it's unreasonable to expect for
example every frontend dev to master a relatively obscure lazy
functional programming language and package system before they can add
or bump some library dependency, it would really be invaluable to have
a very streamlined process for updating say npm or gem dependencies.
Ideally it would just involve editing a package.json and issuing  a
single command (e.g. `nix bump`) to get an updated nix expression.
Bundix, npm2nix are great steps in this direction but they're still
too hard to use, IMO. If nix can't be used successfully in teams were
the majority have little time to invest in learning it, it's unlikely
to ever see wide adoption.

I think a git repo with an evolving, complete worked example how to
provide a reproducible dev environment for a multi-language project
would be super helpful. If there's interest I'd be more than willing
to contribute to this. As noted, there are shortcomings with the
approach I outlined above but maybe it can serve as a starting point
for something more polished with the aid of more experienced community
members.

alexander