8.10. Python

Several versions of Python are available on Nix as well as a high amount of packages. The default interpreter is CPython 2.7.

The recommended method for creating Python environments for development is with nix-shell. Executing

$ nix-shell -p python35Packages.numpy python35Packages.toolz

opens a Nix shell which has available the requested packages and dependencies. Now you can launch the Python interpreter (which is itself a dependency)

[nix-shell:~] python3

If the packages were not available yet in the Nix store, Nix would download or build them automatically. A convenient option with nix-shell is the --run option, with which you can execute a command in the nix-shell. Let’s say we want the above environment and directly run the Python interpreter

$ nix-shell -p python35Packages.numpy python35Packages.toolz --run "python3"

This way you can use the --run option also to directly run a script

$ nix-shell -p python35Packages.numpy python35Packages.toolz --run "python3 myscript.py"

In fact, for this specific use case there is a more convenient method. You can add a shebang to your script specifying which dependencies Nix shell needs. With the following shebang, you can use nix-shell myscript.py and it will make available all dependencies and run the script in the python3 shell.

#! /usr/bin/env nix-shell
#! nix-shell -i python3 -p python3Packages.numpy

import numpy

print(numpy.__version__)

Likely you do not want to type your dependencies each and every time. What you can do is write a simple Nix expression which sets up an environment for you, requiring you only to type nix-shell. Say we want to have Python 3.5, numpy and toolz, like before, in an environment. With a shell.nix file containing

with import <nixpkgs> {};

(pkgs.python35.withPackages (ps: [ps.numpy ps.toolz])).env

executing nix-shell gives you again a Nix shell from which you can run Python.

What’s happening here?

  1. We begin with importing the Nix Packages collections. import <nixpkgs> import the <nixpkgs> function, {} calls it and the with statement brings all attributes of nixpkgs in the local scope. Therefore we can now use pkgs.

  2. Then we create a Python 3.5 environment with the withPackages function.

  3. The withPackages function expects us to provide a function as an argument that takes the set of all python packages and returns a list of packages to include in the environment. Here, we select the packages numpy and toolz from the package set.

  4. And finally, for in interactive use we return the environment by using the env attribute.

Now that you know how to get a working Python environment on Nix, it is time to go forward and start actually developing with Python. We will first have a look at how Python packages are packaged on Nix. Then, we will look how you can use development mode with your code.

On Nix all packages are built by functions. The main function in Nix for building Python packages is buildPythonPackage. Let’s see how we would build the toolz package. According to python-packages.nix toolz is build using

toolz = buildPythonPackage rec{
  name = "toolz-${version}";
  version = "0.7.4";

  src = pkgs.fetchurl{
    url = "mirror://pypi/t/toolz/toolz-${version}.tar.gz";
    sha256 = "43c2c9e5e7a16b6c88ba3088a9bfc82f7db8e13378be7c78d6c14a5f8ed05afd";
  };

  meta = {
    homepage = "http://github.com/pytoolz/toolz/";
    description = "List processing tools and functional utilities";
    license = licenses.bsd3;
    maintainers = with maintainers; [ fridh ];
  };
};

What happens here? The function buildPythonPackage is called and as argument it accepts a set. In this case the set is a recursive set (rec). One of the arguments is the name of the package, which consists of a basename (generally following the name on PyPi) and a version. Another argument, src specifies the source, which in this case is fetched from an url. fetchurl not only downloads the target file, but also validates its hash. Furthermore, we specify some (optional) meta information.

The output of the function is a derivation, which is an attribute with the name toolz of the set pythonPackages. Actually, sets are created for all interpreter versions, so python27Packages, python34Packages, python35Packages and pypyPackages.

The above example works when you’re directly working on pkgs/top-level/python-packages.nix in the Nixpkgs repository. Often though, you will want to test a Nix expression outside of the Nixpkgs tree. If you create a shell.nix file with the following contents

with import <nixpkgs> {};

pkgs.python35Packages.buildPythonPackage rec {
  name = "toolz-${version}";
  version = "0.7.4";

  src = pkgs.fetchurl{
    url = "mirror://pypi/t/toolz/toolz-${version}.tar.gz";
    sha256 = "43c2c9e5e7a16b6c88ba3088a9bfc82f7db8e13378be7c78d6c14a5f8ed05afd";
  };

  meta = {
    homepage = "http://github.com/pytoolz/toolz/";
    description = "List processing tools and functional utilities";
    license = licenses.bsd3;
    maintainers = with maintainers; [ fridh ];
  };
}

and then execute nix-shell will result in an environment in which you can use Python 3.5 and the toolz package. As you can see we had to explicitly mention for which Python version we want to build a package.

The above example considered only a single package. Generally you will want to use multiple packages. If we create a shell.nix file with the following contents

with import <nixpkgs> {};

( let
    toolz = pkgs.python35Packages.buildPythonPackage rec {
      name = "toolz-${version}";
      version = "0.7.4";

      src = pkgs.fetchurl{
        url = "mirror://pypi/t/toolz/toolz-${version}.tar.gz";
        sha256 = "43c2c9e5e7a16b6c88ba3088a9bfc82f7db8e13378be7c78d6c14a5f8ed05afd";
      };

      meta = {
        homepage = "http://github.com/pytoolz/toolz/";
        description = "List processing tools and functional utilities";
        license = licenses.bsd3;
        maintainers = with maintainers; [ fridh ];
      };
    };

  in pkgs.python35.withPackages (ps: [ps.numpy toolz])
).env

and again execute nix-shell, then we get a Python 3.5 environment with our locally defined package as well as numpy which is build according to the definition in Nixpkgs. What did we do here? Well, we took the Nix expression that we used earlier to build a Python environment, and said that we wanted to include our own version of toolz. To introduce our own package in the scope of withPackages we used a let expression. You can see that we used ps.numpy to select numpy from the nixpkgs package set (ps). But we do not take toolz from the nixpkgs package set this time. Instead, toolz will resolve to our local definition that we introduced with let.

Our example, toolz, doesn’t have any dependencies on other Python packages or system libraries. According to the manual, buildPythonPackage uses the arguments buildInputs and propagatedBuildInputs to specify dependencies. If something is exclusively a build-time dependency, then the dependency should be included as a buildInput, but if it is (also) a runtime dependency, then it should be added to propagatedBuildInputs. Test dependencies are considered build-time dependencies.

The following example shows which arguments are given to buildPythonPackage in order to build datashape.

datashape = buildPythonPackage rec {
  name = "datashape-${version}";
  version = "0.4.7";

  src = pkgs.fetchurl {
    url = "mirror://pypi/D/DataShape/${name}.tar.gz";
    sha256 = "14b2ef766d4c9652ab813182e866f493475e65e558bed0822e38bf07bba1a278";
  };

  buildInputs = with self; [ pytest ];
  propagatedBuildInputs = with self; [ numpy multipledispatch dateutil ];

  meta = {
    homepage = https://github.com/ContinuumIO/datashape;
    description = "A data description language";
    license = licenses.bsd2;
    maintainers = with maintainers; [ fridh ];
  };
};

We can see several runtime dependencies, numpy, multipledispatch, and dateutil. Furthermore, we have one buildInput, i.e. pytest. pytest is a test runner and is only used during the checkPhase and is therefore not added to propagatedBuildInputs.

In the previous case we had only dependencies on other Python packages to consider. Occasionally you have also system libraries to consider. E.g., lxml provides Python bindings to libxml2 and libxslt. These libraries are only required when building the bindings and are therefore added as buildInputs.

lxml = buildPythonPackage rec {
  name = "lxml-3.4.4";

  src = pkgs.fetchurl {
    url = "mirror://pypi/l/lxml/${name}.tar.gz";
    sha256 = "16a0fa97hym9ysdk3rmqz32xdjqmy4w34ld3rm3jf5viqjx65lxk";
  };

  buildInputs = with self; [ pkgs.libxml2 pkgs.libxslt ];

  meta = {
    description = "Pythonic binding for the libxml2 and libxslt libraries";
    homepage = http://lxml.de;
    license = licenses.bsd3;
    maintainers = with maintainers; [ sjourdois ];
  };
};

In this example lxml and Nix are able to work out exactly where the relevant files of the dependencies are. This is not always the case.

The example below shows bindings to The Fastest Fourier Transform in the West, commonly known as FFTW. On Nix we have separate packages of FFTW for the different types of floats ("single", "double", "long-double"). The bindings need all three types, and therefore we add all three as buildInputs. The bindings don’t expect to find each of them in a different folder, and therefore we have to set LDFLAGS and CFLAGS.

pyfftw = buildPythonPackage rec {
  name = "pyfftw-${version}";
  version = "0.9.2";

  src = pkgs.fetchurl {
    url = "mirror://pypi/p/pyFFTW/pyFFTW-${version}.tar.gz";
    sha256 = "f6bbb6afa93085409ab24885a1a3cdb8909f095a142f4d49e346f2bd1b789074";
  };

  buildInputs = [ pkgs.fftw pkgs.fftwFloat pkgs.fftwLongDouble];

  propagatedBuildInputs = with self; [ numpy scipy ];

  # Tests cannot import pyfftw. pyfftw works fine though.
  doCheck = false;

  LDFLAGS="-L${pkgs.fftw.dev}/lib -L${pkgs.fftwFloat.out}/lib -L${pkgs.fftwLongDouble.out}/lib"
  CFLAGS="-I${pkgs.fftw.dev}/include -I${pkgs.fftwFloat.dev}/include -I${pkgs.fftwLongDouble.dev}/include"
  '';

  meta = {
    description = "A pythonic wrapper around FFTW, the FFT library, presenting a unified interface for all the supported transforms";
    homepage = http://hgomersall.github.com/pyFFTW/;
    license = with licenses; [ bsd2 bsd3 ];
    maintainer = with maintainers; [ fridh ];
  };
};

Note also the line doCheck = false;, we explicitly disabled running the test-suite.

As a Python developer you’re likely aware of development mode (python setup.py develop); instead of installing the package this command creates a special link to the project code. That way, you can run updated code without having to reinstall after each and every change you make. Development mode is also available on Nix as explained in the Nixpkgs manual. Let’s see how you can use it.

In the previous Nix expression the source was fetched from an url. We can also refer to a local source instead using

src = ./path/to/source/tree;

If we create a shell.nix file which calls buildPythonPackage, and if src is a local source, and if the local source has a setup.py, then development mode is activated.

In the following example we create a simple environment that has a Python 3.5 version of our package in it, as well as its dependencies and other packages we like to have in the environment, all specified with propagatedBuildInputs. Indeed, we can just add any package we like to have in our environment to propagatedBuildInputs.

with import <nixpkgs>;
with pkgs.python35Packages;

buildPythonPackage rec {
  name = "mypackage";
  src = ./path/to/package/source;
  propagatedBuildInputs = [ pytest numpy pkgs.libsndfile ];
};

It is important to note that due to how development mode is implemented on Nix it is not possible to have multiple packages simultaneously in development mode.

Earlier we created a Python environment using withPackages, and included the toolz package via a let expression. Let’s split the package definition from the environment definition.

We first create a function that builds toolz in ~/path/to/toolz/release.nix

{ pkgs, buildPythonPackage }:

buildPythonPackage rec {
  name = "toolz-${version}";
  version = "0.7.4";

  src = pkgs.fetchurl{
    url = "mirror://pypi/t/toolz/toolz-${version}.tar.gz";
    sha256 = "43c2c9e5e7a16b6c88ba3088a9bfc82f7db8e13378be7c78d6c14a5f8ed05afd";
  };

  meta = {
    homepage = "http://github.com/pytoolz/toolz/";
    description = "List processing tools and functional utilities";
    license = licenses.bsd3;
    maintainers = with maintainers; [ fridh ];
  };
};

It takes two arguments, pkgs and buildPythonPackage. We now call this function using callPackage in the definition of our environment

with import <nixpkgs> {};

( let
    toolz = pkgs.callPackage ~/path/to/toolz/release.nix { pkgs=pkgs; buildPythonPackage=pkgs.python35Packages.buildPythonPackage; };
  in pkgs.python35.withPackages (ps: [ ps.numpy toolz ])
).env

Important to remember is that the Python version for which the package is made depends on the python derivation that is passed to buildPythonPackage. Nix tries to automatically pass arguments when possible, which is why generally you don’t explicitly define which python derivation should be used. In the above example we use buildPythonPackage that is part of the set python35Packages, and in this case the python35 interpreter is automatically used.

Versions 2.6, 2.7, 3.3, 3.4 and 3.5 of the CPython interpreter are available on Nix and are available as python26, python27, python33, python34 and python35. The PyPy interpreter is also available as pypy. Currently, the aliases python and python3 correspond to respectively python27 and python35. The Nix expressions for the interpreters can be found in pkgs/development/interpreters/python.

Python packages (libraries) and applications that use setuptools or distutils are typically built with respectively the buildPythonPackage and buildPythonApplication functions.

All Python packages reside in pkgs/top-level/python-packages.nix and all applications elsewhere. Some packages are also defined in pkgs/development/python-modules. It is important that these packages are called in pkgs/top-level/python-packages.nix and not elsewhere, to guarantee the right version of the package is built.

Based on the packages defined in pkgs/top-level/python-packages.nix an attribute set is created for each available Python interpreter. The available sets are

and the aliases

The buildPythonPackage function is implemented in pkgs/development/python-modules/generic/default.nix

and can be used as:

twisted = buildPythonPackage {
  name = "twisted-8.1.0";

  src = pkgs.fetchurl {
    url = http://tmrc.mit.edu/mirror/twisted/Twisted/8.1/Twisted-8.1.0.tar.bz2;
    sha256 = "0q25zbr4xzknaghha72mq57kh53qw1bf8csgp63pm9sfi72qhirl";
  };

  propagatedBuildInputs = [ self.ZopeInterface ];

  meta = {
    homepage = http://twistedmatrix.com/;
    description = "Twisted, an event-driven networking engine written in Python";
    license = stdenv.lib.licenses.mit; };
  };

The buildPythonPackage mainly does four things:

As in Perl, dependencies on other Python packages can be specified in the buildInputs and propagatedBuildInputs attributes. If something is exclusively a build-time dependency, use buildInputs; if it’s (also) a runtime dependency, use propagatedBuildInputs.

By default tests are run because doCheck = true. Test dependencies, like e.g. the test runner, should be added to buildInputs.

By default meta.platforms is set to the same value as the interpreter unless overriden otherwise.

All parameters from mkDerivation function are still supported.

As explained in the user’s guide installing individual Python packages imperatively with nix-env -i or declaratively in environment.systemPackages is not supported. However, it is possible to install a Python environment with packages (python.buildEnv).

In the following examples we create an environment with Python 3.5, numpy and ipython. As you might imagine there is one limitation here, and that’s you can install only one environment at a time. You will notice the complaints about collisions when you try to install a second environment.

Recursively updating a package can be done with pkgs.overridePackages as explained in the Nixpkgs manual. Python attribute sets are created for each interpreter version. We will therefore override the attribute set for the interpreter version we’re interested. In the following example we change the name of the package pandas to foo.

newpkgs = pkgs.overridePackages(self: super: rec {
  python35Packages = super.python35Packages.override {
    self = python35Packages // { pandas = python35Packages.pandas.override{name="foo";};};
  };
});

This can be tested with

with import <nixpkgs> {};

(let

newpkgs = pkgs.overridePackages(self: super: rec {
  python35Packages = super.python35Packages.override {
    self = python35Packages // { pandas = python35Packages.pandas.override{name="foo";};};
  };
});
in newpkgs.python35.withPackages (ps: [ps.blaze])
).env

A typical use case is to switch to another version of a certain package. For example, in the Nixpkgs repository we have multiple versions of django and scipy. In the following example we use a different version of scipy. All packages in newpkgs will now use the updated scipy version.

with import <nixpkgs> {};

(let

newpkgs = pkgs.overridePackages(self: super: rec {
  python35Packages = super.python35Packages.override {
    self = python35Packages // { scipy = python35Packages.scipy_0_16;};
  };
});
in newpkgs.python35.withPackages (ps: [ps.blaze])
).env

The requested package blaze depends upon pandas which itself depends on scipy.

If you get the following error:

could not create '/nix/store/6l1bvljpy8gazlsw2aw9skwwp4pmvyxw-python-2.7.8/etc':
Permission denied

This is a known bug in setuptools. Setuptools install_data does not respect --prefix. An example of such package using the feature is pkgs/tools/X11/xpra/default.nix. As workaround install it as an extra preInstall step:

${python.interpreter} setup.py install_data --install-dir=$out --root=$out
sed -i '/ = data\_files/d' setup.py