[Nix-dev] distutils2nix/python4nix

[Peter Simons]

Hi Florian,

 > first, I forgot nix-dev in Cc, if you agree, please include it in
 > your reply.

sure, I've added a Cc.


 >> The cabal2nix utility is written in Haskell, because the Cabal
 >> library allows developers to parse and access Cabal files easily,
 >> which is about 90% of the task at hand. For "setup.py" files,
 >> matters would be slightly more complicated, though. I reckon, it
 >> would probably make sense to write a "distutils2nix" utility in
 >> Python, rather than in Haskell.
 >
 > [I agree] that distutils2nix (I like the name) should be written in
 > python, though I would enjoy finally to have a haskell project.

There is Berp, <http://hackage.haskell.org/package/berp-0.0.1>, a Python
compiler written in Haskell. Maybe that tool could be used to interpret
setup.py in Haskell? Generally speaking, I'm not so sure whether
choosing that kind of architecture would be wise, though. I mean, if you
have to interpret Python, then using Python for that sounds like the
most obvious choice. :-)


 > From recent discussion on nix-dev, I understand:
 >
 > - nix has no resolver by design: one expression uses exact
 >   attributes, nix does not pick the "best match" for you. If one
 >   needs two versions available, one needs to create two attrs (one
 >   for each).

Yes, exactly. In general, it's not even possible to determine the "best
match" because we don't have the information we would need for that kind
of choice. Haskell packages, for example, usually specify version ranges
for their dependencies, but the version ranges people choose is
typically a guess, at best. So you cannot rely on that information. I'm
not sure what the situation in Python is, though.


 > - nix expressions is the format to specify build instructions for a
 >   package and a user of nix installs nix expressions without
 >   knowledge of the tools that might be used to generate them.

Yes. The expressions cabal2nix generates look almost exactly like those
we wrote manually before we had the tool. Users of Nix don't need to
know about cabal2nix.


 > [I would write] for python:
 >
 > 1. distutils2nix
 > 2. python4nix
 >
 > The set of nix expressions would take into account version constraints
 > specified in setup.py's requirements.
 >
 > I am not clear on how to achieve that. distutils2nix would take a
 > packages requirements and convert it to propagateBuildInputs. Without
 > version constraints that's easy. In case of version constraints (eg
 > odict<1.4) it needs to know the name of an attribute that will always
 > fulfill this constraint.

In cabal2nix, we just don't care about the version constraints. The file
"haskell-packages.nix" -- which puts everything together -- is
maintained manually. The underlying assumption is that most packages
build with the latest versions of their respective dependencies. Every
now and then, we need to maintain multiple version of the same package
to be able to build everything, but that is a rare case, fortunately. If
that happens, I typically e-mail the authors of the packages involved
and complain. :-)


 > In python, packages can be grouped in namespaces separated by dots
 > (eg. plone.app.deco). These need to be translated to "name"
 > (eg. python-plone-app-deco-1.3) and "attribute name"
 > (e.g. python27Packages.plone_app_deco_1_3 not sure about the
 > nomenclature here. Is there a naming convetion already?

In the case of Haskell, we spell names like "abc-def" as "abcDef".
"abc-def-1.3" would become "abcDef_1_3".


 > Apart from generating a nix expression tree taking setup.py version
 > requirements into account, we would need to be able to specify further,
 > exact version constraints (and create trees) to deploy one specific
 > application.

Personally, I would start writing and using "distutils2nix" before I'd
even worry about "python4nix". If you can automate the task of
generating a single expression, you've already achieved a lot. Based on
that experience, the design of the second step is probably going to be
clearer than it is now.

One issue that you might want to consider is that the "pythonPackages"
set needs to support several different versions of Python, i.e. the
python interpreter needs to be an argument to that expression.
Currently, we cannot build our python packages with Python 3.x, for
example, which is really a shame.

Take care,
Peter