[Nix-dev] Rework goPackages

[Kamil Chmielewski]

Hi,

few days ago I started a PR called [WIP] Rework goPackages
https://github.com/NixOS/nixpkgs/pull/16017 which is now on master and can
leads to confusion for everyone creating goPackages before. I was also a
little surprised how fast it was merged but I thought it was the effect of
how many people feel the same pain as me, trying to manage or introduce new
goPackages so there wasn't a lot of discussion there. But now it starts in
few places and I think that this will be better place to make decisions
about Go packaging in Nix.

I want to start with the motivation which led me to this topic. Few months
ago I wanted to introduce Otto (https://www.ottoproject.io/) to Nix. I was
starting with Nix back then i tried to do things as they were in
go-packages.nix.
So I created otto as buildFromGithub[1] and use its bin output in
all-packages.nix[2]. Next nix-build -A otto and I got 23 dependencies
missing[3]. So I started to add dependencies manually doing  nix-build -A
otto tens of times because each of them depends on new things that `go
build` tells me only when I add new sources for build... [4]
After many hours of doing nix-build and new edits in go-packages.nix I
ended up with otto done in 400 lines of Nix [5] I don't want do maintain.
Besides new packages I also need to change version of things that were
there before, potentially breaking other things. I can't even image how
many hours of work it'll cost me to update it to new version :( So there no
otto in nixpkgs.
Instead of making otto I've started go2nix[6] prototype where I want to use
Go toolchain because it can find out all transitive dependencies in one go.
It wasn't hard to write a prototype in Go because everthing for it is
there. So after a minutes I could automatically generate a list of all
dependencies that I need to build any Go binary! Next step was to generate
working Nix expression with this list of dependencies that can be used in
goPackages. How hard can it be? :) It turned out that it's impossible to do
this reusing goPackages in its current state. How could I know that:
* github.com/ugorji/go sould be ugorji.go [7]
* gopkg.in/flosch/pongo2.v3 should be pongo2 [8]
* google.golang.org/api should be google-api-go-client [9]
* github.com/odeke-em/google-api-go-client shoul be
odeke-em.google-api-go-client
[10]
* github.com/stathat/go should be stathat [11]
...

So there were 2 ways of dealing it:
* generate new goPackage without any imports from go-packages.nix and
abandon it completely
* change go-packages.nix into something that can be readable for something
that knows how packages are named in Go land
I thought that go-packages.nix, python-packages.nix ...... says that we
want to reuse common dependencies somewhere but it should be possible to
override some of them if needed. And that was the beginning of [WIP] Rework
goPackages with go-packages.nix "converted" into libs.json that was also
influenced by temptation to make it possible to automate future updates[12].

As a troublemaker I want to involve all interested people to find out what
next moves we should do to make:
* master branch at least as usable as it was before rework
* working with Nix and Go the best experience for both Nixers and Gophers

So lets start with the first one. Most comments I see is related to
libs.json.
Does it mean we want go-packages.nix but in form that can be generated and
imported like libs.json or we don't want it at all moving all the
dependencies to separate derivations? Or maybe we want go-packages.nix as
it was before?

--
Kamil

[1]
https://github.com/kamilchm/nixpkgs/pull/3/files#diff-8631dcd3e45d5f2ceaedeb96a23bee6fR2214
[2]
https://github.com/kamilchm/nixpkgs/pull/3/files#diff-036410e9211b4336186fc613f7200b12R12219
[3]
https://gist.github.com/kamilchm/bec9483a111babafe965abcec9efb013#file-nix-build-a-otto
[4] https://gist.github.com/kamilchm/bec9483a111babafe965abcec9efb013
[5] https://github.com/kamilchm/nixpkgs/pull/3
[6] https://github.com/kamilchm/go2nix
[7]
https://github.com/kamilchm/nixpkgs/blob/otto-before/pkgs/top-level/go-packages.nix#L683
[8]
https://github.com/kamilchm/nixpkgs/blob/otto-before/pkgs/top-level/go-packages.nix#L2070
[9]
https://github.com/kamilchm/nixpkgs/blob/otto-before/pkgs/top-level/go-packages.nix#L861
[10]
https://github.com/kamilchm/nixpkgs/blob/otto-before/pkgs/top-level/go-packages.nix#L872
[11]
https://github.com/kamilchm/nixpkgs/blob/otto-before/pkgs/top-level/go-packages.nix#L2700
[12] https://github.com/NixOS/nixpkgs/pull/13819
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.science.uu.nl/pipermail/nix-dev/attachments/20160617/56c53633/attachment.html>