[Nix-dev] Rework goPackages

Fri Jun 17 19:07:11 CEST 2016

Hi Kamil,

First thank you for working on this! Having go2nix as a tool certainly
makes packaging Go programs much easier! I don't do a lot of Nix
packaging but I do write Go software---so maybe I have a different
perspective on this. Anyhow here are some thoughts:

- Many Go software projects try to be careful to vendor their
dependencies. For example, the Caddy server does this and it works
very well. Nix only needs to record the top-level package's SHA1 and
use fetchFromGithub + buildGoPacakge [1]. No need to recursively
expose all dependencies.

- An exception to this rule are projects that make use of C or C++
linked libraries. [2] For these you would need to add explicit
packages.

- Then I take a look at Otto's Makefile and the updatedeps target [3]
and I am shocked! This is a terrible practice that the Hashicorp folks
should fix. They do things correctly with Vault so I don't see why
they can't fix it here. In any case, here go2nix is awesome since it
can effectively discover each dependency (tooling around the `go list`
command and the contents of GOPATH) and produce Nix derivations for
each component.

- However, putting all of these in a "flat" go-packages.nix seems
incorrect. At the very least, two Go programs may require different
versions of the same package. And, as you mention, there's the
question of how to consistently name these packages. This is even more
complicated since packages may be named one thing but reside somewhere
else. And some packages get "folded in" to the standard library with
subsequent Go releases. [4] Or terrible redirect "services" like
gopkg.in [5]

- What's the goal here? I would expect that Nix has packages for
'major' Go programs and tooling to make it easy to package new
programs or update releases. The tooling should be designed so that an
update to one program doesn't break or alter dependencies for other
programs. I think this implies that the tooling produces a
'go-libs.json' per-program as opposed to a single top-level
go-packages.nix file. This might cause redundant downloads of common
dependencies but that cost seems relatively minor compared to the
organizational headache that a flat-namespace would require.

~ Scott

[1] https://github.com/NixOS/nixpkgs/blob/master/pkgs/servers/caddy/default.nix
[2] https://golang.org/cmd/cgo
[3] https://github.com/hashicorp/otto/blob/master/Makefile#L35
[4] See golang.org/x/net/context -> net/context in go1.7
[5] http://labix.org/gopkg.in

On Fri, Jun 17, 2016 at 6:55 AM, Kamil Chmielewski <kamil.chm at gmail.com> wrote:
> Hi,
>
> few days ago I started a PR called [WIP] Rework goPackages
> https://github.com/NixOS/nixpkgs/pull/16017 which is now on master and can
> leads to confusion for everyone creating goPackages before. I was also a
> little surprised how fast it was merged but I thought it was the effect of
> how many people feel the same pain as me, trying to manage or introduce new
> goPackages so there wasn't a lot of discussion there. But now it starts in
> few places and I think that this will be better place to make decisions
> about Go packaging in Nix.
>
> I want to start with the motivation which led me to this topic. Few months
> ago I wanted to introduce Otto (https://www.ottoproject.io/) to Nix. I was
> starting with Nix back then i tried to do things as they were in
> go-packages.nix.
> So I created otto as buildFromGithub[1] and use its bin output in
> all-packages.nix[2]. Next nix-build -A otto and I got 23 dependencies
> missing[3]. So I started to add dependencies manually doing  nix-build -A
> otto tens of times because each of them depends on new things that `go
> build` tells me only when I add new sources for build... [4]
> After many hours of doing nix-build and new edits in go-packages.nix I ended
> up with otto done in 400 lines of Nix [5] I don't want do maintain. Besides
> new packages I also need to change version of things that were there before,
> potentially breaking other things. I can't even image how many hours of work
> it'll cost me to update it to new version :( So there no otto in nixpkgs.
> Instead of making otto I've started go2nix[6] prototype where I want to use
> Go toolchain because it can find out all transitive dependencies in one go.
> It wasn't hard to write a prototype in Go because everthing for it is there.
> So after a minutes I could automatically generate a list of all dependencies
> that I need to build any Go binary! Next step was to generate working Nix
> expression with this list of dependencies that can be used in goPackages.
> How hard can it be? :) It turned out that it's impossible to do this reusing
> goPackages in its current state. How could I know that:
> * github.com/ugorji/go sould be ugorji.go [7]
> * gopkg.in/flosch/pongo2.v3 should be pongo2 [8]
> * google.golang.org/api should be google-api-go-client [9]
> * github.com/odeke-em/google-api-go-client shoul be
> odeke-em.google-api-go-client [10]
> * github.com/stathat/go should be stathat [11]
> ...
>
> So there were 2 ways of dealing it:
> * generate new goPackage without any imports from go-packages.nix and
> abandon it completely
> * change go-packages.nix into something that can be readable for something
> that knows how packages are named in Go land
> I thought that go-packages.nix, python-packages.nix ...... says that we want
> to reuse common dependencies somewhere but it should be possible to override
> some of them if needed. And that was the beginning of [WIP] Rework
> goPackages with go-packages.nix "converted" into libs.json that was also
> influenced by temptation to make it possible to automate future updates[12].
>
> As a troublemaker I want to involve all interested people to find out what
> next moves we should do to make:
> * master branch at least as usable as it was before rework
> * working with Nix and Go the best experience for both Nixers and Gophers
>
> So lets start with the first one. Most comments I see is related to
> libs.json.
> Does it mean we want go-packages.nix but in form that can be generated and
> imported like libs.json or we don't want it at all moving all the
> dependencies to separate derivations? Or maybe we want go-packages.nix as it
> was before?
>
> --
> Kamil
>
> [1]
> https://github.com/kamilchm/nixpkgs/pull/3/files#diff-8631dcd3e45d5f2ceaedeb96a23bee6fR2214
> [2]
> https://github.com/kamilchm/nixpkgs/pull/3/files#diff-036410e9211b4336186fc613f7200b12R12219
> [3]
> https://gist.github.com/kamilchm/bec9483a111babafe965abcec9efb013#file-nix-build-a-otto
> [4] https://gist.github.com/kamilchm/bec9483a111babafe965abcec9efb013
> [5] https://github.com/kamilchm/nixpkgs/pull/3
> [6] https://github.com/kamilchm/go2nix
> [7]
> https://github.com/kamilchm/nixpkgs/blob/otto-before/pkgs/top-level/go-packages.nix#L683
> [8]
> https://github.com/kamilchm/nixpkgs/blob/otto-before/pkgs/top-level/go-packages.nix#L2070
> [9]
> https://github.com/kamilchm/nixpkgs/blob/otto-before/pkgs/top-level/go-packages.nix#L861
> [10]
> https://github.com/kamilchm/nixpkgs/blob/otto-before/pkgs/top-level/go-packages.nix#L872
> [11]
> https://github.com/kamilchm/nixpkgs/blob/otto-before/pkgs/top-level/go-packages.nix#L2700
> [12] https://github.com/NixOS/nixpkgs/pull/13819
>
>
> _______________________________________________
> nix-dev mailing list
> nix-dev at lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>