[Nix-dev] Rework goPackages

Tue Jun 21 08:20:35 CEST 2016

Hope I didn't kill the thread. :-(

The TL;DR of my post was that I think go2nix is great and needed for
many Go projects that don't vendor go-dependencies. But, from my Go
development experience, one shouldn't try to boil-the-ocean and solve
all Go packaging for everything.

I've also started a PR to add vendoring to github.com/hashicorp/otto. [1]

Cheers,
~ Scott

[1] https://github.com/hashicorp/otto/pull/519

On Fri, Jun 17, 2016 at 10:07 AM, Scott Devoid <sdevoid at gmail.com> wrote:
> Hi Kamil,
>
> First thank you for working on this! Having go2nix as a tool certainly
> makes packaging Go programs much easier! I don't do a lot of Nix
> packaging but I do write Go software---so maybe I have a different
> perspective on this. Anyhow here are some thoughts:
>
> - Many Go software projects try to be careful to vendor their
> dependencies. For example, the Caddy server does this and it works
> very well. Nix only needs to record the top-level package's SHA1 and
> use fetchFromGithub + buildGoPacakge [1]. No need to recursively
> expose all dependencies.
>
> - An exception to this rule are projects that make use of C or C++
> linked libraries. [2] For these you would need to add explicit
> packages.
>
> - Then I take a look at Otto's Makefile and the updatedeps target [3]
> and I am shocked! This is a terrible practice that the Hashicorp folks
> should fix. They do things correctly with Vault so I don't see why
> they can't fix it here. In any case, here go2nix is awesome since it
> can effectively discover each dependency (tooling around the `go list`
> command and the contents of GOPATH) and produce Nix derivations for
> each component.
>
> - However, putting all of these in a "flat" go-packages.nix seems
> incorrect. At the very least, two Go programs may require different
> versions of the same package. And, as you mention, there's the
> question of how to consistently name these packages. This is even more
> complicated since packages may be named one thing but reside somewhere
> else. And some packages get "folded in" to the standard library with
> subsequent Go releases. [4] Or terrible redirect "services" like
> gopkg.in [5]
>
> - What's the goal here? I would expect that Nix has packages for
> 'major' Go programs and tooling to make it easy to package new
> programs or update releases. The tooling should be designed so that an
> update to one program doesn't break or alter dependencies for other
> programs. I think this implies that the tooling produces a
> 'go-libs.json' per-program as opposed to a single top-level
> go-packages.nix file. This might cause redundant downloads of common
> dependencies but that cost seems relatively minor compared to the
> organizational headache that a flat-namespace would require.
>
> ~ Scott
>
> [1] https://github.com/NixOS/nixpkgs/blob/master/pkgs/servers/caddy/default.nix
> [2] https://golang.org/cmd/cgo
> [3] https://github.com/hashicorp/otto/blob/master/Makefile#L35
> [4] See golang.org/x/net/context -> net/context in go1.7
> [5] http://labix.org/gopkg.in
>
> On Fri, Jun 17, 2016 at 6:55 AM, Kamil Chmielewski <kamil.chm at gmail.com> wrote:
>> Hi,
>>
>> few days ago I started a PR called [WIP] Rework goPackages
>> https://github.com/NixOS/nixpkgs/pull/16017 which is now on master and can
>> leads to confusion for everyone creating goPackages before. I was also a
>> little surprised how fast it was merged but I thought it was the effect of
>> how many people feel the same pain as me, trying to manage or introduce new
>> goPackages so there wasn't a lot of discussion there. But now it starts in
>> few places and I think that this will be better place to make decisions
>> about Go packaging in Nix.
>>
>> I want to start with the motivation which led me to this topic. Few months
>> ago I wanted to introduce Otto (https://www.ottoproject.io/) to Nix. I was
>> starting with Nix back then i tried to do things as they were in
>> go-packages.nix.
>> So I created otto as buildFromGithub[1] and use its bin output in
>> all-packages.nix[2]. Next nix-build -A otto and I got 23 dependencies
>> missing[3]. So I started to add dependencies manually doing  nix-build -A
>> otto tens of times because each of them depends on new things that `go
>> build` tells me only when I add new sources for build... [4]
>> After many hours of doing nix-build and new edits in go-packages.nix I ended
>> up with otto done in 400 lines of Nix [5] I don't want do maintain. Besides
>> new packages I also need to change version of things that were there before,
>> potentially breaking other things. I can't even image how many hours of work
>> it'll cost me to update it to new version :( So there no otto in nixpkgs.
>> Instead of making otto I've started go2nix[6] prototype where I want to use
>> Go toolchain because it can find out all transitive dependencies in one go.
>> It wasn't hard to write a prototype in Go because everthing for it is there.
>> So after a minutes I could automatically generate a list of all dependencies
>> that I need to build any Go binary! Next step was to generate working Nix
>> expression with this list of dependencies that can be used in goPackages.
>> How hard can it be? :) It turned out that it's impossible to do this reusing
>> goPackages in its current state. How could I know that:
>> * github.com/ugorji/go sould be ugorji.go [7]
>> * gopkg.in/flosch/pongo2.v3 should be pongo2 [8]
>> * google.golang.org/api should be google-api-go-client [9]
>> * github.com/odeke-em/google-api-go-client shoul be
>> odeke-em.google-api-go-client [10]
>> * github.com/stathat/go should be stathat [11]
>> ...
>>
>> So there were 2 ways of dealing it:
>> * generate new goPackage without any imports from go-packages.nix and
>> abandon it completely
>> * change go-packages.nix into something that can be readable for something
>> that knows how packages are named in Go land
>> I thought that go-packages.nix, python-packages.nix ...... says that we want
>> to reuse common dependencies somewhere but it should be possible to override
>> some of them if needed. And that was the beginning of [WIP] Rework
>> goPackages with go-packages.nix "converted" into libs.json that was also
>> influenced by temptation to make it possible to automate future updates[12].
>>
>> As a troublemaker I want to involve all interested people to find out what
>> next moves we should do to make:
>> * master branch at least as usable as it was before rework
>> * working with Nix and Go the best experience for both Nixers and Gophers
>>
>> So lets start with the first one. Most comments I see is related to
>> libs.json.
>> Does it mean we want go-packages.nix but in form that can be generated and
>> imported like libs.json or we don't want it at all moving all the
>> dependencies to separate derivations? Or maybe we want go-packages.nix as it
>> was before?
>>
>> --
>> Kamil
>>
>> [1]
>> https://github.com/kamilchm/nixpkgs/pull/3/files#diff-8631dcd3e45d5f2ceaedeb96a23bee6fR2214
>> [2]
>> https://github.com/kamilchm/nixpkgs/pull/3/files#diff-036410e9211b4336186fc613f7200b12R12219
>> [3]
>> https://gist.github.com/kamilchm/bec9483a111babafe965abcec9efb013#file-nix-build-a-otto
>> [4] https://gist.github.com/kamilchm/bec9483a111babafe965abcec9efb013
>> [5] https://github.com/kamilchm/nixpkgs/pull/3
>> [6] https://github.com/kamilchm/go2nix
>> [7]
>> https://github.com/kamilchm/nixpkgs/blob/otto-before/pkgs/top-level/go-packages.nix#L683
>> [8]
>> https://github.com/kamilchm/nixpkgs/blob/otto-before/pkgs/top-level/go-packages.nix#L2070
>> [9]
>> https://github.com/kamilchm/nixpkgs/blob/otto-before/pkgs/top-level/go-packages.nix#L861
>> [10]
>> https://github.com/kamilchm/nixpkgs/blob/otto-before/pkgs/top-level/go-packages.nix#L872
>> [11]
>> https://github.com/kamilchm/nixpkgs/blob/otto-before/pkgs/top-level/go-packages.nix#L2700
>> [12] https://github.com/NixOS/nixpkgs/pull/13819
>>
>>
>> _______________________________________________
>> nix-dev mailing list
>> nix-dev at lists.science.uu.nl
>> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>>