[Nix-dev] Rework goPackages

Wed Aug 3 08:25:23 CEST 2016

Hi,

I refactored all Go applications, moving from deps.json to deps.nix and
encapsulating program dependecies instead of single shared libs.json.
I'm waiting for feedback in https://github.com/NixOS/nixpkgs/pull/17254
especially from Nix maintainers ;)

--
Kamil

2016-06-21 8:20 GMT+02:00 Scott Devoid <sdevoid at gmail.com>:

> Hope I didn't kill the thread. :-(
>
> The TL;DR of my post was that I think go2nix is great and needed for
> many Go projects that don't vendor go-dependencies. But, from my Go
> development experience, one shouldn't try to boil-the-ocean and solve
> all Go packaging for everything.
>
> I've also started a PR to add vendoring to github.com/hashicorp/otto. [1]
>
> Cheers,
> ~ Scott
>
> [1] https://github.com/hashicorp/otto/pull/519
>
> On Fri, Jun 17, 2016 at 10:07 AM, Scott Devoid <sdevoid at gmail.com> wrote:
> > Hi Kamil,
> >
> > First thank you for working on this! Having go2nix as a tool certainly
> > makes packaging Go programs much easier! I don't do a lot of Nix
> > packaging but I do write Go software---so maybe I have a different
> > perspective on this. Anyhow here are some thoughts:
> >
> > - Many Go software projects try to be careful to vendor their
> > dependencies. For example, the Caddy server does this and it works
> > very well. Nix only needs to record the top-level package's SHA1 and
> > use fetchFromGithub + buildGoPacakge [1]. No need to recursively
> > expose all dependencies.
> >
> > - An exception to this rule are projects that make use of C or C++
> > linked libraries. [2] For these you would need to add explicit
> > packages.
> >
> > - Then I take a look at Otto's Makefile and the updatedeps target [3]
> > and I am shocked! This is a terrible practice that the Hashicorp folks
> > should fix. They do things correctly with Vault so I don't see why
> > they can't fix it here. In any case, here go2nix is awesome since it
> > can effectively discover each dependency (tooling around the `go list`
> > command and the contents of GOPATH) and produce Nix derivations for
> > each component.
> >
> > - However, putting all of these in a "flat" go-packages.nix seems
> > incorrect. At the very least, two Go programs may require different
> > versions of the same package. And, as you mention, there's the
> > question of how to consistently name these packages. This is even more
> > complicated since packages may be named one thing but reside somewhere
> > else. And some packages get "folded in" to the standard library with
> > subsequent Go releases. [4] Or terrible redirect "services" like
> > gopkg.in [5]
> >
> > - What's the goal here? I would expect that Nix has packages for
> > 'major' Go programs and tooling to make it easy to package new
> > programs or update releases. The tooling should be designed so that an
> > update to one program doesn't break or alter dependencies for other
> > programs. I think this implies that the tooling produces a
> > 'go-libs.json' per-program as opposed to a single top-level
> > go-packages.nix file. This might cause redundant downloads of common
> > dependencies but that cost seems relatively minor compared to the
> > organizational headache that a flat-namespace would require.
> >
> > ~ Scott
> >
> > [1]
> https://github.com/NixOS/nixpkgs/blob/master/pkgs/servers/caddy/default.nix
> > [2] https://golang.org/cmd/cgo
> > [3] https://github.com/hashicorp/otto/blob/master/Makefile#L35
> > [4] See golang.org/x/net/context -> net/context in go1.7
> > [5] http://labix.org/gopkg.in
> >
> > On Fri, Jun 17, 2016 at 6:55 AM, Kamil Chmielewski <kamil.chm at gmail.com>
> wrote:
> >> Hi,
> >>
> >> few days ago I started a PR called [WIP] Rework goPackages
> >> https://github.com/NixOS/nixpkgs/pull/16017 which is now on master and
> can
> >> leads to confusion for everyone creating goPackages before. I was also a
> >> little surprised how fast it was merged but I thought it was the effect
> of
> >> how many people feel the same pain as me, trying to manage or introduce
> new
> >> goPackages so there wasn't a lot of discussion there. But now it starts
> in
> >> few places and I think that this will be better place to make decisions
> >> about Go packaging in Nix.
> >>
> >> I want to start with the motivation which led me to this topic. Few
> months
> >> ago I wanted to introduce Otto (https://www.ottoproject.io/) to Nix. I
> was
> >> starting with Nix back then i tried to do things as they were in
> >> go-packages.nix.
> >> So I created otto as buildFromGithub[1] and use its bin output in
> >> all-packages.nix[2]. Next nix-build -A otto and I got 23 dependencies
> >> missing[3]. So I started to add dependencies manually doing  nix-build
> -A
> >> otto tens of times because each of them depends on new things that `go
> >> build` tells me only when I add new sources for build... [4]
> >> After many hours of doing nix-build and new edits in go-packages.nix I
> ended
> >> up with otto done in 400 lines of Nix [5] I don't want do maintain.
> Besides
> >> new packages I also need to change version of things that were there
> before,
> >> potentially breaking other things. I can't even image how many hours of
> work
> >> it'll cost me to update it to new version :( So there no otto in
> nixpkgs.
> >> Instead of making otto I've started go2nix[6] prototype where I want to
> use
> >> Go toolchain because it can find out all transitive dependencies in one
> go.
> >> It wasn't hard to write a prototype in Go because everthing for it is
> there.
> >> So after a minutes I could automatically generate a list of all
> dependencies
> >> that I need to build any Go binary! Next step was to generate working
> Nix
> >> expression with this list of dependencies that can be used in
> goPackages.
> >> How hard can it be? :) It turned out that it's impossible to do this
> reusing
> >> goPackages in its current state. How could I know that:
> >> * github.com/ugorji/go sould be ugorji.go [7]
> >> * gopkg.in/flosch/pongo2.v3 should be pongo2 [8]
> >> * google.golang.org/api should be google-api-go-client [9]
> >> * github.com/odeke-em/google-api-go-client shoul be
> >> odeke-em.google-api-go-client [10]
> >> * github.com/stathat/go should be stathat [11]
> >> ...
> >>
> >> So there were 2 ways of dealing it:
> >> * generate new goPackage without any imports from go-packages.nix and
> >> abandon it completely
> >> * change go-packages.nix into something that can be readable for
> something
> >> that knows how packages are named in Go land
> >> I thought that go-packages.nix, python-packages.nix ...... says that we
> want
> >> to reuse common dependencies somewhere but it should be possible to
> override
> >> some of them if needed. And that was the beginning of [WIP] Rework
> >> goPackages with go-packages.nix "converted" into libs.json that was also
> >> influenced by temptation to make it possible to automate future
> updates[12].
> >>
> >> As a troublemaker I want to involve all interested people to find out
> what
> >> next moves we should do to make:
> >> * master branch at least as usable as it was before rework
> >> * working with Nix and Go the best experience for both Nixers and
> Gophers
> >>
> >> So lets start with the first one. Most comments I see is related to
> >> libs.json.
> >> Does it mean we want go-packages.nix but in form that can be generated
> and
> >> imported like libs.json or we don't want it at all moving all the
> >> dependencies to separate derivations? Or maybe we want go-packages.nix
> as it
> >> was before?
> >>
> >> --
> >> Kamil
> >>
> >> [1]
> >>
> https://github.com/kamilchm/nixpkgs/pull/3/files#diff-8631dcd3e45d5f2ceaedeb96a23bee6fR2214
> >> [2]
> >>
> https://github.com/kamilchm/nixpkgs/pull/3/files#diff-036410e9211b4336186fc613f7200b12R12219
> >> [3]
> >>
> https://gist.github.com/kamilchm/bec9483a111babafe965abcec9efb013#file-nix-build-a-otto
> >> [4] https://gist.github.com/kamilchm/bec9483a111babafe965abcec9efb013
> >> [5] https://github.com/kamilchm/nixpkgs/pull/3
> >> [6] https://github.com/kamilchm/go2nix
> >> [7]
> >>
> https://github.com/kamilchm/nixpkgs/blob/otto-before/pkgs/top-level/go-packages.nix#L683
> >> [8]
> >>
> https://github.com/kamilchm/nixpkgs/blob/otto-before/pkgs/top-level/go-packages.nix#L2070
> >> [9]
> >>
> https://github.com/kamilchm/nixpkgs/blob/otto-before/pkgs/top-level/go-packages.nix#L861
> >> [10]
> >>
> https://github.com/kamilchm/nixpkgs/blob/otto-before/pkgs/top-level/go-packages.nix#L872
> >> [11]
> >>
> https://github.com/kamilchm/nixpkgs/blob/otto-before/pkgs/top-level/go-packages.nix#L2700
> >> [12] https://github.com/NixOS/nixpkgs/pull/13819
> >>
> >>
> >> _______________________________________________
> >> nix-dev mailing list
> >> nix-dev at lists.science.uu.nl
> >> http://lists.science.uu.nl/mailman/listinfo/nix-dev
> >>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.science.uu.nl/pipermail/nix-dev/attachments/20160803/a30c8e07/attachment.html>