[Nix-dev] Some bug tracker experience and RFC on improvements

[Austin Seipp]

Hi *,

Over the past few days, while I've been waiting for things to compile,
I've been casually triaging the NixOS pull request list and the actual
bug list, adding labels to everything to categorize issues. I think
all of the PRs open as of today are labeled, which is great. The first
few pages of the bug list are also almost labeled, but I haven't gone
much farther.

I want to take a moment emphasize the importance of doing this, and
some thoughts on improving it, based on my experience doing similar
things for other projects. Please note, I'm talking almost exclusively
about the 'nixpkgs' repository.

As many of you may be aware of by now, GitHub has an issue tracker.
It's integrated with the Pull Request capability. It's relatively
simple by design. But this has some knock-on consequences - it has a
relatively anemic ability to extend metadata (e.g. custom issue
fields), and personally I've found the full-text search capabilities
to be relatively poor. Their issue tracker is quite simple like I
said, but what this means is that metadata is vitally important to
triaging bugs and having maintainers get to them easily and in a
timely manner.

For example, I'm generally interested in all kernel bugs, and things
like security bugs and issues so I can get CVEs fixed (I know, I
haven't had much time for this recently.) But I only added the
'kernel' label just on Friday, I think!

Also, I want to know about all the bugs for the 15.05 release that
need to be fixed, which are kernel/systemd bugs. And the ones that are
critical priority, that haven't been fixed. And the ones that have
been fixed, so we can add to the release notes. That way I know what
to do about them.

And, as I'm sure most of you are aware, GitHub's notification system
is not very lean. If you 'watch' a repository, you get spammed for
every update to it! This works OK for small projects where I do
everything - it does not scale very nicely to more people, meaning in
practice I have a GitHub filter in my email with hundreds of unread
things for NixOS subsystems I don't maintain.

All of this makes the need for precise metadata very important. It is
imperative that contributors use time effectively, and doing this
makes it a lot easier. And I think it's really important to ingrain it
in your developers and users to be cleanly about this. I'd really like
to search by metadata to find bugs/issues relevant to me, but that's a
bit hard sometimes.

So there are two things I'd like to note here, to try and alleviate this:

     1) There needs to be a CONTRIBUTING.md file in the Nixpkgs. The
reason for this is that we want it to pop up when you file bugs or
pull requests. For example:
https://github.com/thoughtbot/factory_girl_rails/issues/new

      There is a very clear warning at the top to read about how to
contribute. This can have a quick note that says things like A)
"Please label any bugs you file as you see fit" B) "Format your
messages properly" C) "Brush your teeth before going to bed".

      This doesn't guarantee people follow protocol, but it will
hopefully remind people to file bugs following specific guidelines.

     2) The current issue list is really, really confusing and
haphazard. For example, there is a 'python' label but no 'perl' label,
despite the fact I just labeled 5 perl PRs! And there's no 'systemd'
label, or any label about, say, Xorg, the NixOS installer, etc etc. As
well as no uniformity to color, either :) Also, labels for things like
'high' vs 'low' priority don't exist either (really, these should be
first class concepts)

      So, I'd recommend we take a play from the Rust people and handle
GitHub's simpllistic tracker, and categorize all labels by a
'callsign' - a distinctive single letter - followed by the label name.
For example, a reinvisioning of the current label list with some
additions might look like:

  "Blocker" issues/bugs:
       B-low               -- Low priority
       B-normal          -- (Default) normal priority
       B-high              -- High priority
       B-blocker         -- Blocking bug

  "Infrastructure" issues/bugs:
       I-security               -- Security updates
       I-documentation     -- Documentation
       I-mass-rebuild        -- Massive rebuild

  "Language" issues/bugs:
       L-python                -- Python packages
       L-perl                     -- Perl packages
       L-php                     -- PHP packages
       L-haskell                -- Haskell packages
       L-ruby                    -- Ruby packages

  "Miscellaneous" issues/bugs:
       M-rfc                     -- RFC (AKA 'feedback-requested')
       M-wip                    -- Work-in-progress
       M-question             -- Questions
       M-meta                  -- Meta bugs for other issues.

  "NixOS" issues/bugs:
       N-installer              -- NixOS installer
       N-systemd             -- systemd (NixOS only)
       N-xorg                   -- Xorg/MESA/etc
       N-gnome               -- GNOME packaging
       N-KDE                  -- KDE packaging
       N-kernel                -- Kernel packages

  "Package" issues/bugs:
       P-enhancement     -- Package enhancement, amendment, etc
       P-update                -- Update version package
       P-new                    -- New package

 "System" issues/bugs:
       S-linux                 -- (Default) Linux bugs
       S-darwin              -- Darwin bugs
       S-other                -- Other platforms

  "Invalid/duplicate/wontfix" bugs
       Z-einvalid               -- Invalid issue/bug; -EINVAL
       Z-enotabug             -- Not a bug; -ENOTABUG
       Z-ewontfix              -- Nope; -EWONTFIX
       Z-duplicate             -- Duplicate issue

Note that any issue may combine any of these together. It's also a
much richer set of labels that express more constraints on a bug. A
few things:

  A) There is a specific 'M-meta' to describe metabugs. This should
have no other labels associated, and should simply list other bugs in
a checkbox-list specifying what's blocking them.

  B) Labels are color-coated by *category*, not individually. This
means that all 'P' labels are blue, all Z labels are grey, etc. This
makes things visually clearer as to what they represent, and saves us
recycling colors all the time.

-----------------------------------

Finally, there are some other associated things I was thinking of:

  1) It would also be nice to adopt an official policy of maintainers
for the source tree; e.g. who is responsible for certain NixOS or
Nixpkgs subsystems, or Darwin/FreeBSD support, etc. This makes it
easier to track down the right person, IMO.

  2) It would be nice if we could do a round-robin scheduling or
something of maintainers who have to triage issues, even very very
briefly. Every day one person is elected to just add labels to things,
and set the ticket to a milestone. Some projects have done this in the
past; in practice, I think if we had a random selection of people who
could once a day add labels and set them, and we were diligent, most
days would only see 5-10 issues needing labels. Which is not too hard;
appropriate metadata is pretty easy to infer from just reading an
issue.

-----------------------------------

There are other things I could think of (e.g. milestoning policies for
a release), but that's getting a bit ahead of myself. So I'd like to
get the discussion out there.

FWIW, if people are supportive of this, I'd at least be willing to
author a CONTRIBUTING.md file to the tree, and re-name all the
existing labels, and add others as specified above (pending a final
list some of us can come to an agreement on). That can be done
relatively easily with an overall bet benefit, I think.

-- 
Regards,
Austin - PGP: 4096R/0x91384671