[Nix-dev] Keeping nixpkgs up to date

[Alexander Kjeldaas]

On Tue, Sep 2, 2014 at 11:33 AM, Michael Raskin <7c6f434c at mail.ru> wrote:

> >>       1 not sure if 0.8.0.3 is relevant on linux
> >>       1 patch OK
> >>       2 hard to say
> >>       2 not linked on homepage
> >>       3 build failure
> >>      11 patch ok
> >>      12 build pending
> >>      23 irrelevant
> >>      29 no patch
> >>
> >>
> >What does "no patch" mean? (a new release without a patch or a tarball
> >makes no sense)
>
> We know some other distribution has a newer version but cannot
> automatically patch out expression to fetch the new tarball.
>

Thanks for the explanation.  I think analyzing .deb or .rpm files is doable.

A program that takes v1.deb and v2.deb can output the list of patches added
and removed, as well as whether the meta-data for the package (scripts/* ?)
was changed.

For both of these packages it is fairly easy to generate a fully patched
source directory which can be diffed.  Alternatively the patches are also
kept in a structured manner and can be extracted.

Then only if the meta-data was changed is it hard to automate.

I can see a need for adding a bit more meta-data into nixpkgs when
"upstream" is no longer the author of the software, but "upstream" is a
collection of distributions as well.  For example, the latest .deb or .rpm
that has been "processed" or "consumed" so that only new patches relative
to those bases should be considered by the monitor.



>
> >"irrelevant" could be baked into monitor.nixos.org, if it is non-security
> >related.
>
> These should be just ignored — some of these are trivial to automate.
>
> >"not linked on homepage" means that monitor.nixos.org missed it?
>
> No, it misses way more.
>
> There are multiple packages where some distributions package versions
> not acknowledged by software's homepage.
>
> Think gcc-2.96, but with less critical software.
>
>
monitor.nixos.org could also issue a "warning" when the number of commits
from the last tag or release starts getting significant.  Like "this
package has 200 commits since last release.. time to investigate?".


> >"hard to say"??
>
> Looking at two versions it is not always obvious if the package is
> newer. I didn't want to spend inordinate effort but I didn't want to
> hide throwing data out.
>

Yes, this is the core of what  maintainer needs to do.  Figure out if this
is a new version, whether it is important, security-related etc.


>
> >It seems like you have 3 "build failures" and 2 "not linked on homepage"
> >that could not be automated.  Am I reading your numbers correctly?
>
> Nope, our real problem is 30 «no patch» + 3 build failures + (expected)
> 3 out of 12 pending builds failing — vs. 12 patch OK + (expected) 9 more
> patches from pending builds.
>

I think those numbers are pretty good.  It seems like getting support for
digesting .deb .rpg, gentoo, arch linux or whoever has the most up-to-date
packages is important, but the patches exist, they are just a bit hard to
get at.

Here is a sketch of how this can be realized:  Create a large git-tree
(with submodules for each nixpkgs package) that contains all source code
for all distributions.  Whenever the monitor finds a new .deb, .rpm or
similar it will unpack and commit to the distribution's branch with a
proper tag.  Then another tool could create patches based on that.  The
gist is to use git as the shared state for monitor.nixos.org.  Using git as
the shared state makes it possible to create lots of smaller tools that can
be used by developers directly without having to bake everything into the
monitor (and being dependent on it being up - that's the hackage failure
mode).  The files are already (mostly) stored on github already, so this
doesn't cost much in terms of storage for github.

In the meta-data for packages in nixpkgs, references into that git tree can
be used as distribution-neutral meta-data that describes which patches have
been processed.

Alexander
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.science.uu.nl/pipermail/nix-dev/attachments/20140902/104890bf/attachment.html