[Nix-dev] Reminder about nixpkgs branches

[Mathnerd314]

On Thu, Jul 24, 2014 at 12:52 PM, Vladimír Čunát <vcunat at gmail.com> wrote:

>
>  On 07/23/2014 11:29 AM, Mathijs Kwik wrote:
>>>
>> If stuff causes regressions, revert it and move it to a feature-branch
>> to debug.
>>
>
> Perhaps. I'm not very clear about that. Typically I first try to fix the
> regressions within days, if possible.
>
> Note that this revert+move workflow tends to hit a problem with unclear
> semantics of reverts+merges. The catch is that in the case of *temporarily*
> undoing some changes, people typically understand revert the other way than
> practically all VCS (including git), which leads to surprises. IMO it's
> also connected to being state-centric instead of (arguably more natural)
> change-centric approach.
>
> Details:
> https://raw.githubusercontent.com/tonyg/revctrl.org/master/output-raw/Rollback
>
> So let me describe a little of my recent activities. I wanted the firefox
from
https://github.com/NixOS/nixpkgs/commit/872860e6de8e460fb11a79c46b8092bdcff35da8,
the quassel from ~/nixpkgs, and the rest of the system from the latest
hydra channel; I ended up making nixpkgs checkouts for each of those,
hard-coding their paths in import statements, and manually adding each
package (and some of their dependencies) to packageOverrides. But even that
wasn't enough, because I also needed to override the NixOS module for
quassel; unfortunately it seems the list of imports is hard-coded and there
is no moduleOverride function. So, I made yet another nixpkgs checkout, at
exactly the revision of the hydra channel, with my module changes on top,
and set NIX_PATH to point there.

Note that, throughout this process, I was *creating* and *picking*, but I
never needed to revert a commit. I just checked out older versions. I did
end up trying out a single rollback, when I was looking at Quassel's NixOS
module; however, it was completely useless, as the problem was in systemd's
unit dependencies and thus *all* of my versions were broken. NixOS is
inherently a state-centric system; it's not "this change broke my system"
but "version V of thing X is broken".

However, to set up my system the way I wanted it (i.e., so that all my
programs worked), I did need 4 different nixpkgs checkouts. And I've only
been using NixOS for a few weeks; as I set up more programs and services
I'm probably going to need even more than that. Looking at
https://github.com/MarcWeber/nixpkgs/branches/active it seems he has almost
40 branches. His workflow is slightly different from mine because he uses
TopGit instead of letting Nix do most of the work, but the branch-explosion
is similar. When we start using nixpkgs-monitor, the problem is going to
get even worse: a steady, near-realtime stream of patches that bump
versions, URL's, and hashes, with no QA or error checking at all. We
already have this, except not automated, with scripts like hackage2nix.
What branches should these updates go to? AFAIK there is no good answer.
They can't go to master or staging, because they might break Hydra's
builds. A new topic branch per patch means we get several thousand
branches, i.e. the branch explosion again. There are 126 open pull
requests, and probably hundreds more sitting in offline git repositories;
as the project grows this will presumably get worse. The branch approach
doesn't scale; if I do a lot of version-picking, all of these nixpkgs
checkouts will end up in my configuration.nix! The checkouts are a waste of
disk space and contain irrelevant information. Not only that, they're
fragile; if I change the checkout, then the next time I do a nixos-rebuild
the changes get picked up. It's basically a hack, hard-coding all of these
checkouts into Nix.

The problem is that, currently, Nix expressions are used for two purposes -
building packages and specifying systems. Due to integration in the nixpkgs
repository, the coupling between these has grown strong. But, as I hope my
example has shown, these are orthogonal and independent activities.
Sometimes I want to build packages, e.g. when I want to "build the latest
version of Firefox from mozilla-central". Other times I want to specify my
system, e.g., "use this known-good version of Firefox". If you look closer,
you can see the difference: in the first case, I am referring to a dynamic,
changing process. In the second case, I just need a store path. However,
currently Nix is using the extensional model which intermingles these two.
For me this is most obvious in fetchurl; our current fetchurl is basically
a hard-coded store path, with a special exception of "fixed-output
derivations" hacked into Nix. With an intensional store model, that goes
away; a fetchurl is literally "fetchurl.sh <url>" and nothing more. If the
URL breaks, then the store path changes and you can just keep using the old
store path. If the content moves to a different URL, you can just change
the URL and the content hash is still the same, with no fixed-output hack
needed. And of course these advantages apply in other cases besides network
resources, e.g. system time dependency, cosmic radiation, evil
hackers/crackers, etc.

On the other side, the release process and picking process becomes easier.
Hydra fetches all of the store paths, and it's just a matter of examining
the results and picking the builds that seem to work and pass tests. A
release is basically a list of known-good store paths; there should be no
dependency at all on the source NixExpr's, just on the derivations. The
OpenSSL problem goes away, because now patching binaries to point to a
different store path is a required operation (it's used for storing
derivations in the first place!) rather than a quick sed hack. There's
still the question of the actual command syntax for saying "build this
tarball with these dependencies" and how/where to store the path-rewriting
operation, but these are basically just bikeshedding. Regardless, it should
be Hydra or NixOS and their respective configurations that pick "stdenv
uses gcc 4.9", not nixpkgs. We could share the configuration and call it
"nixreleases" or something. The other thing this allows is using e.g.
Guix's GCC, which again currently requires a bunch of hacks.

After this, my vision is that the nixpkgs repository will only contain very
high-level instructions, for example "to install firefox: got to mozilla
ftp directory; load a folder (branch); download tarball; install to $out
}". This will mean that the majority of updates (stable or unstable) will
be completely automatic and invisible from a nixpkgs perspective, which
means that the repository explosion will go away. We can merge in projects
like hackage2nix and others since they will not be lost in the update
noise. The only development that will happen in nixpkgs will be adding new
download sources and fixing build scripts. One master branch should be
sufficient for that, modulo forks or global refactorings. Hydra/NixOS/Nix
will do everything else - picking versions, running programs,
calculating/storing hashes, etc.

Anyways, I filed https://github.com/NixOS/nix/issues/296 a week ago, and
received no response besides Dolstra tagging it as "feature" and "Nix 2.0".
So my conclusion is that nobody actually cares enough for this to be
implemented now, besides possibly me (I'm already doing a GSOC; two
projects at once is probably not a good idea). But I could be wrong, please
convince people to work on #296 if you want to remove the hacks and get
these nice features. :-)

-- Mathnerd314
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.science.uu.nl/pipermail/nix-dev/attachments/20140724/e7c8df20/attachment-0001.html