[Nix-dev] Fwd: Reminder about nixpkgs branches

[Mathnerd314]

On Thu, Jul 24, 2014 at 11:03 PM, Eelco Dolstra <eelco.dolstra at logicblox.com
> wrote:

> Hi,
>
> On 24/07/14 21:32, Mathnerd314 wrote:
>
> Here it's really more appropriate to use nix-env (i.e. the imperative
> package
> management style), which makes it easy to mix packages from different
> Nixpkgs
> revisions.
>
> The declarative style (environment.systemPackages) doesn't work very well
> for
> this. It's doable if you only want a few packages from one other branch,
> but as
> you point out, it doesn't scale very well.
>


> Yes, mixing modules from different NixOS versions doesn't really work.
> Here it
>
> is easier to have a local branch (e.g. based/rebased on release-14.04) onto
> which you can cherry-pick the changes you need.
>
So in this case, I needed to override the packages used by the modules.
AFAICT there is no way to do that in nix-env, yet here you are telling me
to use nix-env to manage packages. I must have missed that section of the
manual...

>
> > NixOS is inherently a
> > state-centric system; it's not "this change broke my system" but
> "version V of
> > thing X is broken".
>
> I don't know what this means.
>
I believe darcs is pretty much the only example of a "change-based" system,
so this is more a reply to Vladimír Čunát.

Well, nixpkgs-monitor is intended to help package maintainers, not make them
> redundant :-) Maintainers should still do testing before they push a patch
> from
> the monitor.
>
And that's impossible, since there are always architectures and
configurations that maintainers don't have access to. So I say go the other
way: if you can't test the dynamic stuff completely, don't test it at all.
On the other hand nix-instantiate is (at least theoretically)
system-independent so it is reasonable to run it over all packages; it can
check syntax, types, etc. - basically everything static. But the dynamic
stuff, e.g. Hydra's builds and tests, should only run as often as you want
to release, not as often as you want to update a package.

>
> > The problem is that, currently, Nix expressions are used for two
> purposes -
> > building packages and specifying systems. Due to integration in the
> nixpkgs
> > repository, the coupling between these has grown strong. But, as I hope
> my
> > example has shown, these are orthogonal and independent activities.
> Sometimes I
> > want to build packages, e.g. when I want to "build the latest version of
> Firefox
> > from mozilla-central". Other times I want to specify my system, e.g.,
> "use this
> > known-good version of Firefox". If you look closer, you can see the
> difference:
> > in the first case, I am referring to a dynamic, changing process. In the
> second
> > case, I just need a store path. However, currently Nix is using the
> extensional
> > model which intermingles these two.
>
> I don't see the connection with the extensional model.

Since the extensional model only identifies the inputs, it automatically
assumes that any two runs of a derivation are the same.

> It allows you to specify
> a specific version just fine, e.g.
>
>   environment.systemPackages = [ (builtins.storePath
> /nix/store/gv67fwcwbyha77kxsmgcs41baqxh2ysi-firefox-31.0) ];
>
There's no way to refer to "the result of calling fetchurl http://google.com
on July 1st 2013", which can be described with the intensional model (a
hash of the contents, as usual).

>
> (Not the prettiest syntax, but NixOS wasn't really intended for this kind
> of
> binary-only package management.)
>
Well, since that's what all my package paths look like, I guess that's the
problem in a nutshell.

>
> The main advantage of the intensional model is that it allows untrusted
> users to
> fetch binaries from untrusted binary caches.
>
It also allows derivations to fetch or produce volatile or time-dependent
data, which I think is a bigger advantage.

>
> > On the other side, the release process and picking process becomes
> easier. Hydra
> > fetches all of the store paths, and it's just a matter of examining the
> results
> > and picking the builds that seem to work and pass tests.
>
> This is pretty much the furthest you can get from declarative package
> management.

Not really, the furthest you could get would be using no package management
at all (a.k.a. "life without commodities").

> It makes a release the sum of a bunch of packages from different
> Nixpkgs revisions, all built against whatever version of the dependencies
> happened to be the most recently built at the time.

No, they're built with whatever dependencies happened to be *available* at
the time, which is a different story (availability is determined by Hydra;
it can use whichever versions it wants, and the versions it uses are
sha256'd hashed as usual).

> Reproducing such a
> configuration is pretty much impossible.

No, the only commands you're running are of the form "execute the script
<sha256 hash> on <list of files' sha256 hashes>". The only time they're not
reproducible is when they grab external state. If we continue following our
packaging process, then commands which grab external state will be isolated
to a few fetch* derivations and the rest of it will be completely
deterministic (besides the timestamp fluff, which ideally will go away in
https://github.com/NixOS/nixpkgs/pull/2281).

> (This is in fact the classic Linux
> package model, where it is typically doubtful whether all binary packages
> can be
> reproduced from their specifications.)
>
Not at all, because between each build step the result gets passed through
the Nix store and sha256'd. If it's not reproducible you know as soon as
you do a full rebuild, and you exactly where to look to see the script
causing the problem. Except, in the case of fetchurl, I don't think it's a
problem at all, since URL's are often unstable and there's no way to "fix"
them besides switching to a different URL. And if you're going to switch
URL's, which by the way changes the hash of the inputs and is thus
incompatible with the extensional store model without hacks, then you might
as well go all the way and write a URL-generating program. And then why not
make that depend an external service like Google, for convenience? It's all
hashed in the end anyways so all it does it add a few more files to the Nix
store. We had three different bootstrap stdenv's, last time I counted; why
not have 3 different URL-finding bootstrap steps as well?

>
> > After this, my vision is that the nixpkgs repository will only contain
> very
> > high-level instructions, for example "to install firefox: got to mozilla
> ftp
> > directory; load a folder (branch); download tarball; install to $out }".
>
> I assume you mean "fetch the latest version". That's fundamentally
> incompatible
> with the idea of purely functional package management.

So this is where it gets slightly tricky. Most projects have several
branches, e.g. Firefox has Nightly, Aurora, Beta, Mainline, and ESR. Each
branch has a different "latest version".
However, you can also go further than that, e.g. I could specify "the
latest version of
http://ftp.mozilla.org/pub/mozilla.org/firefox/releases/31.0b2/linux-i686/en-GB/firefox-31.0b2.tar.bz2".
This is what we currently have with fetchurl.
In each case, the download is hashed and stored in the Nix store. Once you
have those inputs, then you can start talking about "purely functional".
Before then, though, the term is meaningless; there is no way to implement
a "purely functional downloader". The only difference I'm arguing for is
where the hash is stored; I say store it in Hydra, NixOS, and/or Nix's
database, since they are the tools that can make use of it and decide
whether to proceed with the build, perhaps substituting one package for
another because the user told it to. There is no reason to store the hash
in nixpkgs since it changes so frequently and without notice; the only
thing the hash does there is break builds irreparably.

> It would mean that
> running "nix-build -A firefox" on a specific Nixpkgs revision would give a
> different result at different points in time.

Just like running nix-env -i firefox can give different results at
different points in time.
However, nix-store --realise /nix/store/<...>.drv will give the same result
up to the reproducibility of the derivation, and nix-store --realise
/nix/store/<file> will fail if it cannot download or reproducibly build the
requested file.


> Switching to the intensional model would be a huge undertaking which I
> don't see
> happening soon.
>
Yeah, no disagreement there. Although we might have different definitions
of "not soon"; I've often heard it defined as "never", while I hope it is
closer to "in a few months" in this case.

-- Mathnerd314
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.science.uu.nl/pipermail/nix-dev/attachments/20140725/3f929345/attachment-0001.html