[Nix-dev] Move NodeJS to separate overlay repository?

[zimbatm]

I've been thinking of the subject in general for a while. It's a bit long
so please bear with me.

First here are various personas with different and potentially conflicting
interests:

1. As a nixpkgs user, when I want to install a package, I don't really care
in what language it's implemented in. I shouldn't have to know that my
package is in nodePackages or pythonPackage, ... or search for the right
overlay. I also don't care about libraries unless they allow me to install
a package because it's a required dependency. And finally, I don't want
nixpkgs to be much larger and slow down my `nix-channel --update` because
it's downloading all the haskell, ruby, python, ... library descriptions.

2. As a nixpkgs maintainer, I don't want to maintain a library set where
all the versions work together. Something like stackage for haskell is a
tremendous amount of effort. And they have a strong type system which
certainly helps. Finding the right combination of library versions is a
combinatorial problem. The search space is N factorial and without proper
tooling it's very time intensive. Especially with Go dependencies where N
is the number of libraries times the number of commits.

3. As a developer, I have my own $lang dependency set. Most languages have
tooling that can find the right combination of dependencies using version
constraints already. In cases where at dependency takes a long time to
compile, I want to be able to benefit from a binary cache. If nix can
provide me with the equivalent of stackage for my language I would also be
happy provided I can override some of the versions and the versions aren't
too old in general.

So as you can see, to satisfy (1) and (2) it would make sense to remove the
$langPackages from nixpkgs. But (3) wants a binary cache and (4) wants the
security updates.

As another data point, I created https://github.com/zimbatm/nix-rubygems which
is a sort of nixpkgs overlay for all the ruby gems, created before the
overlay feature. It contains the sha256 of all the rubygems until I last
updated it (24 Nov 2016). When checking out the repo, `du -sh. ` gives me
3.5GB for 720312 gems, that's ~5KB per gem probably due to all the inodes
and the .git repo. I think it's possible to bring it down to ~50MB using
binary packing but I would guess 99% of these gems are not really useful
and would be a waste of compute and storage resource.

So far I think the best solution to improve the life of $lang developer is
to:
a. Work with the $lang package manager so that they compute, distribute and
embed the sha256 into their lockfiles like yarn does. There should be a
$lang native tool that produces the locked-down dependency set which nix
can read as well. It means the developer doesn't *have* to have nix
installed, unless he wants the binary cache goodies.
b. Make it easy to setup their own binary cache, integrated with their CI,
and developer re-use it. I think that's an important step for any
organisation who wants to adopt Nix.
c. It's not really clear how yet but I believe nix could become useful for
$lang to get automated reports on build failures and non-reproducible
builds. If we can become that foundation we'll see much better adoption in
general.

I'm not saying that overlays are not useful, potentially we can also have
overlays built by hydra just for the binary cache and (c). It's just some
thoughts. Well done for reading until here :)

On Sat, 11 Mar 2017 at 17:29 David Izquierdo <thecofee at gmail.com> wrote:

> Couldn't this become a kind of dependency hell? I remember from Gentoo
> that having dependencies across overlays is not a fun problem to solve.
> However, Exherbo mostly solved this by having overlays be analogous to
> packages, they become a kind of dependency that must be explicitly
> user-solved before continuing. In the default ebuild tree they have
> indexes of every package in each registered overlay.
>
>
> I think this should only be done if there's nothing anywhere else
> depending on a derivation from this overlay-to-be.
>
>
> On 11/03/17 09:12, Wout Mertens wrote:
> > Hi all,
> >
> > now that we got these wonderful overlays as a Chrismas gift (
> > http://lists.science.uu.nl/pipermail/nix-dev/2016-December/022386.html),
> I
> > was wondering if we can move some things out of nixpkgs into their own
> > repos.
> >
> > There are a few package groups that I believe are not used in NixOS core
> > (boot, containers, ...) and are not updated as much as they could be.
> Node,
> > Haskell, others?
> >
> > Specifically, nodePackages is always out of date, and it would be nice if
> > there could be a repository or maybe just a local process that updates
> them
> > separately and adds a lot more builds.
> >
> > Furthermore, building node packages from scratch is ok, because that's
> what
> > npm does anyway. So any caching offered by Hydra is a bonus, not a
> > necessity.
> >
> > For example, maybe the overlay could be auto-generated every hour from
> npm?
> > And then some service fetches it to /etc/nixos/overlays/nodePackages?
> >
> > Does this seem like a fair assessment? What should be done to achieve
> this?
> >
> > Wout.
> >
> >
> >
> > _______________________________________________
> > nix-dev mailing list
> > nix-dev at lists.science.uu.nl
> > http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
> _______________________________________________
> nix-dev mailing list
> nix-dev at lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.science.uu.nl/pipermail/nix-dev/attachments/20170311/64ac7c84/attachment-0001.html>