[Nix-dev] Simplify node packages?

[Sander van der Burg]

>
> I have read the articles from Sander van der Burg, but they don't seem to
> explain why we went with "one derivation per library", which requires us to
> manually symlink things into the node_modules folder.
>

I'm not sure if you have read my blog articles correctly, but what node2nix
currently does is one derivation per application and statically bundles all
NPM dependencies with it. The old approach that npm2nix used to implement
was a derivation per dependency and symlinking between them. As explained
in this blog post:
http://sandervanderburg.blogspot.com/2016/02/managing-npm-flat-module-installations.html
this works for many packages, but does not reliably work in all cases.
Moreover, it does not work with packages having cyclic dependencies.

* A simple approach to packaging NPM applications would be to split the
> package for an NPM application into two:
>   - fetch: let NPM download all the dependencies. it should(?) be possible
> to make this a fixed-output derivation, as long as NPM downloads the same
> set of dependencies each time
>   - build: use the cache from the fetch phase, should require no network
> access
>

This might be an alternative approach worth investigating. However, dealing
with NPM's conflicting dependency management is not that simple -- for
example, there are version specifiers always triggering network
communication (such as version specifiers referring to tags or HTTP URLs),
regardless whether a version has already been provided or not. Furthermore,
there are dependencies that bundle C code with them requiring them to be
source compiled e.g. with gcc. Running NPM outside a Nix builder
environment first and "importing" them into Nix through a fixed-output
derivation does not reliably work with such packages.

Well, perhaps that's the algorithm it uses right now, but I feel that we
>> depend very much on implementation details of npm here, which IMO is not a
>> good solution.
>> If we get upstream to support enough for our use case, the solution
>> should be much more stable.
>>
>
It would obviously be great if we could convince the NPM developers to
change/reconsider some of their "design choices", but I think this will be
quite difficult. One of the reasons I have given a talk at FOSDEM 2017
about this subject is to explain some of the undesired implications of NPM
in the hope that somebody would look at it. However, as with many open
source projects, it is difficult to change someone's mind/opinion.
Moreover, even if there are people willing to change, there is still the
argument that legacy applications need to be supported.

I think this is the best way forward. Yarn is *much* better than npm and is
> being enthusiastically adopted by the node community. Whatever
> compatibility issues that currently exist with yarn will soon be resolved.
> (Nix users have basically no influence, but package maintainers can't
> afford to ignore yarn users.)
>

Now yarn (and other approaches such as ied) do obviously a much better job
than NPM in some aspects. Unfortunately, as of today, NPM remains the
de-facto standard for Node.js package deployment, and I'm not sure if this
will ever change.

The purpose of node2nix is not to be "the better NPM" -- it's only
objective is to integrate NPM packages and development projects into the
Nix ecosystem with minimal disruptions. Ideally, whenever an NPM project
works, node2nix should work as well. (Obviously: this experience is not
always perfect, but we try as hard as we can).

Finally, I'd like to note that the some of the concepts that NPM (and in
some extent yarn) implement are fundamentally different and somewhat
conflicting with how Nix does things. For example, what we ideally like to
see is that dependencies can be shared. For example, when a project depends
on lodash-4.17.4 we ideally only want one copy of it in the store that all
dependent projects can use. Unfortunately, the way NPM composes packages is
context-dependent (e.g. in some scenario's lodash's dependencies resolve to
other versions depending on which common dependencies are already used in
the project). As a result, sharing NPM package dependencies among
projects/application will not work. Because this is an NPM/CommonJS
limitation, it is a limitation in Nix as well.

The latter aspect can only be solved if NPM makes their dependency
addressing and composition facilities better.


On Tue, Apr 25, 2017 at 10:37 AM, Profpatsch <mail at profpatsch.de> wrote:

> On 17-04-25 08:20am, Benno Fünfstück wrote:
> > If we get upstream to support enough for our use case, the solution
> should
> > be much more stable.
>
> Upstream support might be helpful,
> but that’s a wholly different beast.
>
> > Hmm, so perhaps in we can unpack the tarballs already in `phase 1` and
> tell
> > yarn to just look in $DIR for unpacked tarballs? Yarn should already have
> > most of the code that is required for correct symlinking, no?
>
> The code for correct symlinking is <4 lines,
> maybe a few more for manpages and binaries.
>
> I’d argue integrating it into yarn is even more frickle:
> 1. With yarn there’s now two implementations of npm packages.
> 2. We have to invest into lobbying so our integration
>    isn’t broken by upstream sometimes. Then lobby again.
> 3. You’d definitely have to change yarn to have
>    an offline mode where it doesn’t try to download stuff.
>    I’m pretty sure upstream doesn’t care about that very much.
>
> These are the problem points I see.
>
> --
> Proudly written in Mutt with Vim on NixOS.
> Q: Why is this email five sentences or less?
> A: http://five.sentenc.es
> May take up to five days to read your message. If it’s urgent, call me.
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.science.uu.nl/pipermail/nix-dev/attachments/20170425/d3c7526a/attachment-0001.html>