[Nix-dev] Reengineered npm2nix: call for feedback

[Sander van der Burg]

I think most of you probably know it already, but I've written a blog post
about the reengineered implementation, that can be found here:
http://sandervanderburg.blogspot.com/2014/10/deploying-npm-packages-with-nix-package.html

Second, I have some good and some bad news.

Good news is that there were a few people lately, having problems with
deploying certain packages with the old npm2nix (one time there was another
cyclic dependency). I tested the same packages with the new npm2nix and
they seem to deploy just fine without problems. So I think it that the new
approach really solves some very nasty problems and accurately simulates
most of NPM's way of managing dependencies.

The only bad thing so far is that the new approach is a lot slower.  I'm
specifically talking about the time it takes to deploy something with
nix-build or nix-env.

The slowness is basically caused by the fact that for each dependency I
want to include, I have to tell which shared dependencies have been
provided already to correctly simulate the way version ranges are resolved.
I need to run semver for this. Moreover, to also prevent infinite recursion
because of cyclic dependencies, I use a expression generation trick.

I think I already have a few optimizations in mind, but they are difficult
to implement and I might break stuff again. For example, if I'm absolutely
sure there are no cycles in a deployment, I can skip the dependency
generation trick, for example. Question is: how to detect this?

I don't think the slowness is problematic for private projects, but it
might be too expensive to use that for the set of NPM packages part of
Nixpkgs.

On the other hand, I also think that in the future, most packages should be
deployed privately in NPM projects. The main reason why I recommend this is
that version range specifiers do not always resolve to their latest
compatible versions. The actual version used is context specific -- if some
project requires async 0.2.x, the latest conforming version is 0.2.9, but a
shared dependency is 0.2.5, then 0.2.5 is used. That's why you have to
reconsider the complete tree of dependencies for every project.

The only packages that still need to be deployed from Nixpkgs IMHO are
end-user utilities (command-line utilities) and libraries used by non-NPM
projects.

So I'm still thinking a bit about it and what I can do to make the
performance better. In the worst case, I have to throw away the entire
approach and invent something new, if it turns out that the performance is
too problematic.

What do you think?

-- Sander


On Tue, Sep 30, 2014 at 12:13 PM, Sander van der Burg <svanderburg at gmail.com
> wrote:

> Hi everybody,
>
> Since the sprint in Ljubljana until now, I've been working on npm2nix's
> issues. I basically reengineered most of it (e.g. rewritten it to
> JavaScript and modularized it a bit further) and I think the implementation
> is fairly complete now. I've incorporated as much of Shea's functionaility
> into the reengineered version as I could.
>
> The new implementation handles dependencies in such a way that its
> behavior is closer to what NPM does. Moreover, it should also properly cope
> with cyclic dependencies. The way dependencies are resolved is actually
> much more advanced than the original npm2nix implementation. For example,
> it uses semver to make matches between version ranges.
>
> In theory, also cyclic dependencies that match on version ranges should be
> handled properly, although I haven't encountered them yet.
>
> Using the reengineered version is straight forward. You can obtain it from
> my private Git fork:
>
> https://github.com/svanderburg/npm2nix/tree/reengineering
>
> I have updated the README.md file of npm2nix to elaborate a bit more about
> the possible use cases of npm2nix. So I think it wouldn't be a bad idea to
> read it first.
>
> Installation can be done by opening the git checkout, and running:
>
> $ nix-env -f default.nix -iA build
>
> Moreover, you don't need to update Nixpkgs. The new build function is
> actually part of npm2nix.
>
> I've used the reengineered npm2nix on a private project for the company I
> work for, as well as some other utilities and it seems to work fine for me.
>
> Hopefully, you can also try npm2nix on your projects to find out whether
> there any additional issues. :)
>
> Furthermore, I'm still working a blog post to rationalize all the stuff,
> which should give you better insights in the problem, details and the
> choices I made.
>
> Let me know what you think!
>
> Regards,
>
> Sander
>
>
>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.science.uu.nl/pipermail/nix-dev/attachments/20141012/c3510704/attachment-0001.html