[Nix-dev] Reengineered npm2nix: call for feedback

[Rok Garbas]

Quoting Sander van der Burg (2014-10-12 14:21:17)
> I think most of you probably know it already, but I've written a blog post
> about the reengineered implementation, that can be found here: http://
> sandervanderburg.blogspot.com/2014/10/
> deploying-npm-packages-with-nix-package.html
> 
> Second, I have some good and some bad news.
> 
> Good news is that there were a few people lately, having problems with
> deploying certain packages with the old npm2nix (one time there was another
> cyclic dependency). I tested the same packages with the new npm2nix and they
> seem to deploy just fine without problems. So I think it that the new approach
> really solves some very nasty problems and accurately simulates most of NPM's
> way of managing dependencies.
> 
> The only bad thing so far is that the new approach is a lot slower.  I'm
> specifically talking about the time it takes to deploy something with nix-build
> or nix-env.
> 
> The slowness is basically caused by the fact that for each dependency I want to
> include, I have to tell which shared dependencies have been provided already to
> correctly simulate the way version ranges are resolved. I need to run semver
> for this. Moreover, to also prevent infinite recursion because of cyclic
> dependencies, I use a expression generation trick.
> 
> I think I already have a few optimizations in mind, but they are difficult to
> implement and I might break stuff again. For example, if I'm absolutely sure
> there are no cycles in a deployment, I can skip the dependency generation
> trick, for example. Question is: how to detect this?
> 
> I don't think the slowness is problematic for private projects, but it might be
> too expensive to use that for the set of NPM packages part of Nixpkgs.
> 
> On the other hand, I also think that in the future, most packages should be
> deployed privately in NPM projects. The main reason why I recommend this is
> that version range specifiers do not always resolve to their latest compatible
> versions. The actual version used is context specific -- if some project
> requires async 0.2.x, the latest conforming version is 0.2.9, but a shared
> dependency is 0.2.5, then 0.2.5 is used. That's why you have to reconsider the
> complete tree of dependencies for every project.
> 
> The only packages that still need to be deployed from Nixpkgs IMHO are end-user
> utilities (command-line utilities) and libraries used by non-NPM projects.
> 
> So I'm still thinking a bit about it and what I can do to make the performance
> better. In the worst case, I have to throw away the entire approach and invent
> something new, if it turns out that the performance is too problematic.
> 
> What do you think?
> 
> -- Sander
> 
> 
> On Tue, Sep 30, 2014 at 12:13 PM, Sander van der Burg <svanderburg at gmail.com>
> wrote:
> 
>     Hi everybody,
> 
>     Since the sprint in Ljubljana until now, I've been working on npm2nix's
>     issues. I basically reengineered most of it (e.g. rewritten it to
>     JavaScript and modularized it a bit further) and I think the implementation
>     is fairly complete now. I've incorporated as much of Shea's functionaility
>     into the reengineered version as I could.
> 
>     The new implementation handles dependencies in such a way that its behavior
>     is closer to what NPM does. Moreover, it should also properly cope with
>     cyclic dependencies. The way dependencies are resolved is actually much
>     more advanced than the original npm2nix implementation. For example, it
>     uses semver to make matches between version ranges.
> 
>     In theory, also cyclic dependencies that match on version ranges should be
>     handled properly, although I haven't encountered them yet.
> 
>     Using the reengineered version is straight forward. You can obtain it from
>     my private Git fork:
> 
>     https://github.com/svanderburg/npm2nix/tree/reengineering
> 
>     I have updated the README.md file of npm2nix to elaborate a bit more about
>     the possible use cases of npm2nix. So I think it wouldn't be a bad idea to
>     read it first.
> 
>     Installation can be done by opening the git checkout, and running:
> 
>     $ nix-env -f default.nix -iA build
> 
>     Moreover, you don't need to update Nixpkgs. The new build function is
>     actually part of npm2nix.
> 
>     I've used the reengineered npm2nix on a private project for the company I
>     work for, as well as some other utilities and it seems to work fine for me.
> 
>     Hopefully, you can also try npm2nix on your projects to find out whether
>     there any additional issues. :)
> 
>     Furthermore, I'm still working a blog post to rationalize all the stuff,
>     which should give you better insights in the problem, details and the
>     choices I made.
> 
>     Let me know what you think!
> 
>     Regards,
> 
>     Sander
> 
> 
> 
> 
> 
> 

hi Sander,

sorry for slow response, i was too busy last weeks.

i've tried your fork of npm2nix on current project. my package.json looks like:

{
  "name": "XXX",
  "version": "0.0.1",
  "dependencies": {
    "css-loader": "~0.9.0",
    "domready": "~1.0.7",
    "file-loader": "~0.7.2",
    "jsx-loader": "~0.11.2",
    "less-loader": "~0.7.7",
    "lodash": "~2.4.1",
    "react": "~0.11.2",
    "react-bootstrap": "~0.12.0",
    "react-router": "~0.9.2",
    "script-loader": "~0.6.0",
    "socket.io-client": "~0.9.17",
    "style-loader": "~0.8.1",
    "url-loader": "~0.5.5",
    "webpack": "~1.4.4"
  }
}

generating default.nix, node-env.nix and registry.nix works but when trying to
build it using `nix-build -A build` i get the following error:
https://gist.github.com/garbas/1ae5f5cf688561d03be5

i remember in the past we had problems with fsevents but i can remember how we
fixed them.


--
Rok Garbas - http://www.garbas.si