[Nix-dev] Reengineered npm2nix: call for feedback

[Sander van der Burg]

Ah I've seen somebody else reporting this issue as well.

So does the fsevents package compile correctly with the old npm2nix? If I
see the output, apparently some native library dependency is missing that
we somehow have to provide by other means.

Tomorrow, when I'm in the office (where I have access to a Mac machine) I
can check this.

Regards,

Sander


On Sun, Oct 12, 2014 at 8:04 PM, Rok Garbas <rok at garbas.si> wrote:

> Quoting Sander van der Burg (2014-10-12 14:21:17)
> > I think most of you probably know it already, but I've written a blog
> post
> > about the reengineered implementation, that can be found here: http://
> > sandervanderburg.blogspot.com/2014/10/
> > deploying-npm-packages-with-nix-package.html
> >
> > Second, I have some good and some bad news.
> >
> > Good news is that there were a few people lately, having problems with
> > deploying certain packages with the old npm2nix (one time there was
> another
> > cyclic dependency). I tested the same packages with the new npm2nix and
> they
> > seem to deploy just fine without problems. So I think it that the new
> approach
> > really solves some very nasty problems and accurately simulates most of
> NPM's
> > way of managing dependencies.
> >
> > The only bad thing so far is that the new approach is a lot slower.  I'm
> > specifically talking about the time it takes to deploy something with
> nix-build
> > or nix-env.
> >
> > The slowness is basically caused by the fact that for each dependency I
> want to
> > include, I have to tell which shared dependencies have been provided
> already to
> > correctly simulate the way version ranges are resolved. I need to run
> semver
> > for this. Moreover, to also prevent infinite recursion because of cyclic
> > dependencies, I use a expression generation trick.
> >
> > I think I already have a few optimizations in mind, but they are
> difficult to
> > implement and I might break stuff again. For example, if I'm absolutely
> sure
> > there are no cycles in a deployment, I can skip the dependency generation
> > trick, for example. Question is: how to detect this?
> >
> > I don't think the slowness is problematic for private projects, but it
> might be
> > too expensive to use that for the set of NPM packages part of Nixpkgs.
> >
> > On the other hand, I also think that in the future, most packages should
> be
> > deployed privately in NPM projects. The main reason why I recommend this
> is
> > that version range specifiers do not always resolve to their latest
> compatible
> > versions. The actual version used is context specific -- if some project
> > requires async 0.2.x, the latest conforming version is 0.2.9, but a
> shared
> > dependency is 0.2.5, then 0.2.5 is used. That's why you have to
> reconsider the
> > complete tree of dependencies for every project.
> >
> > The only packages that still need to be deployed from Nixpkgs IMHO are
> end-user
> > utilities (command-line utilities) and libraries used by non-NPM
> projects.
> >
> > So I'm still thinking a bit about it and what I can do to make the
> performance
> > better. In the worst case, I have to throw away the entire approach and
> invent
> > something new, if it turns out that the performance is too problematic.
> >
> > What do you think?
> >
> > -- Sander
> >
> >
> > On Tue, Sep 30, 2014 at 12:13 PM, Sander van der Burg <
> svanderburg at gmail.com>
> > wrote:
> >
> >     Hi everybody,
> >
> >     Since the sprint in Ljubljana until now, I've been working on
> npm2nix's
> >     issues. I basically reengineered most of it (e.g. rewritten it to
> >     JavaScript and modularized it a bit further) and I think the
> implementation
> >     is fairly complete now. I've incorporated as much of Shea's
> functionaility
> >     into the reengineered version as I could.
> >
> >     The new implementation handles dependencies in such a way that its
> behavior
> >     is closer to what NPM does. Moreover, it should also properly cope
> with
> >     cyclic dependencies. The way dependencies are resolved is actually
> much
> >     more advanced than the original npm2nix implementation. For example,
> it
> >     uses semver to make matches between version ranges.
> >
> >     In theory, also cyclic dependencies that match on version ranges
> should be
> >     handled properly, although I haven't encountered them yet.
> >
> >     Using the reengineered version is straight forward. You can obtain
> it from
> >     my private Git fork:
> >
> >     https://github.com/svanderburg/npm2nix/tree/reengineering
> >
> >     I have updated the README.md file of npm2nix to elaborate a bit more
> about
> >     the possible use cases of npm2nix. So I think it wouldn't be a bad
> idea to
> >     read it first.
> >
> >     Installation can be done by opening the git checkout, and running:
> >
> >     $ nix-env -f default.nix -iA build
> >
> >     Moreover, you don't need to update Nixpkgs. The new build function is
> >     actually part of npm2nix.
> >
> >     I've used the reengineered npm2nix on a private project for the
> company I
> >     work for, as well as some other utilities and it seems to work fine
> for me.
> >
> >     Hopefully, you can also try npm2nix on your projects to find out
> whether
> >     there any additional issues. :)
> >
> >     Furthermore, I'm still working a blog post to rationalize all the
> stuff,
> >     which should give you better insights in the problem, details and the
> >     choices I made.
> >
> >     Let me know what you think!
> >
> >     Regards,
> >
> >     Sander
> >
> >
> >
> >
> >
> >
>
> hi Sander,
>
> sorry for slow response, i was too busy last weeks.
>
> i've tried your fork of npm2nix on current project. my package.json looks
> like:
>
> {
>   "name": "XXX",
>   "version": "0.0.1",
>   "dependencies": {
>     "css-loader": "~0.9.0",
>     "domready": "~1.0.7",
>     "file-loader": "~0.7.2",
>     "jsx-loader": "~0.11.2",
>     "less-loader": "~0.7.7",
>     "lodash": "~2.4.1",
>     "react": "~0.11.2",
>     "react-bootstrap": "~0.12.0",
>     "react-router": "~0.9.2",
>     "script-loader": "~0.6.0",
>     "socket.io-client": "~0.9.17",
>     "style-loader": "~0.8.1",
>     "url-loader": "~0.5.5",
>     "webpack": "~1.4.4"
>   }
> }
>
> generating default.nix, node-env.nix and registry.nix works but when
> trying to
> build it using `nix-build -A build` i get the following error:
> https://gist.github.com/garbas/1ae5f5cf688561d03be5
>
> i remember in the past we had problems with fsevents but i can remember
> how we
> fixed them.
>
>
> --
> Rok Garbas - http://www.garbas.si
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.science.uu.nl/pipermail/nix-dev/attachments/20141012/abb3915d/attachment-0001.html