[Nix-dev] Reengineered npm2nix: call for feedback

[Sander van der Burg]

Just tried building fsevents on my office mac machine both with the old and
new npm2nix and both seem to result in the same error. I don't think the
problem is caused by the generation, but by something else. Still have to
figure out why the header file can't be found.

-- Sander

On Sun, Oct 12, 2014 at 9:46 PM, Sander van der Burg <svanderburg at gmail.com>
wrote:

> Ah I've seen somebody else reporting this issue as well.
>
> So does the fsevents package compile correctly with the old npm2nix? If I
> see the output, apparently some native library dependency is missing that
> we somehow have to provide by other means.
>
> Tomorrow, when I'm in the office (where I have access to a Mac machine) I
> can check this.
>
> Regards,
>
> Sander
>
>
> On Sun, Oct 12, 2014 at 8:04 PM, Rok Garbas <rok at garbas.si> wrote:
>
>> Quoting Sander van der Burg (2014-10-12 14:21:17)
>> > I think most of you probably know it already, but I've written a blog
>> post
>> > about the reengineered implementation, that can be found here: http://
>> > sandervanderburg.blogspot.com/2014/10/
>> > deploying-npm-packages-with-nix-package.html
>> >
>> > Second, I have some good and some bad news.
>> >
>> > Good news is that there were a few people lately, having problems with
>> > deploying certain packages with the old npm2nix (one time there was
>> another
>> > cyclic dependency). I tested the same packages with the new npm2nix and
>> they
>> > seem to deploy just fine without problems. So I think it that the new
>> approach
>> > really solves some very nasty problems and accurately simulates most of
>> NPM's
>> > way of managing dependencies.
>> >
>> > The only bad thing so far is that the new approach is a lot slower.  I'm
>> > specifically talking about the time it takes to deploy something with
>> nix-build
>> > or nix-env.
>> >
>> > The slowness is basically caused by the fact that for each dependency I
>> want to
>> > include, I have to tell which shared dependencies have been provided
>> already to
>> > correctly simulate the way version ranges are resolved. I need to run
>> semver
>> > for this. Moreover, to also prevent infinite recursion because of cyclic
>> > dependencies, I use a expression generation trick.
>> >
>> > I think I already have a few optimizations in mind, but they are
>> difficult to
>> > implement and I might break stuff again. For example, if I'm absolutely
>> sure
>> > there are no cycles in a deployment, I can skip the dependency
>> generation
>> > trick, for example. Question is: how to detect this?
>> >
>> > I don't think the slowness is problematic for private projects, but it
>> might be
>> > too expensive to use that for the set of NPM packages part of Nixpkgs.
>> >
>> > On the other hand, I also think that in the future, most packages
>> should be
>> > deployed privately in NPM projects. The main reason why I recommend
>> this is
>> > that version range specifiers do not always resolve to their latest
>> compatible
>> > versions. The actual version used is context specific -- if some project
>> > requires async 0.2.x, the latest conforming version is 0.2.9, but a
>> shared
>> > dependency is 0.2.5, then 0.2.5 is used. That's why you have to
>> reconsider the
>> > complete tree of dependencies for every project.
>> >
>> > The only packages that still need to be deployed from Nixpkgs IMHO are
>> end-user
>> > utilities (command-line utilities) and libraries used by non-NPM
>> projects.
>> >
>> > So I'm still thinking a bit about it and what I can do to make the
>> performance
>> > better. In the worst case, I have to throw away the entire approach and
>> invent
>> > something new, if it turns out that the performance is too problematic.
>> >
>> > What do you think?
>> >
>> > -- Sander
>> >
>> >
>> > On Tue, Sep 30, 2014 at 12:13 PM, Sander van der Burg <
>> svanderburg at gmail.com>
>> > wrote:
>> >
>> >     Hi everybody,
>> >
>> >     Since the sprint in Ljubljana until now, I've been working on
>> npm2nix's
>> >     issues. I basically reengineered most of it (e.g. rewritten it to
>> >     JavaScript and modularized it a bit further) and I think the
>> implementation
>> >     is fairly complete now. I've incorporated as much of Shea's
>> functionaility
>> >     into the reengineered version as I could.
>> >
>> >     The new implementation handles dependencies in such a way that its
>> behavior
>> >     is closer to what NPM does. Moreover, it should also properly cope
>> with
>> >     cyclic dependencies. The way dependencies are resolved is actually
>> much
>> >     more advanced than the original npm2nix implementation. For
>> example, it
>> >     uses semver to make matches between version ranges.
>> >
>> >     In theory, also cyclic dependencies that match on version ranges
>> should be
>> >     handled properly, although I haven't encountered them yet.
>> >
>> >     Using the reengineered version is straight forward. You can obtain
>> it from
>> >     my private Git fork:
>> >
>> >     https://github.com/svanderburg/npm2nix/tree/reengineering
>> >
>> >     I have updated the README.md file of npm2nix to elaborate a bit
>> more about
>> >     the possible use cases of npm2nix. So I think it wouldn't be a bad
>> idea to
>> >     read it first.
>> >
>> >     Installation can be done by opening the git checkout, and running:
>> >
>> >     $ nix-env -f default.nix -iA build
>> >
>> >     Moreover, you don't need to update Nixpkgs. The new build function
>> is
>> >     actually part of npm2nix.
>> >
>> >     I've used the reengineered npm2nix on a private project for the
>> company I
>> >     work for, as well as some other utilities and it seems to work fine
>> for me.
>> >
>> >     Hopefully, you can also try npm2nix on your projects to find out
>> whether
>> >     there any additional issues. :)
>> >
>> >     Furthermore, I'm still working a blog post to rationalize all the
>> stuff,
>> >     which should give you better insights in the problem, details and
>> the
>> >     choices I made.
>> >
>> >     Let me know what you think!
>> >
>> >     Regards,
>> >
>> >     Sander
>> >
>> >
>> >
>> >
>> >
>> >
>>
>> hi Sander,
>>
>> sorry for slow response, i was too busy last weeks.
>>
>> i've tried your fork of npm2nix on current project. my package.json looks
>> like:
>>
>> {
>>   "name": "XXX",
>>   "version": "0.0.1",
>>   "dependencies": {
>>     "css-loader": "~0.9.0",
>>     "domready": "~1.0.7",
>>     "file-loader": "~0.7.2",
>>     "jsx-loader": "~0.11.2",
>>     "less-loader": "~0.7.7",
>>     "lodash": "~2.4.1",
>>     "react": "~0.11.2",
>>     "react-bootstrap": "~0.12.0",
>>     "react-router": "~0.9.2",
>>     "script-loader": "~0.6.0",
>>     "socket.io-client": "~0.9.17",
>>     "style-loader": "~0.8.1",
>>     "url-loader": "~0.5.5",
>>     "webpack": "~1.4.4"
>>   }
>> }
>>
>> generating default.nix, node-env.nix and registry.nix works but when
>> trying to
>> build it using `nix-build -A build` i get the following error:
>> https://gist.github.com/garbas/1ae5f5cf688561d03be5
>>
>> i remember in the past we had problems with fsevents but i can remember
>> how we
>> fixed them.
>>
>>
>> --
>> Rok Garbas - http://www.garbas.si
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://lists.science.uu.nl/pipermail/nix-dev/attachments/20141013/6fc65d23/attachment-0001.html