[Nix-dev] On npm2nix and the NPM package set in Nixpkgs

Sander van der Burg svanderburg at gmail.com
Tue Jul 5 14:12:38 CEST 2016 

Thank you for the responses so far....

To remind you about the set of packages I intend to include: I only want
end-user software (such as command-line utilities) and packages that are
dependencies of non-NPM projects to appear in Nixpkgs. All the other
packages will be removed from node-packages.json.


On Tue, Jul 5, 2016 at 9:56 AM, Michael Fellinger <m.fellinger at gmail.com>
wrote:

> For what it’s worth, I’m using the re-engineeering2 branch standalone for
> projects with hundreds of npm dependencies. The way I use it right now is
> like this:
>
> https://gist.github.com/manveru/20d22586d9dceae90930be528cbc49ce
>
> Having it as a part of nixpkgs would be nice, but won’t really change how
> I build and use it, and having npm dependencies listed in nixpkgs isn’t
> very productive, the ecosystem for JS libraries changes so fast, that we’d
> have to automatically update the index every day and hope nothing breaks,
> people depending on js libraries listed in nixpkgs will always be
> frustrated.
>
> I prefer having the checksums generated by npm2nix, and giving each app
> that is packaged in nixpkgs their own list of dependencies generated by
> npm2nix.
>
> So for now, I think adding it under a different name to nixpkgs and
> gradually changing things to use the new approach might be the best
> solution.
>
> There are two things that I’m still hoping could be done better: one is
> that the location to the package.json should be more flexible, while you
> can specify where it is located when running npm2nix, it won’t be found
> later when it’s in a directory above the one you put the npm2nix-generated
> files in. I opted for putting npm-generated files into their own
> subdirectory, and then run
>
> `ln -s "$(nix-build ./nix/npm.nix -A
> package)/lib/node_modules/myapp/node_modules" node_modules`
>
> Maybe there’s a better way, but that’s what I figured out on my own.
>
> On 5 July 2016 at 11:17:05, Tomasz Czyż (tomasz.czyz at gmail.com) wrote:
>
> Rok,
>
> what about people who are already using previous solution? Why break their
> workflows?
>
> 2016-07-05 7:36 GMT+01:00 Rok Garbas <rok at garbas.si>:
>
>> +1 for just keeping the name npm2nix and bumping up the version.
>>
>> i'm not using it on any active project, but i'm going to in the near
>> future.
>>
>>
>>
>> On Mon, Jul 4, 2016 at 10:11 PM, Tobias Pflug <tobias.pflug at gmx.net>
>> wrote:
>> > Hi Sander,
>> >
>> > sorry for my very late response. I'll make this one brief as I am sadly
>> on
>> > my phone.
>> >
>> > I  belong to one of those who tried your new npm2nix and in fact am
>> already
>> > using it regularly. I am very much in favor of having your
>> re-engineeering2
>> > branch replacing npm2nix as the de-facto node integration tool.
>> >
>> > I also definitely want to see the current set of auto-generated node
>> > packages removed from nix. They are almost exclusively *totally*
>> outdated.
>> >
>> > Thank you a lot for your continued efforts on this. Working with
>> npm/node is
>> > annoying but we are better off with your contributions.
>> >
>> > cheers,
>> > Tobi
>> >
>> > On 22 Jun 2016, at 20:24, Sander van der Burg <svanderburg at gmail.com>
>> wrote:
>> >
>> > Hello Nix and Node.js users,
>> >
>> > I have been absent for a while in this discussion, but as far as I know
>> the
>> > state of the NPM packages in Nixpkgs is still quite bad and despite some
>> > discussions on the mailing list we have not really come to any consensus
>> > yet.
>> >
>> > As some of you may know, I have my own re-engineered version of npm2nix
>> that
>> > lives in a specific branch in my own personal fork
>> > (https://github.com/svanderburg/npm2nix/tree/reengineering2). A few
>> months
>> > ago, I did some major efforts in getting npm 3.x's behaviour supported,
>> > which I have documented in this blog post:
>> >
>> http://sandervanderburg.blogspot.com/2016/02/managing-npm-flat-module-installations.html
>> >
>> > I have been using this reengineering2 branch for all my public and some
>> of
>> > my private projects since the beginning of this year, and for me it
>> seems to
>> > work quite well, despite the fact that some of npm 3.x's flat module
>> > installation oddities are still not accurately supported yet.
>> >
>> > I also received a couple of reports from other people claiming that
>> their
>> > projects work and even encountered some people saying that it should
>> replace
>> > the current npm2nix. :)
>> >
>> > Obviously, I do not want to claim that my implementation is the perfect
>> > solution as it (for example) is much slower than the vanilla npm2nix,
>> and it
>> > composes the entire set of dependencies in one derivation as opposed to
>> > generating a Nix store path per NPM dependency. (I do this for a very
>> good
>> > reason. For more details, please read my blog post).
>> >
>> > Furthermore, I have also spoken to people that suggested completely
>> > different kinds of approaches in getting NPM supported in a Nix
>> environment.
>> >
>> > Something that I have not done yet is investigating whether this
>> > reengineered solution could be a potential replacement for the NPM
>> packages
>> > set in Nixpkgs.
>> >
>> > Today, I have been working on an integration pattern, and the good news
>> is:
>> > it seems that I was able to generate Nix expressions for almost all
>> packages
>> > that are in pkgs/top-level/node-packages.json. The only exceptions were
>> the
>> > node-xmpp-* and bip-* packages, but some of them seem to have broken
>> > dependencies, which is not npm2nix's fault.
>> >
>> > If we would proceed integrating, we have a number of practical
>> implications:
>> >
>> > - I believe it is desired to have both Node.js 4.x and Node.js 5.x, 6.x
>> > supported (I actually need all of them). To support all of these, we
>> need
>> > two different sets of generated Nix expressions. The former uses npm 2.x
>> > with the classic dependency addressing approach and the latter uses npm
>> 3.x
>> > with flat module installations.
>> > - I think most library packages should be removed from
>> node-packages.json:
>> > as explained in my blog post: how a package gets composed and to which
>> > version a range resolve depends on the state of the includer. When
>> somebody
>> > wants their own NPM project to be deployed, he should use npm2nix
>> directly
>> > on package.json, and not refer to any NPM libraries in Nixpkgs.
>> > - Some NPM packages must be overridden to provide native dependencies.
>> The
>> > mechanisms that the reengineering2 branch use are different. It would
>> > probably take a bit of effort to get these migrated.
>> >
>> > For example, this is how I override the webdrvr package to provide
>> phantomjs
>> > and the Selenium webdriver:
>> >
>> > {pkgs, system}:
>> >
>> > let
>> >   nodePackages = import ./composition-v4.nix {
>> >     inherit pkgs system;
>> >   };
>> > in
>> > nodePackages // {
>> >   webdrvr = nodePackages.webdrvr.override (oldAttrs: {
>> >     buildInputs = oldAttrs.buildInputs ++ [ pkgs.phantomjs ];
>> >
>> >     preRebuild = ''
>> >       mkdir $TMPDIR/webdrvr
>> >
>> >       ln -s ${pkgs.fetchurl {
>> >         url =
>> > "
>> https://selenium-release.storage.googleapis.com/2.43/selenium-server-standalone-2.43.1.jar
>> ";
>> >         sha1 = "ef1b5f8ae9c99332f99ba8794988a1d5b974d27b";
>> >       }} $TMPDIR/webdrvr/selenium-server-standalone-2.43.1.jar
>> >       ln -s ${pkgs.fetchurl {
>> >         url =
>> > "
>> http://chromedriver.storage.googleapis.com/2.10/chromedriver_linux64.zip
>> ";
>> >         sha1 = "26220f7e43ee3c0d714860db61c4d0ecc9bb3d89";
>> >       }} $TMPDIR/webdrvr/chromedriver_linux64.zip
>> >
>> >     '';
>> >   });
>> > }
>> >
>> >
>> > Although we have some practical issues, I think none of them would
>> impose a
>> > serious problem.
>> >
>> > Then about npm2nix itself: Obviously, we could say that my version
>> replaces
>> > the upstream npm2nix and gets "blessed" into the new "official"
>> version, but
>> > I don't know whether everybody likes it.
>> >
>> > Alternatively, we could be a bit more pragmatic: I stop calling my
>> > reengineering2 version npm2nix, I give it a different name and I
>> release it
>> > as a different package. This makes it possible for those who want it, to
>> > still use the 'vanilla' npm2nix alongside my version.
>> >
>> > Then in Nixpkgs we can decide to:
>> >
>> > - to keep npm2nix the default and provide my tool as a package
>> > - or to make the reengineering2 version the default, and provide
>> npm2nix as
>> > a package
>> > - in theory: support both package sets, but this might be a bit
>> overkill :)
>> >
>> > For those who don't know: although my repository is a fork of npm2nix,
>> the
>> > reengineering2 version is basically a rewrite of npm2nix and quite
>> different
>> > than the upstream version. It is written in JavaScript (as opposed to
>> > CoffeeScript), has a different modular structure and different
>> command-line
>> > interface, so that's why I'm very careful in proposing to replace the
>> > upstream npm2nix.
>> >
>> > Moreover, it also does not share any git revision history with the
>> upstream
>> > npm2nix. :)
>> >
>> > As a final note: for those who do not know about this: the
>> reengineering2
>> > tool can already be used outside Nixpkgs and this is what I have been
>> doing
>> > for all my projects. The expressions that it generates are based on the
>> > principles I have described in this blog post:
>> >
>> http://sandervanderburg.blogspot.com/2014/07/managing-private-nix-packages-outside.html
>> >
>> > My apologies for this very long email, but I'd like to have your
>> feedback
>> > and I don't want my preferences to disrupt other people's workflows.
>> >
>> > What do you think?
>> >
>> > Best,
>> >
>> > Sander
>> >
>> > _______________________________________________
>> > nix-dev mailing list
>> > nix-dev at lists.science.uu.nl
>> > http://lists.science.uu.nl/mailman/listinfo/nix-dev
>> >
>> >
>> > _______________________________________________
>> > nix-dev mailing list
>> > nix-dev at lists.science.uu.nl
>> > http://lists.science.uu.nl/mailman/listinfo/nix-dev
>> >
>>
>>
>>
>> --
>> Rok Garbas
>> http://www.garbas.si
>> rok at garbas.si
>> _______________________________________________
>> nix-dev mailing list
>> nix-dev at lists.science.uu.nl
>> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>>
>
>
>
> --
> Tomasz Czyż
> _______________________________________________
> nix-dev mailing list
> nix-dev at lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>
> _______________________________________________
> nix-dev mailing list
> nix-dev at lists.science.uu.nl
> http://lists.science.uu.nl/mailman/listinfo/nix-dev
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.science.uu.nl/pipermail/nix-dev/attachments/20160705/0a1e3bc4/attachment-0001.html>

More information about the nix-dev mailing list