[Nix-dev] Auto-generated expressions for applications

[Benno Fünfstück]

I think that as a first step, we have to separate two use cases for nixpkgs:

* creating development environments to use with nix-shell etc
* installing applications

Package sets are key here. An application is built against a specific
package set, and a development environment is based on a package set. A
package set
is a consistent set of packages of a given language.

For nixpkgs itself, we should focus on providing access to *applications*.
It is not necessary to provide the latest and greatest of every library on
the planet, as long as we have no application that requires it.

Developers can use the tools we use for nixpkgs to generate package sets to
get access to the libraries they need. We may also have some "presets",
like for example "all of stackage" for Haskell.

The abstraction of a "package set" should be unified for all languages that
require consistent package sets (= no more than one version of the same
package in the dep chain. This includes Python, Haskell, .. but not for
example NodeJS, as npm supports "private" dependencies). There should be
*no* differences in how the functions are called, as this creates
confusion. They should use the same technique.

IMO, it is fine if each application uses is build against a package set
that is generated just for this application, and includes just the
dependencies for this app. However,
if we find that there is much duplication here, we could also adopt the
haskell approach which is:

* have one "superset" that includes all the needed package expressions.
this is not a package set by itself, as it may include multiple versions of
a dependencies
* then, for each application, take a consistent subset of packages out of
this set.
* we may also have a default package set if many packages share the same
versions of some dependencies. each application would then only need to
override
  a small number of dependencies in this default set.

Note that overrides on package sets must be deep overrides, so that if I
replace `foo-x` with `foo-y`, then all libraries previously depending on
`foo-x` should now also depend on `foo-y`. This ensures consistency.

If we had proper support for import-from-derivation, we could even support
the following syntax (as used in the haskell set):

packageSet.override { overrides = self: super: {
    some-package = self.callHackage "some-package" "1.2.43"
}};

so the needed non-default packages can be generated on the fly.

My opinion is that for the future, we should strive to auto-generate as
much as possible in nixpkgs. I view `nix` more as a assembly language, a
common format that auto-generators can use. Wouldn't it be nice if we
could, for each package, just `anything2nix package-version` and it would
spit out a derivation for that package? Then we can focus more on the
composition of those packages. In a way, I believe that security could even
be better with such an approach as we have more time to focus on package
versions and don't need to do as much "grunt" work like figuring out easy
dependencies of packages etc


I hope this wasn't too long of an email :),

Benno

Profpatsch <mail at profpatsch.de> schrieb am Mi., 31. Mai 2017 um 06:03 Uhr:

> On 17-05-30 08:02am, Wout Mertens wrote:
> > This actually ties into my question about nodePackages. It seems to me
> that
> > for these large packaging systems, we should have separate repos that
> > update from their source, and you can then include them into your nixpkgs
> > configuration.
>
> nodePackages is a bad example, because “node dependencies”
> make no sense on their own. Each application brings with
> it its own set of dependencies.
>
> >
> > Only packages that are useful by themselves should get a derivation in
> > nixpkgs, the rest would be in these add-on repos. For nodejs, that would
> be
> > e.g. yarn.
> > No idea how to accomplish this though.
>
> Exactly.
> As long as we keep generated formats in nice, diffable formats
> they shouldn’t impose too much noise to the system.
> Unfortunalely, the current output of node2nix is quite verbose.
>
> As far as CVEs go, one would have to integrate sites like
> https://snyk.io/vuln somehow.
>
>
> --
> Proudly written in Mutt with Vim on NixOS.
> Q: Why is this email five sentences or less?
> A: http://five.sentenc.es
> May take up to five days to read your message. If it’s urgent, call me.
> _______________________________________________
> nix-dev mailing list
> nix-dev at lists.science.uu.nl
> https://mailman.science.uu.nl/mailman/listinfo/nix-dev
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://mailman.science.uu.nl/pipermail/nix-dev/attachments/20170531/24ab0389/attachment.html>