[Nix-dev] Using Nix as the preferred package manager for a new language

[stewart mackenzie]

Hi Thomas, (I'm unsure who you were addressing :-) but here's my response)
 I've been studying Peter and Gleb's work. This function is a thing of
beauty: [1]

On Wed, Feb 10, 2016 at 7:14 PM, Thomas Hunger <tehunger at gmail.com> wrote:
> I think the idea is exciting but difficult. Some notes / questions:
>
> 1/ Nix stores its data in /nix/store - is your package manager OK with root
> access? There are ways around that, e.g. using --with-store-dir but see [-1].

Well so far nix is our package manager.. it is our.. everything..
Where one considers a package as a shared library exposing a C ABI.
Nix literally glues everything together.
Nix builds each component and at the final step composes some six to
eight of those components forming the virtual machine, which can
execute fbp markup.
So we have really deep integration with nix. (It's really great when
nix catches circular dependencies...).

Now Rok's idea is to have a vm wrapper nix expression which
recursively parses the argument fbp file and builds a virtual machine
with exactly those deps. At this stage I really like this approach, it
keeps everything nix and riding on the goodness of the nix community
as well as contributing back, but of course it means fractalide has a
hard dependency on nix, which given the design goals of a system
coordinator and high level monitoring system. How else is one going to
have such deep control over a system?

> 2/ Do you want semver versioning or some form of always-follow-HEAD with
> stabilization branches like e.g. nixos or stackage LTS do?

I really don't want to follow the semver approach and instead will
annotate a component's inputs and outputs with a stability factor [2]
(still vapourware atm)
Whatever is on head should be (re-)usable. There should be no
stabilization branches, every single component should be completely
reusable right out the gates.
- component names are stable and do not change (unless every port is
experimental)
- port names remain stable and are never* renamed or deleted (* only
when the port in question isn't "experimental")
- capnproto contracts on each port must evolve (using capnproto
features) allowing for backward compatibility.
Eventually I'd really like to have a bit of code to map-reduce over
the port connection graph to determine what's being used to
automatically annotate ports... and most importantly who breaks the
contracts. My goal is rock solid software one could sign stability SLA
agreements on.

> 3/ Will you ever bind to c libraries like postgresql?

For trivial things like a component wrapping Iron (a rust webserver on
crates.io) which needs openssl to build, it's not a problem [3].
But when it comes to setting up more complex components such as
postgres which requires service activation I see a problem with this
approach.
It's at this point where I'd say going the guix approach might be best
but I'm a noob so I don't know. It's a design goal to enable complex
components which would setup services and configure a running
system... eventually I'd like it to setup across a network barrier. I
need to think about this more... it's still on the horizon, riding nix
as much as possible would be best.

The frontend (vapourware atm) will be a hypercard reimplementation,
input starts there (or cli) and flows through a system as complex as
fbp components on nixpkgs eventually back again - maybe it's useful.

> If you can fall back on binaries from your underlying system then maybe 1
> isn't a problem - at the cost of reproducibility.

Rust compile times are super slow to the point of annoying, so binary
distribution is a must, though the end goal is binary distribution
over a named data network, that's why _everything_ is build from named
components.

> Personally I think I'd follow Domen's advice of machine readable metadata
> and convert those into one big nix-expression on the fly before running e.g.
> nix-build.
> There are already several good examples for packaging *all* libraries for an
> ecosystem from metadata (Peter's excellent Haskell work, Gleb's hex
> packages, ...).

Actually I really like Domen's advice an a good specification of
packaging metadata with neat functions to convert to and from.

Though i think my mileage is pretty far given that 1) nix is going to
become more portable (via shealevy's awesome work) 2) nix has become a
super huge monster that lazily evaluates, it would be really sad to
depart from nix 3) a nix file that's as simple as this [3] isn't too
bad.

the `src` bit can be replaced with a fetchgit or something like that
so folks can develop elsewhere if needed.

> [-1]
> A while ago I spent some time trimming down nix to just stdenv.mkDerivation
> + lib + {git,svn,url} fetchers so I could use it as "import <nixpkgs-core>
> {}". Rebuilding that with a different --with-store-dir still takes a long
> time (using a path other than /nix/store disables binary substitution).

I really must look into this! Seems like awesome work!

[1] https://github.com/NixOS/nixpkgs/blob/69add60b5cee1b54379554b35406a1de64f9f681/pkgs/development/haskell-modules/default.nix#L79-L83
[2] "experimental", "stable", "legacy", "deprecated" - and have a set
of rules surrounding the evolution of these contracts.
[3] ```
{ stdenv, buildFractalideComponent, filterContracts, genName, openssl, ...}:

buildFractalideComponent rec {
  name = genName ./.;
  src = ./.;
  filteredContracts = filterContracts ["path" "domain_port"];
  buildInputs = [ openssl ];
  depsSha256 = "0n96kni0zlrm5jxwpc83w3b3jkj0zzr8h7mqg3qyzayky31s2g6i";

  meta = with stdenv.lib; {
    description = "Component: Iron webserver";
    homepage = https://github.com/fractalide/fractalide/tree/master/components/web/server;
    longDescription = "Component: Iron webserver
    example usage:
    'path:(path="/path/to/your/www/dir")' -> www_dir www(web_server)
    'domain_port:(domainPort="localhost:8080")' -> domain_port www()
    ";
    license = with licenses; [ mpl20 ];
    maintainers = with upkeepers; [ dmichiels sjmackenzie ];
  };
}
```
/sjm

> On 10 February 2016 at 05:48, stewart mackenzie <setori88 at gmail.com> wrote:
>>
>> Rok! You're tickling me pink! Cheers!