[Nix-dev] Call For Maintainers - Fractalide BETA release

[stewart mackenzie]

Greetings all,

So after some 946 commits a number of rewrites starting in 2014 when
Fractalide was implemented in a neat programming language called
Mozart Oz, we've gone BETA. At that stage Fractalide was the subject
of a Master thesis for Denis Michiels under Professor Peter Van Roy of
UCL Belgium (Peter is a co-author of the book Concepts Techniques and
Models of Computer Programming). Both Peter and Denis are fantastic
fellows. Since then Fractalide has moved out of academia and into
industry reincarnated as Rust and Nix. I had the good fortune of
meeting Denis face to face in Belgium last year when we had our first
"fraconf" of only two participants! Hopefully this conference can grow
in numbers :-)

It would seem that Fractalide does in fact occupy a very important
part of the industry. Dataflow languages in the form of Ab Initio [0]
serve enterprises and charge a very heavy price tag. A couple of years
ago Google implemented this dataflow programming language called
TensorFlow, it's aimed at Machine Learning but it has other uses. The
NSA also open sourced their dataflow programming language called
Niagrafiles, now renamed to Apache-NiFi.

As each node in Fractalide is essentially a reusable function stuck in
a nix derivation one might say that Fractalide adds "Nixfuncs" to
"Nixpkgs", where nixpkgs are monolith applications that don't
(rarely?) compose and "nixfuncs" are reusable functions that can
compose.

It makes sense this general purpose dataflow language needs to
specialize in creating reusable, reproducible services, the power of
nix makes this quite possible to achieve. It's also a very trendy
area, which can lead into an even trendier area that of the Internet
of S... Things!

There has been a few guiding principles along the path.

1) ensure we have a language that cooperates with a congruent
configuration system.
2) safety both within and outside the application boundary.
3) primitives that make executing in a distributed environment easier
to reason about.
4) fast, fast as greased lightning.

Going BETA means it's a signal to the public that underlying nodes are
headed for stable and their public APIs won't change with a high
probability. After stabilization they won't ever be deleted or
changed. In other words Fractalide is going to be a rock solid
platform to build things on, for decades to come.

We're not yet ready to go STABLE because there are a few things in Nix
that need to happen such as the landing of this [1], which has this:
[2] and nixcrates needs a final round of work elbow work.

We're adopting the Pieter Hintjens style of community, whereby, should
jobs come in they get farmed out to trusted contributing members of
the community. This approach is more like forming an IETF group who
ralley around problems and solve them quickly, efficiently then share
the spoils, going back to their daily lives. These brilliant types of
people rarely tolerate enterprise environments and prefer working
independently or in small groups. Basically our company Noware Ltd. is
giving the Fractalide codebase away to the community, copyright is
spread far and wide to prevent corporate takeover and lockins, so as a
whole the community grows it into highly evolved software. As
Fractalide is so heavily dependent on Nix, it's fair to hand over a
percentage of the earnings to the Nix Foundation to help keep the
lights on.

On that note, if you're in business and you have expensive problems
you want to solve that fit Fractalide's domain please lets get in
touch. I'd like to start repaying favours the nix community has
offered me.

Onto some code.

So this is a contrived example of an NAND `agent`, (it's not a real world node):

#[macro_use]
extern crate rustfbp;
extern crate capnp;

agent! {
  input(a: prim_bool, b: prim_bool),
  output(output: prim_bool),
  fn run(&mut self) -> Result<Signal> {
    let a = {
        let mut msg_a = try!(self.input.a.recv());
        let boolean: prim_bool::Reader = msg_a.read_schema()?;
        boolean.get_bool()
    };
    let b = {
        let mut msg_b = try!(self.input.b.recv());
        let boolean: prim_bool::Reader = msg_b.read_schema()?;
        boolean.get_bool()
    };

    let mut out_msg = Msg::new();
    {
      let mut boolean = out_msg.build_schema::<prim_bool::Builder>();
      boolean.set_bool(if a == true && b == true {false} else {true});
    }
    try!(self.output.output.send(out_msg));
    Ok(End)
  }
}

Here is the associated nix code that sets up the dependencies needed
to build the above NAND `agent`

{ agent, edges, crates, pkgs }:

agent {
  src = ./.;
  edges = with edges; [ prim_bool ];
  crates = with crates; [ rustfbp capnp ];
  osdeps = with pkgs; [];
}

This code can be executed by running:

$ git clone https://github.com/fractalide/fractalide.git
$ cd fractalide
$ nix-build --argstr node test_nand
$ ./result
boolean : false

The arguments `--argstr node test_nand` build a file that looks like this:

$ cat /nix/store/kzw9p6a1snkhswwlnzb0i2a5rzg9vymd-test_nand
/nix/store/2wh7azff76a3g3gdcxdp3b713p3a7ij2-fvm/bin/fvm
/nix/store/x8q03byc501n7ygm46cwdda2bxbyjbkk-test_nand

Notice /nix/store/...-fvm is the Fractalide Virtual Machine (FVM is
super lightweight), it too is built up of dataflow agents.

The FVM executes the /nix/store/...-test_nand subgraph, which looks like this:

{ subgraph, nodes, edges }:

subgraph {
 src = ./.;
 flowscript = with nodes; with edges; ''
  '${prim_bool}:(bool=true)' -> a nand(${maths_boolean_nand}) output
-> input io_print(${maths_boolean_print})
  '${prim_bool}:(bool=true)' -> b nand()
 '';
}

The subgraph function writes a textfile based on the contents of the
flowscript attribute, where the ${} contents are resolved to capnproto
schemas, agents and other subgraphs in the /nix/store. The FVM
understands where to find the relevant files in the /nix/store and
recursively loads the entire graphs of agents into memory.

In this example capnproto schema with the value 'false' are passed
into the input ports a and b of the above NAND agent code, notice
their schema are type prim_bool, which corresponds with the types in
the rust code. The data flowing in gets transformed by the run
function and is sent outwards on the output port, which is then sent
to the ${maths_boolean_print} `agent` where it's printed to the
screen.

Notice there is no trace of a Cargo.toml in any agents folder, just a
tidy lib.rs and default.rs, nixcrates/nix completely handles all the
crates.io dependencies for us without calling cargo. Also notice the
nodes ${maths_boolean_nand} and ${maths_boolean_print} are truly
reproducible thanks to nix. They can be called anywhere from any
subgraph. This feature is what sets Fractalide apart from neigh on any
other dataflow solution out there. It's also what makes for a vibrant
FOSS community, as the mix-and-matchability factor is extremely high,
because these dataflow functions are what? ... reusable!

So it's time to put out a Call for Maintainers, those interested
please have a read of this: [3] if it gels with you, please ping me
and you'll be given Maintainer access to the repository. Noware's Ltd
responsibility is setting and enforcing the code collaboration
contract, we'll also boot psychopaths (with evidence) from the
community. The maintainers responsibility is quite different from your
average software project, including nixpkgs, so a mental shift is
needed. The onus of keeping the system up and running comes down to
the contributors not the maintainers. Patches are merged quickly even
if there are errors, or newbie mistakes, this is how we learn. A
blatant mistake will be reverted if the contributor doesn't respond to
queries. "Discussion" takes place in the form of patches in response
to mistakes in code not upfront consensus on PRs. The mentality behind
this is Amdahl's law, upfront consensus means locks take hold and and
the whole thing slows down - remove the locks!. From time to time
there will be stabilization releases, we won't use semantic versioning
which gives a license to break stable code and cause distrust in the
community (i.e. basically the equivalent of `bumping major is
forbidden`). The only thing Maintainers must do is ensure that patch
coding styles accord with convention (still to be decided) and ensure
stable public contracts are not broken, there are a few others, it's
all in CONTRIBUTING.md. This approach ensures maintainers do not burn
out, and the codebase can be treated as a kind of wiki, a concept
proven time and time again.

You can read the README and associated documentation; for more information [4].

Lastly a most gracious thank you goes out to Joachim and Paul of
Nixcloud [5] . They implemented [6] under time constraints. This
software is a drop in replacement for Cargo, Rust's language level
package manager. This ensures Fractalide has a hermetically sealed
build environment. Nixcrates may be used for other rust/nix projects!

This is the second time I've worked with them, and a third one coming
up, so if ever you need Nix heavy guns! I highly recommend them!

An equivalent sort of mail will be extended to the Rust and Flow-based
Programming community. So it's my hopes that the nix programmers can
get their head around this code and help out creating basic reusable
reproducible services! It's a harder sell for those guys and girls
cause many haven't come to appreciate nix. So the invitation process
will be staggered.

[0] - http://abinitio.com/
[1] - https://github.com/NixOS/nixpkgs/issues/20156
[2] - https://github.com/systemd/systemd/blob/master/NEWS#L53
[3] - https://github.com/fractalide/fractalide/blob/master/CONTRIBUTING.md
[4] - https://github.com/fractalide/fractalide
[5] - https://nixcloud.io
[6] - https://github.com/fractalide/nixcrates