[Nix-dev] How to bootstrap a server using Nixops?

[Jeffrey David Johnson]

On Fri, 1 May 2015 08:46:14 +0000
Rob Vermaas <rob.vermaas at gmail.com> wrote:

> Hi Jeff,
> 
> 
> On Thu, Apr 30, 2015 at 11:17 PM, Jeffrey David Johnson <jefdaj at gmail.com>
> wrote:
> 
> > It's possible I'm getting the wrong idea about what nixops is for, but I
> > used
> > it to set up an Amazon EC2 instance with the idea of logging into the
> > server
> > and administering it using local commands afterward.
> 
> 
> That's an interesting and unexpected usecase of nixops. I wouldn't use
> nixops at all if you plan on doing this. Why not just start instance in the
> Amazon AWS Console or using the command line tools of AWS? (for ami's see
> https://nixos.org/wiki/NixOS_on_Amazon_EC2)

You're right that looks more like what I wanted! Sorry just didn't see
it before. I'll post my scripts below for clarification but don't worry
about troubleshooting them. I wonder if the NixOps manual should
include a link to that page? Something like "If you just want to
install NixOS on the server, see here instead". It's the kind of
distinction, like installing the OS itself to a USB drive vs installing
the LiveCD installer, that seems obvious in hindsight but new
people might miss.

> > The initial install goes
> > fine. Afterward I clone my nixcfg repo (including nixpkgs submodule) and
> > try to
> > set it up the way I would a physical machine, by copying everything from
> > /etx/nixos/* into nixcfg/configs/cyno.nix and modifying it to
> > point inside the repo:
> >
> > { config, pkgs, ... }:
> >
> > {
> >   imports = [
> >     ../nixpkgs/nixos/modules/virtualization/amazon-config.nix
> >   ];
> >   ec2.hvm = true;
> >   services.journald.rateLimitBurst = 0;
> > }
> >
> >
> > When I do a `nixos-rebuild` (part of install.sh) though, it gives this
> > error:
> >
> > [root at cyno:~/nixcfg]# ./install.sh test
> > building Nix...
> > error: getting status of
> > ‘/root/nixcfg/nixpkgs/nixos/modules/virtualization/amazon-config.nix’: No
> > such file or directory
> > (use ‘--show-trace’ to show detailed location information)
> > error: getting status of
> > ‘/root/nixcfg/nixpkgs/nixos/modules/virtualization/amazon-config.nix’: No
> > such file or directory
> > (use ‘--show-trace’ to show detailed location information)
> > building the system configuration...
> > error: getting status of
> > ‘/root/nixcfg/nixpkgs/nixos/modules/virtualization/amazon-config.nix’: No
> > such file or directory
> > (use ‘--show-trace’ to show detailed location information)
> >
> > Now I notice something strange:
> > /root/nixcfg/nixpkgs/nixos/modules/virtualization/amazon-config.nix
> > exists, but
> > `ls` throws an error when listing it:
> >
> > ls /root/nixcfg/nixpkgs/nixos/modules/virtualization/amazon-config.nix
> > ls: cannot access
> > /root/nixcfg/nixpkgs/nixos/modules/virtualization/amazon-config.nix: No
> > such file or directory
> >
> > If I approach it "slowly" by listing /root/nixcfg, then
> > /root/nixcfg/nixpkgs,
> > all the way to the full path it *does* work. I googled around a little and
> > according to this:
> >
> > http://superuser.com/questions/446280/no-such-file-or-directory
> >
> > The error can also mean a missing dynamically linked library. So I tried
> > ldd as
> > directed (though I don't understand the output), and got this:
> >
> > ldd $(which ls)
> >         linux-vdso.so.1 (0x00007fff01dec000)
> >         libacl.so.1 =>
> > /nix/store/6mz0jhl389h7panazs5sayrydajia1s2-acl-2.2.52/lib/libacl.so.1
> > (0x00007f55d0f54000)
> >         libc.so.6 =>
> > /nix/store/93zfs0zzndi7pkjkjxawlafdj8m90kg5-glibc-2.20/lib/libc.so.6
> > (0x00007f55d0bb7000)
> >         libattr.so.1 =>
> > /nix/store/ndszpck395bi55jnz50ny99y4pvj6dqn-attr-2.4.47/lib/libattr.so.1
> > (0x00007f55d09b3000)
> >
> > /nix/store/93zfs0zzndi7pkjkjxawlafdj8m90kg5-glibc-2.20/lib/ld-linux-x86-64.so.2
> > (0x00007f55d115c000)
> >
> > Anyway I think there's something wrong with how the system is set up, but
> > not
> > sure what. Is there a correct way to bootstrap it to the point of using
> > the same config as my other machines?
> >
> 
> Can you perhaps show us the contents of install.sh? Can you give us the AMI
> that was used (see AWS Console or commandline tools) ? Also, how did you
> copy the contents of /etc/nixos/* to /root/nixcfg ? I have never seen this
> behavior before, and tested something similar yesterday successfully.

install.sh is just a wrapper to point nixos-rebuild to my repo:

#!/usr/bin/env bash
# Automates rebuilding nixos from my repos.
# Takes the same args as nixos-rebuild.
# TODO also put NIXPKGS_CONFIG in here instead of home dir

NIXCFG="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
[[ -z "$@" ]] && NIXARGS="test --fast" || NIXARGS="$@"

nixos-rebuild \
  -I mypkgs="${NIXCFG}/mypkgs" \
  -I nixpkgs="${NIXCFG}/nixpkgs" \
  -I nixos-config="${NIXCFG}/configs/$(hostname).nix" \
  ${NIXARGS}

Sorry I should have been clearer about "copied", I just meant I wrote
nixcfg/configs/cyno.nix to match what I thought the contents
of /etx/nixos did: import amazon-config.nix and set a couple other
settings. The other files seemed redundant with ones in nixpkgs,
which was the next thing I planned to figure out. I tried importing the
file with this notation too:
<nixpkgs/nixos/modules/virtualization/amazon-config.nix> Then changed
it because that wasn't found. But the same ls error could have caused
that too.

The EC2 instance itself was created with these logical and physical
specifications:

# cyno.nix
let
  cynoTest = { ... }: {
    environment.systemPackages = with import <mypkgs>; [
      nix
      nix-repl
    ];
  };

in {
  cyno = cynoTest;
  network.description = "Cyno test";
}

# cyno-ec2.nix
let
  testKey = {
    region = "us-west-1";
    accessKeyId = "personal";
  };

  ec2Test = { resources, ... }: {
    deployment = {
      targetEnv = "ec2";
      ec2 = {
        inherit (testKey) region accessKeyId;
        ami = "ami-ca534a8f"; # TODO remove once newer nixops comes out
        keyPair = resources.ec2KeyPairs.test;
        instanceType = "t2.micro";
      };
    };
  };

in {
  cyno = ec2Test;
  resources.ec2KeyPairs = { test = testKey; };
}

> Cheers,
> Rob
> 
> -- 
> Rob Vermaas
> 
> [email] rob.vermaas at gmail.com