[Nix-dev] execute LXC guest as none root user

[Joachim Schiele]

hey,

we got lxc with NixOS as guest on NixOS as host working but we can only
execute it using the 'root' users.

however, for security reasons it seems to be a good thing to use the
kernel 'user namespaces' but i deliberately fail to get this running.

we are using nix-rehash from offlinehacker:
  https://github.com/nixcloud/nix-rehash


the latest issue when using:
  /etc/subuid
  /etc/subgid


lxc-start -d -n "CONTAINERNAME" \
  -f "container.conf" \
  -s lxc.rootfs=$CONTAINER_ROOT \
  -F
  "/nix/store/05sy0bz81426798qzrj66m64ncb3pymd-nixos-15.09.756.88765a/init"
Using /var/lib/containers/CONTAINERNAME as rootfs
lxc-start: cgmanager.c: lxc_cgmanager_enter: 698 call to
cgmanager_move_pid_sync failed: invalid request
lxc-start: start.c: __lxc_start: 1172 failed to spawn 'CONTAINERNAME'
lxc-start: lxc_start.c: main: 344 The container failed to start.
lxc-start: lxc_start.c: main: 348 Additional information can be obtained
by setting the --logfile and --logpriority options.


cgmanager --debug:
Connection from private client
ListControllers: Client fd is: 6 (pid=5956, uid=1000, gid=100)
Disconnected from private client
Connection from private client
Disconnected from private client
Connection from private client
Disconnected from private client
Connection from private client
ListControllers: Client fd is: 6 (pid=5958, uid=1000, gid=100)
Disconnected from private client
Connection from private client
Disconnected from private client
Connection from private client
Create: Client fd is: 6 (pid=5956, uid=1000, gid=100)
Created /run/cgmanager/fs/blkio/Me/lxc/CONTAINERNAME for 5956 (1000:100)
Created /run/cgmanager/fs/cpu/Me/lxc/CONTAINERNAME for 5956 (1000:100)
Created /run/cgmanager/fs/cpuset/Me/lxc/CONTAINERNAME for 5956 (1000:100)
Created /run/cgmanager/fs/devices/Me/lxc/CONTAINERNAME for 5956 (1000:100)
Created /run/cgmanager/fs/freezer/Me/lxc/CONTAINERNAME for 5956 (1000:100)
Created /run/cgmanager/fs/memory/Me/lxc/CONTAINERNAME for 5956 (1000:100)
Created /run/cgmanager/fs/net_cls/Me/lxc/CONTAINERNAME for 5956 (1000:100)
cgmanager:do_create_main: pid 5956 (uid 1000 gid 100) may not create
under
/run/cgmanager/fs/none,name=systemd/user.slice/user-1000.slice/session-1.scope
cgmanager_create: returning 0; existed is -1
Disconnected from private client
Connection from private client
Disconnected from private client
Connection from private client
MovePid: Client fd is: 6 (pid=5956, uid=1000, gid=100)
5964 moved to blkio:lxc/CONTAINERNAME by 5956's request
MovePid: Client fd is: 6 (pid=5956, uid=1000, gid=100)
5964 moved to cpu:lxc/CONTAINERNAME by 5956's request
MovePid: Client fd is: 6 (pid=5956, uid=1000, gid=100)
5964 moved to cpuset:lxc/CONTAINERNAME by 5956's request
MovePid: Client fd is: 6 (pid=5956, uid=1000, gid=100)
5964 moved to devices:lxc/CONTAINERNAME by 5956's request
MovePid: Client fd is: 6 (pid=5956, uid=1000, gid=100)
5964 moved to freezer:lxc/CONTAINERNAME by 5956's request
MovePid: Client fd is: 6 (pid=5956, uid=1000, gid=100)
5964 moved to memory:lxc/CONTAINERNAME by 5956's request
MovePid: Client fd is: 6 (pid=5956, uid=1000, gid=100)
5964 moved to net_cls:lxc/CONTAINERNAME by 5956's request
MovePid: Client fd is: 6 (pid=5956, uid=1000, gid=100)
cgmanager: Invalid path
/run/cgmanager/fs/none,name=systemd/user.slice/user-1000.slice/session-1.scope/lxc/CONTAINERNAME
cgmanager:per_ctrl_move_pid_main: Invalid path
/run/cgmanager/fs/none,name=systemd/user.slice/user-1000.slice/session-1.scope/lxc/CONTAINERNAME
Disconnected from private client
Connection from private client
Remove: Client fd is: 6 (pid=5956, uid=1000, gid=100)
Removed /run/cgmanager/fs/blkio/Me/lxc/CONTAINERNAME for 5956 (1000:100)
Removed /run/cgmanager/fs/cpu/Me/lxc/CONTAINERNAME for 5956 (1000:100)
Removed /run/cgmanager/fs/cpuset/Me/lxc/CONTAINERNAME for 5956 (1000:100)
Removed /run/cgmanager/fs/devices/Me/lxc/CONTAINERNAME for 5956 (1000:100)
Removed /run/cgmanager/fs/freezer/Me/lxc/CONTAINERNAME for 5956 (1000:100)
Removed /run/cgmanager/fs/memory/Me/lxc/CONTAINERNAME for 5956 (1000:100)
Removed /run/cgmanager/fs/net_cls/Me/lxc/CONTAINERNAME for 5956 (1000:100)
Disconnected from private client




The config I'm using:

    lxc.utsname = CONTAINERNAME
    lxc.arch = "x86_64"

    lxc.cgroup.devices.deny = a # Deny all access to devices
    lxc.tty = 4
    lxc.pts = 1024

    ## Capabilities
    lxc.cap.drop = audit_control audit_write mac_admin mac_override
mknod setfcap
    lxc.cap.drop = sys_boot sys_module sys_pacct sys_rawio sys_time

    ## Devices
    lxc.cgroup.devices.deny = a # Deny access to all devices

    # Allow to mknod all devices (but not using them)
    lxc.cgroup.devices.allow = c *:* m
    lxc.cgroup.devices.allow = b *:* m
    # /dev/null
    lxc.cgroup.devices.allow = c 1:3 rwm
    lxc.cgroup.devices.allow = c 1:5 rwm
    #consoles / tty
    lxc.cgroup.devices.allow = c 4:0 rwm
    lxc.cgroup.devices.allow = c 4:1 rwm
    #lxc.cgroup.devices.allow = c 4:2 rwm
    #lxc.cgroup.devices.allow = c 4:3 rwm
    lxc.cgroup.devices.allow = c 5:0 rwm
    lxc.cgroup.devices.allow = c 5:1 rwm
    # /dev/random
    lxc.cgroup.devices.allow = c 1:8 rwm
    lxc.cgroup.devices.allow = c 1:9 rwm
    # /dev/pts/*
    lxc.cgroup.devices.allow = c 5:2 rwm
    lxc.cgroup.devices.allow = c 136:* rwm
    #rtc
    lxc.cgroup.devices.allow = c 254:0 rwm
    # /dev/fuse
    lxc.cgroup.devices.allow = c 10:229 rwm


    # FIXME: a hack that it works! needs to be fixed properly (qknight)
    lxc.aa_allow_incomplete = 1

    ## Mounts
    lxc.mount.entry = /nix/store nix/store none defaults,bind.ro 0.0
    lxc.autodev = 1

    ## Network
    lxc.network.type = veth
    lxc.network.name = eth0
    lxc.network.flags = up
#    lxc.network.link = br0
    lxc.network.name = eth0
    lxc.network.mtu = 1500
#    lxc.network.ipv4 = 192.168.10.0/24

    #give USER access to /var/lib/containers/
    lxc.id_map = u 0 100000 65536
    lxc.id_map = g 0 100000 65536


    # FIXME: another hack (qknight)
    # When using LXC with apparmor, uncomment the next line to run
unconfined:
    lxc.aa_profile = unconfined





if someone has a working config or good advice, please share.
thanks very much.

regards,
joachim