[Nix-dev] execute LXC guest as none root user

[Joachim Schiele]

On 06.01.2016 12:03, Joachim Schiele wrote:
> hey,
> 
> we got lxc with NixOS as guest on NixOS as host working but we can only
> execute it using the 'root' users.
> 
> however, for security reasons it seems to be a good thing to use the
> kernel 'user namespaces' but i deliberately fail to get this running.
> 
> we are using nix-rehash from offlinehacker:
>   https://github.com/nixcloud/nix-rehash
> 
> 
> the latest issue when using:
>   /etc/subuid
>   /etc/subgid
> 
> 
> lxc-start -d -n "CONTAINERNAME" \
>   -f "container.conf" \
>   -s lxc.rootfs=$CONTAINER_ROOT \
>   -F
>   "/nix/store/05sy0bz81426798qzrj66m64ncb3pymd-nixos-15.09.756.88765a/init"
> Using /var/lib/containers/CONTAINERNAME as rootfs
> lxc-start: cgmanager.c: lxc_cgmanager_enter: 698 call to
> cgmanager_move_pid_sync failed: invalid request
> lxc-start: start.c: __lxc_start: 1172 failed to spawn 'CONTAINERNAME'
> lxc-start: lxc_start.c: main: 344 The container failed to start.
> lxc-start: lxc_start.c: main: 348 Additional information can be obtained
> by setting the --logfile and --logpriority options.
> 
> 
> cgmanager --debug:
> Connection from private client
> ListControllers: Client fd is: 6 (pid=5956, uid=1000, gid=100)
> Disconnected from private client
> Connection from private client
> Disconnected from private client
> Connection from private client
> Disconnected from private client
> Connection from private client
> ListControllers: Client fd is: 6 (pid=5958, uid=1000, gid=100)
> Disconnected from private client
> Connection from private client
> Disconnected from private client
> Connection from private client
> Create: Client fd is: 6 (pid=5956, uid=1000, gid=100)
> Created /run/cgmanager/fs/blkio/Me/lxc/CONTAINERNAME for 5956 (1000:100)
> Created /run/cgmanager/fs/cpu/Me/lxc/CONTAINERNAME for 5956 (1000:100)
> Created /run/cgmanager/fs/cpuset/Me/lxc/CONTAINERNAME for 5956 (1000:100)
> Created /run/cgmanager/fs/devices/Me/lxc/CONTAINERNAME for 5956 (1000:100)
> Created /run/cgmanager/fs/freezer/Me/lxc/CONTAINERNAME for 5956 (1000:100)
> Created /run/cgmanager/fs/memory/Me/lxc/CONTAINERNAME for 5956 (1000:100)
> Created /run/cgmanager/fs/net_cls/Me/lxc/CONTAINERNAME for 5956 (1000:100)
> cgmanager:do_create_main: pid 5956 (uid 1000 gid 100) may not create
> under
> /run/cgmanager/fs/none,name=systemd/user.slice/user-1000.slice/session-1.scope
> cgmanager_create: returning 0; existed is -1
> Disconnected from private client
> Connection from private client
> Disconnected from private client
> Connection from private client
> MovePid: Client fd is: 6 (pid=5956, uid=1000, gid=100)
> 5964 moved to blkio:lxc/CONTAINERNAME by 5956's request
> MovePid: Client fd is: 6 (pid=5956, uid=1000, gid=100)
> 5964 moved to cpu:lxc/CONTAINERNAME by 5956's request
> MovePid: Client fd is: 6 (pid=5956, uid=1000, gid=100)
> 5964 moved to cpuset:lxc/CONTAINERNAME by 5956's request
> MovePid: Client fd is: 6 (pid=5956, uid=1000, gid=100)
> 5964 moved to devices:lxc/CONTAINERNAME by 5956's request
> MovePid: Client fd is: 6 (pid=5956, uid=1000, gid=100)
> 5964 moved to freezer:lxc/CONTAINERNAME by 5956's request
> MovePid: Client fd is: 6 (pid=5956, uid=1000, gid=100)
> 5964 moved to memory:lxc/CONTAINERNAME by 5956's request
> MovePid: Client fd is: 6 (pid=5956, uid=1000, gid=100)
> 5964 moved to net_cls:lxc/CONTAINERNAME by 5956's request
> MovePid: Client fd is: 6 (pid=5956, uid=1000, gid=100)
> cgmanager: Invalid path
> /run/cgmanager/fs/none,name=systemd/user.slice/user-1000.slice/session-1.scope/lxc/CONTAINERNAME
> cgmanager:per_ctrl_move_pid_main: Invalid path
> /run/cgmanager/fs/none,name=systemd/user.slice/user-1000.slice/session-1.scope/lxc/CONTAINERNAME
> Disconnected from private client
> Connection from private client
> Remove: Client fd is: 6 (pid=5956, uid=1000, gid=100)
> Removed /run/cgmanager/fs/blkio/Me/lxc/CONTAINERNAME for 5956 (1000:100)
> Removed /run/cgmanager/fs/cpu/Me/lxc/CONTAINERNAME for 5956 (1000:100)
> Removed /run/cgmanager/fs/cpuset/Me/lxc/CONTAINERNAME for 5956 (1000:100)
> Removed /run/cgmanager/fs/devices/Me/lxc/CONTAINERNAME for 5956 (1000:100)
> Removed /run/cgmanager/fs/freezer/Me/lxc/CONTAINERNAME for 5956 (1000:100)
> Removed /run/cgmanager/fs/memory/Me/lxc/CONTAINERNAME for 5956 (1000:100)
> Removed /run/cgmanager/fs/net_cls/Me/lxc/CONTAINERNAME for 5956 (1000:100)
> Disconnected from private client
> 
> 
> 
> 
> The config I'm using:
> 
>     lxc.utsname = CONTAINERNAME
>     lxc.arch = "x86_64"
> 
>     lxc.cgroup.devices.deny = a # Deny all access to devices
>     lxc.tty = 4
>     lxc.pts = 1024
> 
>     ## Capabilities
>     lxc.cap.drop = audit_control audit_write mac_admin mac_override
> mknod setfcap
>     lxc.cap.drop = sys_boot sys_module sys_pacct sys_rawio sys_time
> 
>     ## Devices
>     lxc.cgroup.devices.deny = a # Deny access to all devices
> 
>     # Allow to mknod all devices (but not using them)
>     lxc.cgroup.devices.allow = c *:* m
>     lxc.cgroup.devices.allow = b *:* m
>     # /dev/null
>     lxc.cgroup.devices.allow = c 1:3 rwm
>     lxc.cgroup.devices.allow = c 1:5 rwm
>     #consoles / tty
>     lxc.cgroup.devices.allow = c 4:0 rwm
>     lxc.cgroup.devices.allow = c 4:1 rwm
>     #lxc.cgroup.devices.allow = c 4:2 rwm
>     #lxc.cgroup.devices.allow = c 4:3 rwm
>     lxc.cgroup.devices.allow = c 5:0 rwm
>     lxc.cgroup.devices.allow = c 5:1 rwm
>     # /dev/random
>     lxc.cgroup.devices.allow = c 1:8 rwm
>     lxc.cgroup.devices.allow = c 1:9 rwm
>     # /dev/pts/*
>     lxc.cgroup.devices.allow = c 5:2 rwm
>     lxc.cgroup.devices.allow = c 136:* rwm
>     #rtc
>     lxc.cgroup.devices.allow = c 254:0 rwm
>     # /dev/fuse
>     lxc.cgroup.devices.allow = c 10:229 rwm
> 
> 
>     # FIXME: a hack that it works! needs to be fixed properly (qknight)
>     lxc.aa_allow_incomplete = 1
> 
>     ## Mounts
>     lxc.mount.entry = /nix/store nix/store none defaults,bind.ro 0.0
>     lxc.autodev = 1
> 
>     ## Network
>     lxc.network.type = veth
>     lxc.network.name = eth0
>     lxc.network.flags = up
> #    lxc.network.link = br0
>     lxc.network.name = eth0
>     lxc.network.mtu = 1500
> #    lxc.network.ipv4 = 192.168.10.0/24
> 
>     #give USER access to /var/lib/containers/
>     lxc.id_map = u 0 100000 65536
>     lxc.id_map = g 0 100000 65536
> 
> 
>     # FIXME: another hack (qknight)
>     # When using LXC with apparmor, uncomment the next line to run
> unconfined:
>     lxc.aa_profile = unconfined
> 
> 
> 
> 
> 
> if someone has a working config or good advice, please share.
> thanks very much.

maybe this is the issue:
unprivileged containers does not yet work with systemd:
  https://www.flockport.com/lxc-and-lxd-support-across-distributions/