My CentOS LXC containers are not starting anymore on an Ubuntu 14.10 machine. I think that the problem started after a reboot, but I am not sure.
I had a similar problem after an yum update, when the init scripts were replaced with stock ones that are not LXC aware. They were trying to start udev, etc... But this timeI have this problem for all CentOS instances, even for those newly created.
Host OS: Ubuntu14.10 64bit
Guest OS: Centos 6.5 64bit
root@ubuntu-mvutcovici:~# lxc-start --logfile stash-lxc.log --logpriority DEBUG -dn stash
lxc-start: lxc_start.c: main: 337 The container failed to start.
lxc-start: lxc_start.c: main: 339 To get more details, run the container in foreground mode.
lxc-start: lxc_start.c: main: 341 Additional information can be obtained by setting the --logfile and --logpriority options.
root@ubuntu-mvutcovici:~#
Here is the content of stash-lxc.log file:
lxc-start 1416596262.928 INFO lxc_start_ui - lxc_start.c:main:265 - using rcfile /var/lib/lxc/stash/config
lxc-start 1416596262.928 WARN lxc_confile - confile.c:config_pivotdir:1685 - lxc.pivotdir is ignored. It will soon become an error.
lxc-start 1416596262.928 WARN lxc_log - log.c:lxc_log_init:316 - lxc_log_init called with log already initialized
lxc-start 1416596262.929 INFO lxc_start - start.c:lxc_check_inherited:209 - closed inherited fd 4
lxc-start 1416596262.934 INFO lxc_lsm - lsm/lsm.c:lsm_init:48 - LSM security driver AppArmor
lxc-start 1416596262.934 INFO lxc_seccomp - seccomp.c:parse_config_v2:285 - processing: .[all].
lxc-start 1416596262.934 INFO lxc_seccomp - seccomp.c:parse_config_v2:285 - processing: .kexec_load errno 1.
lxc-start 1416596262.934 INFO lxc_seccomp - seccomp.c:parse_config_v2:358 - Adding non-compat rule for kexec_load action 327681
lxc-start 1416596262.934 INFO lxc_seccomp - seccomp.c:parse_config_v2:369 - Adding compat rule for kexec_load action 327681
lxc-start 1416596262.934 INFO lxc_seccomp - seccomp.c:parse_config_v2:382 - Really adding compat rule bc nr1 == nr2 (283, 246)
lxc-start 1416596262.934 INFO lxc_seccomp - seccomp.c:parse_config_v2:285 - processing: .open_by_handle_at errno 1.
lxc-start 1416596262.934 INFO lxc_seccomp - seccomp.c:parse_config_v2:358 - Adding non-compat rule for open_by_handle_at action 327681
lxc-start 1416596262.934 INFO lxc_seccomp - seccomp.c:parse_config_v2:369 - Adding compat rule for open_by_handle_at action 327681
lxc-start 1416596262.934 INFO lxc_seccomp - seccomp.c:parse_config_v2:382 - Really adding compat rule bc nr1 == nr2 (342, 304)
lxc-start 1416596262.934 INFO lxc_seccomp - seccomp.c:parse_config_v2:285 - processing: .init_module errno 1.
lxc-start 1416596262.934 INFO lxc_seccomp - seccomp.c:parse_config_v2:358 - Adding non-compat rule for init_module action 327681
lxc-start 1416596262.934 INFO lxc_seccomp - seccomp.c:parse_config_v2:369 - Adding compat rule for init_module action 327681
lxc-start 1416596262.934 INFO lxc_seccomp - seccomp.c:parse_config_v2:382 - Really adding compat rule bc nr1 == nr2 (128, 175)
lxc-start 1416596262.934 INFO lxc_seccomp - seccomp.c:parse_config_v2:285 - processing: .finit_module errno 1.
lxc-start 1416596262.934 INFO lxc_seccomp - seccomp.c:parse_config_v2:358 - Adding non-compat rule for finit_module action 327681
lxc-start 1416596262.934 WARN lxc_seccomp - seccomp.c:do_resolve_add_rule:196 - Seccomp: got negative # for syscall: finit_module
lxc-start 1416596262.934 WARN lxc_seccomp - seccomp.c:do_resolve_add_rule:197 - This syscall will NOT be blacklisted
lxc-start 1416596262.934 INFO lxc_seccomp - seccomp.c:parse_config_v2:369 - Adding compat rule for finit_module action 327681
lxc-start 1416596262.934 INFO lxc_seccomp - seccomp.c:parse_config_v2:377 - Adding non-compat rule bc nr1 == nr2 (-10085, -10085)
lxc-start 1416596262.934 WARN lxc_seccomp - seccomp.c:do_resolve_add_rule:196 - Seccomp: got negative # for syscall: finit_module
lxc-start 1416596262.934 WARN lxc_seccomp - seccomp.c:do_resolve_add_rule:197 - This syscall will NOT be blacklisted
lxc-start 1416596262.934 INFO lxc_seccomp - seccomp.c:parse_config_v2:285 - processing: .delete_module errno 1.
lxc-start 1416596262.934 INFO lxc_seccomp - seccomp.c:parse_config_v2:358 - Adding non-compat rule for delete_module action 327681
lxc-start 1416596262.934 INFO lxc_seccomp - seccomp.c:parse_config_v2:369 - Adding compat rule for delete_module action 327681
lxc-start 1416596262.935 INFO lxc_seccomp - seccomp.c:parse_config_v2:382 - Really adding compat rule bc nr1 == nr2 (129, 176)
lxc-start 1416596262.935 INFO lxc_seccomp - seccomp.c:parse_config_v2:390 - Merging in the compat seccomp ctx into the main one
lxc-start 1416596262.935 DEBUG lxc_conf - conf.c:lxc_create_tty:3504 - allocated pty '/dev/pts/2' (5/6)
lxc-start 1416596262.935 DEBUG lxc_conf - conf.c:lxc_create_tty:3504 - allocated pty '/dev/pts/4' (7/8)
lxc-start 1416596262.935 DEBUG lxc_conf - conf.c:lxc_create_tty:3504 - allocated pty '/dev/pts/5' (9/10)
lxc-start 1416596262.935 DEBUG lxc_conf - conf.c:lxc_create_tty:3504 - allocated pty '/dev/pts/7' (11/12)
lxc-start 1416596262.935 INFO lxc_conf - conf.c:lxc_create_tty:3515 - tty's configured
lxc-start 1416596262.935 DEBUG lxc_start - start.c:setup_signal_fd:247 - sigchild handler set
lxc-start 1416596262.935 DEBUG lxc_console - console.c:lxc_console_peer_default:536 - no console peer
lxc-start 1416596262.935 INFO lxc_start - start.c:lxc_init:443 - 'stash' is initialized
lxc-start 1416596262.936 DEBUG lxc_start - start.c:__lxc_start:1061 - Not dropping cap_sys_boot or watching utmp
lxc-start 1416596262.936 INFO lxc_start - start.c:lxc_check_inherited:209 - closed inherited fd 4
lxc-start 1416596262.940 INFO lxc_monitor - monitor.c:lxc_monitor_sock_name:177 - using monitor sock name lxc/ad055575fe28ddd5//var/lib/lxc
lxc-start 1416596262.943 DEBUG lxc_conf - conf.c:instanciate_veth:2842 - instanciated veth 'vethF4JUT8/vethVOPS0P', index is '11'
lxc-start 1416596262.943 INFO lxc_cgroup - cgroup.c:cgroup_init:62 - cgroup driver cgmanager initing for stash
lxc-start 1416596262.948 INFO lxc_cgmanager - cgmanager.c:cgm_setup_limits:1241 - cgroup limits have been setup
lxc-start 1416596262.977 DEBUG lxc_conf - conf.c:lxc_assign_network:3259 - move '(null)' to '11664'
lxc-start 1416596262.978 DEBUG lxc_conf - conf.c:setup_rootfs:1536 - mounted '/var/lib/lxc/stash/rootfs' on '/usr/lib/x86_64-linux-gnu/lxc'
lxc-start 1416596262.978 INFO lxc_conf - conf.c:setup_utsname:896 - 'stash' hostname has been setup
lxc-start 1416596263.005 DEBUG lxc_conf - conf.c:setup_hw_addr:2392 - mac address 'fe:fb:95:37:ac:3c' on 'eth0' has been setup
lxc-start 1416596263.005 DEBUG lxc_conf - conf.c:setup_netdev:2619 - 'eth0' has been setup
lxc-start 1416596263.005 INFO lxc_conf - conf.c:setup_network:2640 - network has been setup
lxc-start 1416596263.005 INFO lxc_conf - conf.c:setup_ttydir_console:1688 - created /usr/lib/x86_64-linux-gnu/lxc/dev/lxc
lxc-start 1416596263.005 INFO lxc_conf - conf.c:setup_ttydir_console:1734 - console has been setup on lxc/console
lxc-start 1416596263.006 INFO lxc_conf - conf.c:setup_tty:1023 - 4 tty(s) has been setup
lxc-start 1416596263.006 INFO lxc_conf - conf.c:do_tmp_proc_mount:3809 - I am 1, /proc/self points to '1'
lxc-start 1416596263.029 DEBUG lxc_conf - conf.c:setup_rootfs_pivot_root:1078 - pivot_root syscall to '/usr/lib/x86_64-linux-gnu/lxc' successful
lxc-start 1416596263.029 INFO lxc_conf - conf.c:setup_pts:1605 - created new pts instance
lxc-start 1416596263.029 INFO lxc_conf - conf.c:setup_personality:1622 - set personality to '0x0'
lxc-start 1416596263.029 DEBUG lxc_conf - conf.c:setup_caps:2303 - drop capability 'mac_admin' (33)
lxc-start 1416596263.029 DEBUG lxc_conf - conf.c:setup_caps:2303 - drop capability 'mac_override' (32)
lxc-start 1416596263.029 DEBUG lxc_conf - conf.c:setup_caps:2303 - drop capability 'sys_time' (25)
lxc-start 1416596263.029 DEBUG lxc_conf - conf.c:setup_caps:2303 - drop capability 'sys_module' (16)
lxc-start 1416596263.029 DEBUG lxc_conf - conf.c:setup_caps:2303 - drop capability 'setfcap' (31)
lxc-start 1416596263.029 DEBUG lxc_conf - conf.c:setup_caps:2303 - drop capability 'setpcap' (8)
lxc-start 1416596263.029 DEBUG lxc_conf - conf.c:setup_caps:2303 - drop capability 'sys_nice' (23)
lxc-start 1416596263.029 DEBUG lxc_conf - conf.c:setup_caps:2303 - drop capability 'sys_pacct' (20)
lxc-start 1416596263.029 DEBUG lxc_conf - conf.c:setup_caps:2303 - drop capability 'sys_rawio' (17)
lxc-start 1416596263.029 DEBUG lxc_conf - conf.c:setup_caps:2312 - capabilities have been setup
lxc-start 1416596263.029 NOTICE lxc_conf - conf.c:lxc_setup:4144 - 'stash' is setup.
lxc-start 1416596263.029 DEBUG lxc_cgmanager - cgmanager.c:cgm_setup_limits:1237 - cgroup 'devices.deny' set to 'a'
lxc-start 1416596263.029 DEBUG lxc_cgmanager - cgmanager.c:cgm_setup_limits:1237 - cgroup 'devices.allow' set to 'c *:* m'
lxc-start 1416596263.030 DEBUG lxc_cgmanager - cgmanager.c:cgm_setup_limits:1237 - cgroup 'devices.allow' set to 'b *:* m'
lxc-start 1416596263.030 DEBUG lxc_cgmanager - cgmanager.c:cgm_setup_limits:1237 - cgroup 'devices.allow' set to 'c 1:3 rwm'
lxc-start 1416596263.030 DEBUG lxc_cgmanager - cgmanager.c:cgm_setup_limits:1237 - cgroup 'devices.allow' set to 'c 1:5 rwm'
lxc-start 1416596263.030 DEBUG lxc_cgmanager - cgmanager.c:cgm_setup_limits:1237 - cgroup 'devices.allow' set to 'c 1:7 rwm'
lxc-start 1416596263.031 DEBUG lxc_cgmanager - cgmanager.c:cgm_setup_limits:1237 - cgroup 'devices.allow' set to 'c 5:0 rwm'
lxc-start 1416596263.031 DEBUG lxc_cgmanager - cgmanager.c:cgm_setup_limits:1237 - cgroup 'devices.allow' set to 'c 5:1 rwm'
lxc-start 1416596263.031 DEBUG lxc_cgmanager - cgmanager.c:cgm_setup_limits:1237 - cgroup 'devices.allow' set to 'c 5:2 rwm'
lxc-start 1416596263.031 DEBUG lxc_cgmanager - cgmanager.c:cgm_setup_limits:1237 - cgroup 'devices.allow' set to 'c 1:8 rwm'
lxc-start 1416596263.031 DEBUG lxc_cgmanager - cgmanager.c:cgm_setup_limits:1237 - cgroup 'devices.allow' set to 'c 1:9 rwm'
lxc-start 1416596263.031 DEBUG lxc_cgmanager - cgmanager.c:cgm_setup_limits:1237 - cgroup 'devices.allow' set to 'c 136:* rwm'
lxc-start 1416596263.031 INFO lxc_cgmanager - cgmanager.c:cgm_setup_limits:1241 - cgroup limits have been setup
lxc-start 1416596263.031 ERROR lxc_apparmor - lsm/apparmor.c:mount_feature_enabled:61 - Permission denied - Error mounting securityfs
lxc-start 1416596263.032 WARN lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:184 - Incomplete AppArmor support in your kernel
lxc-start 1416596263.032 ERROR lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:186 - If you really want to start this container, set
lxc-start 1416596263.032 ERROR lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:187 - lxc.aa_allow_incomplete = 1
lxc-start 1416596263.032 ERROR lxc_apparmor - lsm/apparmor.c:apparmor_process_label_set:188 - in your container configuration file
lxc-start 1416596263.032 ERROR lxc_sync - sync.c:__sync_wait:51 - invalid sequence number 1. expected 4
lxc-start 1416596263.032 ERROR lxc_start - start.c:__lxc_start:1087 - failed to spawn 'stash'
lxc-start 1416596263.032 WARN lxc_commands - commands.c:lxc_cmd_rsp_recv:172 - command get_init_pid failed to receive response
lxc-start 1416596263.032 ERROR lxc_cgmanager - cgmanager.c:cgm_remove_cgroup:503 - call to cgmanager_remove_sync failed: invalid request
lxc-start 1416596263.032 ERROR lxc_cgmanager - cgmanager.c:cgm_remove_cgroup:505 - Error removing name=systemd:lxc/stash-3
lxc-start 1416596263.032 WARN lxc_cgmanager - cgmanager.c:cgm_get:946 - do_cgm_get exited with error
lxc-start 1416596263.032 ERROR lxc_cgmanager - cgmanager.c:cgm_remove_cgroup:503 - call to cgmanager_remove_sync failed: invalid request
lxc-start 1416596263.032 ERROR lxc_cgmanager - cgmanager.c:cgm_remove_cgroup:505 - Error removing perf_event:lxc/stash-3
lxc-start 1416596263.033 ERROR lxc_cgmanager - cgmanager.c:cgm_remove_cgroup:503 - call to cgmanager_remove_sync failed: invalid request
lxc-start 1416596263.033 ERROR lxc_cgmanager - cgmanager.c:cgm_remove_cgroup:505 - Error removing net_prio:lxc/stash-3
lxc-start 1416596263.033 ERROR lxc_cgmanager - cgmanager.c:cgm_remove_cgroup:503 - call to cgmanager_remove_sync failed: invalid request
lxc-start 1416596263.033 ERROR lxc_cgmanager - cgmanager.c:cgm_remove_cgroup:505 - Error removing net_cls:lxc/stash-3
lxc-start 1416596263.033 ERROR lxc_cgmanager - cgmanager.c:cgm_remove_cgroup:503 - call to cgmanager_remove_sync failed: invalid request
lxc-start 1416596263.033 ERROR lxc_cgmanager - cgmanager.c:cgm_remove_cgroup:505 - Error removing memory:lxc/stash-3
lxc-start 1416596263.033 ERROR lxc_cgmanager - cgmanager.c:cgm_remove_cgroup:503 - call to cgmanager_remove_sync failed: invalid request
lxc-start 1416596263.033 ERROR lxc_cgmanager - cgmanager.c:cgm_remove_cgroup:505 - Error removing hugetlb:lxc/stash-3
lxc-start 1416596263.033 ERROR lxc_cgmanager - cgmanager.c:cgm_remove_cgroup:503 - call to cgmanager_remove_sync failed: invalid request
lxc-start 1416596263.033 ERROR lxc_cgmanager - cgmanager.c:cgm_remove_cgroup:505 - Error removing freezer:lxc/stash-3
lxc-start 1416596263.034 ERROR lxc_cgmanager - cgmanager.c:cgm_remove_cgroup:503 - call to cgmanager_remove_sync failed: invalid request
lxc-start 1416596263.034 ERROR lxc_cgmanager - cgmanager.c:cgm_remove_cgroup:505 - Error removing devices:lxc/stash-3
lxc-start 1416596263.034 ERROR lxc_cgmanager - cgmanager.c:cgm_remove_cgroup:503 - call to cgmanager_remove_sync failed: invalid request
lxc-start 1416596263.034 ERROR lxc_cgmanager - cgmanager.c:cgm_remove_cgroup:505 - Error removing cpuset:lxc/stash-3
lxc-start 1416596263.034 ERROR lxc_cgmanager - cgmanager.c:cgm_remove_cgroup:503 - call to cgmanager_remove_sync failed: invalid request
lxc-start 1416596263.034 ERROR lxc_cgmanager - cgmanager.c:cgm_remove_cgroup:505 - Error removing cpuacct:lxc/stash-3
lxc-start 1416596263.034 ERROR lxc_cgmanager - cgmanager.c:cgm_remove_cgroup:503 - call to cgmanager_remove_sync failed: invalid request
lxc-start 1416596263.034 ERROR lxc_cgmanager - cgmanager.c:cgm_remove_cgroup:505 - Error removing cpu:lxc/stash-3
lxc-start 1416596263.035 ERROR lxc_cgmanager - cgmanager.c:cgm_remove_cgroup:503 - call to cgmanager_remove_sync failed: invalid request
lxc-start 1416596263.035 ERROR lxc_cgmanager - cgmanager.c:cgm_remove_cgroup:505 - Error removing blkio:lxc/stash-3
lxc-start 1416596268.038 ERROR lxc_start_ui - lxc_start.c:main:337 - The container failed to start.
lxc-start 1416596268.038 ERROR lxc_start_ui - lxc_start.c:main:339 - To get more details, run the container in foreground mode.
lxc-start 1416596268.038 ERROR lxc_start_ui - lxc_start.c:main:341 - Additional information can be obtained by setting the --logfile and --logpriority options.
To create all those CentOS instances I've used:
root@ubuntu-mvutcovici:~# lxc-create -t centos -f lxc-mircea.conf -n stash
root@ubuntu-mvutcovici:~# cat lxc-mircea.conf
lxc.network.type = veth
lxc.network.link = br0
lxc.network.flags = up
EDIT: It seems that adding lxc.aa_allow_incomplete = 1 into /var/lib/lxc/stash/config file is a workaround for the start problem. How can I make app armor coexist again with CentOS LXC?
From lxc.container.conf man page:
lxc.aa_allow_incomplete
Apparmor profiles are pathname based. Therefore many file restrictions require mount restrictions to be effective against a determined attacker. However, these mount restrictions are not yet implemented in the
upstream kernel. Without the mount restrictions, the apparmor profiles still protect against accidental damager.
If this flag is 0 (default), then the container will not be started if the kernel lacks the apparmor mount features, so that a regression after a kernel upgrade will be detected. To start the container under partial
apparmor protection, set this flag to 1.
EDIT2: added original /var/lib/lxc/stash/config file:
# Template used to create this container: /usr/share/lxc/templates/lxc-centos
# Parameters passed to the template:
# For additional config options, please look at lxc.container.conf(5)
lxc.network.type = veth
lxc.network.link = br0
lxc.network.hwaddr = fe:98:41:37:ca:3d
lxc.network.flags = up
lxc.rootfs = /var/lib/lxc/stash/rootfs
# Include common configuration
lxc.include = /usr/share/lxc/config/centos.common.conf
lxc.arch = x86_64
lxc.utsname = stash
lxc.autodev = 0
# When using LXC with apparmor, uncomment the next line to run unconfined:
#lxc.aa_profile = unconfined
# example simple networking setup, uncomment to enable
#lxc.network.type = veth
#lxc.network.flags = up
#lxc.network.link = lxcbr0
#lxc.network.name = eth0
# Additional example for veth network type
# static MAC address,
#lxc.network.hwaddr = 00:16:3e:77:52:20
# persistent veth device name on host side
# Note: This may potentially collide with other containers of same name!
#lxc.network.veth.pair = v-stash-e0
