I have a LXC cluster with three nodes that are all KVM virtual machines with Kubuntu 18.04 with LXD 3.0.3 installed via apt
Also the host machine is a Kubuntu 18.04
I installed LXD on the first node (kvmnode1) and then I joined to the cluster the other two nodes (kvmnode2, kvmnode3)
Then I verified that starting the nodes from the first to the last (waiting for every node to complete the startup) and closing the nodes from the last to the first always gives a fully oerational cluster
Today I forgot to wait for the complete startup of every node and I ended up with a cluster with the first node non starting LXD end the other 2 nodes working correctly
On the failing first node I see
sysop@kvmnode1:~/Scaricati$ sudo systemctl status lxd
[sudo] password di sysop:
● lxd.service - LXD - main daemon
Loaded: loaded (/lib/systemd/system/lxd.service; indirect; vendor preset: enabled)
Active: activating (start-post) (Result: exit-code) since Mon 2019-04-22 10:36:10 CEST; 1min 47s ago
Docs: man:lxd(1)
Process: 1592 ExecStart=/usr/bin/lxd --group lxd --logfile=/var/log/lxd/lxd.log (code=exited, status=1/FAILURE)
Process: 1560 ExecStartPre=/usr/lib/x86_64-linux-gnu/lxc/lxc-apparmor-load (code=exited, status=0/SUCCESS)
Main PID: 1592 (code=exited, status=1/FAILURE); Control PID: 1593 (lxd)
Tasks: 8
CGroup: /system.slice/lxd.service
└─1593 /usr/lib/lxd/lxd waitready --timeout=600
apr 22 10:36:10 kvmnode1 systemd[1]: Starting LXD - main daemon...
apr 22 10:36:10 kvmnode1 lxd[1592]: t=2019-04-22T10:36:10+0200 lvl=warn msg="CGroup memory swap accounting is disabled, swap limits will be ignored."
apr 22 10:36:19 kvmnode1 lxd[1592]: t=2019-04-22T10:36:19+0200 lvl=eror msg="Failed to start the daemon: failed to open cluster database: failed to ensure schema: failed to update node version info: upda
apr 22 10:36:19 kvmnode1 lxd[1592]: Error: failed to open cluster database: failed to ensure schema: failed to update node version info: updated 0 rows instead of 1
apr 22 10:36:19 kvmnode1 systemd[1]: lxd.service: Main process exited, code=exited, status=1/FAILURE
sysop@kvmnode1:~/Scaricati$
And on the second node I see
sysop@kvmnode2:~$ lxc cluster list
+----------+-----------------------------+----------+---------+----------------------------------------+
| NAME | URL | DATABASE | STATE | MESSAGE |
+----------+-----------------------------+----------+---------+----------------------------------------+
| kvmnode1 | https://192.168.201.11:8443 | YES | OFFLINE | no heartbeat since 134h36m2.926365228s |
+----------+-----------------------------+----------+---------+----------------------------------------+
| kvmnode2 | https://192.168.201.12:8443 | YES | ONLINE | fully operational |
+----------+-----------------------------+----------+---------+----------------------------------------+
| kvmnode3 | https://192.168.201.13:8443 | YES | ONLINE | fully operational |
+----------+-----------------------------+----------+---------+----------------------------------------+
sysop@kvmnode2:~$
What can I do to have kvmnode1 starting correctly and join the cluster?
P.S. I tried to start/stop the three nodes in every sequence but I always end in the same status
UPPDATE
I tried to start lxd in debug mode and it report the same failure
sysop@kvmnode1:~$ sudo lxd --debug --group lxd
DBUG[04-22|12:19:42] Connecting to a local LXD over a Unix socket
DBUG[04-22|12:19:42] Sending request to LXD method=GET url=http://unix.socket/1.0 etag=
INFO[04-22|12:19:42] LXD 3.0.3 is starting in normal mode path=/var/lib/lxd
INFO[04-22|12:19:42] Kernel uid/gid map:
INFO[04-22|12:19:42] - u 0 0 4294967295
INFO[04-22|12:19:42] - g 0 0 4294967295
INFO[04-22|12:19:42] Configured LXD uid/gid map:
INFO[04-22|12:19:42] - u 0 165536 65536
INFO[04-22|12:19:42] - g 0 165536 65536
WARN[04-22|12:19:42] CGroup memory swap accounting is disabled, swap limits will be ignored.
INFO[04-22|12:19:42] Kernel features:
INFO[04-22|12:19:42] - netnsid-based network retrieval: no
INFO[04-22|12:19:42] - unprivileged file capabilities: yes
INFO[04-22|12:19:42] Initializing local database
DBUG[04-22|12:19:42] Initializing database gateway
DBUG[04-22|12:19:42] Connecting to a local LXD over a Unix socket
DBUG[04-22|12:19:42] Sending request to LXD method=GET url=http://unix.socket/1.0 etag=
DBUG[04-22|12:19:42] Detected stale unix socket, deleting
INFO[04-22|12:19:42] Starting /dev/lxd handler:
INFO[04-22|12:19:42] - binding devlxd socket socket=/var/lib/lxd/devlxd/sock
INFO[04-22|12:19:42] REST API daemon:
INFO[04-22|12:19:42] - binding Unix socket socket=/var/lib/lxd/unix.socket
INFO[04-22|12:19:42] - binding TCP socket socket=[::]:8443
INFO[04-22|12:19:42] Initializing global database
DBUG[04-22|12:19:42] Found cert k=0
DBUG[04-22|12:19:42] Dqlite: server connection failed err=failed to establish network connection: some nodes are behind this node's version address=192.168.201.11:8443 attempt=0
DBUG[04-22|12:19:42] Dqlite: connected address=192.168.201.12:8443 attempt=0
DBUG[04-22|12:19:42] Database error: failed to update node version info: updated 0 rows instead of 1
EROR[04-22|12:19:42] Failed to start the daemon: failed to open cluster database: failed to ensure schema: failed to update node version info: updated 0 rows instead of 1
INFO[04-22|12:19:42] Starting shutdown sequence
INFO[04-22|12:19:42] Stopping REST API handler:
INFO[04-22|12:19:42] - closing socket socket=[::]:8443
INFO[04-22|12:19:42] - closing socket socket=/var/lib/lxd/unix.socket
INFO[04-22|12:19:42] Stopping /dev/lxd handler
INFO[04-22|12:19:42] - closing socket socket=/var/lib/lxd/devlxd/sock
DBUG[04-22|12:19:42] Stop database gateway
INFO[04-22|12:19:42] Stopping REST API handler:
INFO[04-22|12:19:42] Stopping /dev/lxd handler
INFO[04-22|12:19:42] Stopping REST API handler:
INFO[04-22|12:19:42] Stopping /dev/lxd handler
DBUG[04-22|12:19:42] Not unmounting temporary filesystems (containers are still running)
INFO[04-22|12:19:42] Saving simplestreams cache
INFO[04-22|12:19:42] Saved simplestreams cache
Error: failed to open cluster database: failed to ensure schema: failed to update node version info: updated 0 rows instead of 1
sysop@kvmnode1:~$
Any hint?
0 Answers