cape's questions -server

cape

Asked: 2021-06-22 05:03:34 +0800 CST

Clustered vsftpd service stops working with no error log

We've got a pacemaker cluster of vsftpd server made of two nodes in a active/pasive scheme. We have experienced the following behaviour.

vsftpd.log stops recording activity while xferlog keeps recording (transfers keeps ocurring). vsftpd.log shows no error or warn message.
Between 10 to 12 hours later service is unavailable and no new connections are happening
We try to move resource but it fails a lot, we need to perform multiple clean-ups before service properly starts on the pasive node.

corsync config:

totem {
    version: 2
    secauth: off
    cluster_name: myCluster
    transport: udpu
}

nodelist {
    node {
        ring0_addr: node1
        nodeid: 1
    }

    node {
        ring0_addr: node2
        nodeid: 2
    }
}

quorum {
    provider: corosync_votequorum
    two_node: 1
}

logging {
    to_logfile: yes
    logfile: /var/log/cluster/corosync.log
    to_syslog: yes
}

vsftpd.conf

anonymous_enable=NO
local_enable=YES
write_enable=YES
local_umask=002
allow_ftpd_full_access
dirmessage_enable=YES
xferlog_enable=YES
connect_from_port_20=YES
xferlog_file=/var/log/xferlog

chroot_local_user=YES
chroot_list_enable=YES
chroot_list_file=/etc/vsftpd/chroot_list

listen=NO
listen_ipv6=YES

pam_service_name=vsftpd
userlist_enable=YES
tcp_wrappers=YES
allow_writeable_chroot=YES
dual_log_enable=YES
vsftpd_log_file=/var/log/vsftpd.log

The only significant log I've found it that produced by journalctl, which shows timeout while trying to stop server and a huge number of failure, that to be honest I don't how they happen.

Jun 09 17:50:02 node2.company.corp pengine[4720]:   notice: Stop    FTP_SERVICE        (node2.company.corp)
Jun 09 17:50:02 node2.company.corp pengine[4720]:   notice: Calculated transition 5460, saving inputs in /var/lib/pacemaker/pengine/pe-input-1567.bz2
Jun 09 17:50:02 node2.company.corp crmd[4721]:   notice: Initiating stop operation FTP_SERVICE_stop_0 locally on node2.company.corp | action 8
Jun 09 17:50:02 node2.company.corp systemd[1]: Reloading.
Jun 09 17:50:02 node2.company.corp systemd[1]: Stopping Vsftpd ftp daemon...
-- Subject: Unit vsftpd.service has begun shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit vsftpd.service has begun shutting down.
Jun 09 17:50:22 node2.company.corp lrmd[4718]:   notice: Giving up on FTP_SERVICE stop (rc=196): timeout (elapsed=19988ms, remaining=12ms)
Jun 09 17:50:22 node2.company.corp crmd[4721]:    error: Result of stop operation for FTP_SERVICE on node2.company.corp: Timed Out | call=255 key=FTP_SERVICE_stop_0 timeout=20000ms
Jun 09 17:50:22 node2.company.corp crmd[4721]:  warning: Action 8 (FTP_SERVICE_stop_0) on node2.company.corp failed (target: 0 vs. rc: 198): Error
Jun 09 17:50:22 node2.company.corp crmd[4721]:   notice: Transition aborted by operation FTP_SERVICE_stop_0 'modify' on node2.company.corp: Event failed | magic=2:198;8:5460:0:bf959758-83cf-4e4b-bd33-a360ffa08557 cib=0.449.1 source=match_graph_event:310 complete=false
Jun 09 17:50:22 node2.company.corp crmd[4721]:  warning: Action 8 (FTP_SERVICE_stop_0) on node2.company.corp failed (target: 0 vs. rc: 198): Error
Jun 09 17:50:22 node2.company.corp crmd[4721]:   notice: Transition 5460 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=1, Source=/var/lib/pacemaker/pengine/pe-input-1567.bz2): Complete
Jun 09 17:50:22 node2.company.corp pengine[4720]:   notice: On loss of CCM Quorum: Ignore
Jun 09 17:50:22 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:50:22 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:50:22 node2.company.corp pengine[4720]:   notice: Calculated transition 5461, saving inputs in /var/lib/pacemaker/pengine/pe-input-1568.bz2
Jun 09 17:50:22 node2.company.corp pengine[4720]:   notice: On loss of CCM Quorum: Ignore
Jun 09 17:50:22 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:50:22 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:50:22 node2.company.corp pengine[4720]:  warning: Forcing FTP_SERVICE away from node2.company.corp after 1000000 failures (max=1000000)
Jun 09 17:50:22 node2.company.corp pengine[4720]:   notice: Calculated transition 5462, saving inputs in /var/lib/pacemaker/pengine/pe-input-1569.bz2
Jun 09 17:50:22 node2.company.corp crmd[4721]:   notice: Transition 5462 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-1569.bz2): Complete
Jun 09 17:50:22 node2.company.corp crmd[4721]:   notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd
Jun 09 17:50:26 node2.company.corp crmd[4721]:   notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph
Jun 09 17:50:26 node2.company.corp pengine[4720]:   notice: On loss of CCM Quorum: Ignore
Jun 09 17:50:26 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:50:26 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:50:26 node2.company.corp pengine[4720]:  warning: Forcing FTP_SERVICE away from node2.company.corp after 1000000 failures (max=1000000)
Jun 09 17:50:26 node2.company.corp pengine[4720]:   notice: Calculated transition 5463, saving inputs in /var/lib/pacemaker/pengine/pe-input-1570.bz2
Jun 09 17:50:26 node2.company.corp crmd[4721]:   notice: Transition 5463 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-1570.bz2): Complete
Jun 09 17:50:26 node2.company.corp crmd[4721]:   notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd
Jun 09 17:50:33 node2.company.corp crmd[4721]:  warning: No reason to expect node 2 to be down
Jun 09 17:50:33 node2.company.corp crmd[4721]:   notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph
Jun 09 17:50:33 node2.company.corp crmd[4721]:  warning: No reason to expect node 2 to be down
Jun 09 17:50:33 node2.company.corp pengine[4720]:   notice: On loss of CCM Quorum: Ignore
Jun 09 17:50:33 node2.company.corp pengine[4720]:   notice: Start   FTP_SERVICE        (node2.company.corp)
Jun 09 17:50:33 node2.company.corp pengine[4720]:   notice: Calculated transition 5464, saving inputs in /var/lib/pacemaker/pengine/pe-input-1571.bz2
Jun 09 17:50:34 node2.company.corp pengine[4720]:   notice: On loss of CCM Quorum: Ignore
Jun 09 17:50:34 node2.company.corp pengine[4720]:   notice: Start   FTP_SERVICE        (node2.company.corp)
Jun 09 17:50:34 node2.company.corp pengine[4720]:   notice: Calculated transition 5465, saving inputs in /var/lib/pacemaker/pengine/pe-input-1571.bz2
Jun 09 17:50:34 node2.company.corp crmd[4721]:   notice: Initiating monitor operation FTP_SERVICE_monitor_0 locally on node2.company.corp | action 5
Jun 09 17:50:34 node2.company.corp crmd[4721]:   notice: Result of probe operation for FTP_SERVICE on node2.company.corp: 196 (OCF_PENDING) | call=260 key=FTP_SERVICE_monitor_0 confirmed=true cib-update=5999
Jun 09 17:50:34 node2.company.corp crmd[4721]:  warning: Action 5 (FTP_SERVICE_monitor_0) on node2.company.corp failed (target: 7 vs. rc: 196): Error
Jun 09 17:50:34 node2.company.corp crmd[4721]:   notice: Transition aborted by operation FTP_SERVICE_monitor_0 'create' on node2.company.corp: Event failed | magic=0:196;5:5465:7:bf959758-83cf-4e4b-bd33-a360ffa08557 cib=0.451.1 source=match_graph_event:310 complete=false
Jun 09 17:50:34 node2.company.corp crmd[4721]:  warning: Action 5 (FTP_SERVICE_monitor_0) on node2.company.corp failed (target: 7 vs. rc: 196): Error
Jun 09 17:50:34 node2.company.corp crmd[4721]:   notice: Transition 5465 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=2, Source=/var/lib/pacemaker/pengine/pe-input-1571.bz2): Complete
Jun 09 17:50:34 node2.company.corp pengine[4720]:   notice: On loss of CCM Quorum: Ignore
Jun 09 17:50:34 node2.company.corp pengine[4720]:  warning: Processing failed op monitor for FTP_SERVICE on node2.company.corp: OCF_PENDING (196)
Jun 09 17:50:34 node2.company.corp pengine[4720]:  warning: Processing failed op monitor for FTP_SERVICE on node2.company.corp: OCF_PENDING (196)
Jun 09 17:50:34 node2.company.corp pengine[4720]:   notice: Recover FTP_SERVICE        (Started node2.company.corp)
Jun 09 17:50:34 node2.company.corp pengine[4720]:   notice: Calculated transition 5466, saving inputs in /var/lib/pacemaker/pengine/pe-input-1572.bz2
Jun 09 17:50:34 node2.company.corp crmd[4721]:   notice: Initiating stop operation FTP_SERVICE_stop_0 locally on node2.company.corp | action 4
Jun 09 17:50:34 node2.company.corp systemd[1]: Reloading.
Jun 09 17:50:54 node2.company.corp lrmd[4718]:   notice: Giving up on FTP_SERVICE stop (rc=196): timeout (elapsed=19988ms, remaining=12ms)
Jun 09 17:50:54 node2.company.corp crmd[4721]:    error: Result of stop operation for FTP_SERVICE on node2.company.corp: Timed Out | call=261 key=FTP_SERVICE_stop_0 timeout=20000ms
Jun 09 17:50:54 node2.company.corp crmd[4721]:  warning: Action 4 (FTP_SERVICE_stop_0) on node2.company.corp failed (target: 0 vs. rc: 198): Error
Jun 09 17:50:54 node2.company.corp crmd[4721]:   notice: Transition aborted by operation FTP_SERVICE_stop_0 'modify' on node2.company.corp: Event failed | magic=2:198;4:5466:0:bf959758-83cf-4e4b-bd33-a360ffa08557 cib=0.451.2 source=match_graph_event:310 complete=false
Jun 09 17:50:54 node2.company.corp crmd[4721]:  warning: Action 4 (FTP_SERVICE_stop_0) on node2.company.corp failed (target: 0 vs. rc: 198): Error
Jun 09 17:50:54 node2.company.corp crmd[4721]:   notice: Transition 5466 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=3, Source=/var/lib/pacemaker/pengine/pe-input-1572.bz2): Complete
Jun 09 17:50:54 node2.company.corp pengine[4720]:   notice: On loss of CCM Quorum: Ignore
Jun 09 17:50:54 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:50:54 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:50:54 node2.company.corp pengine[4720]:   notice: Calculated transition 5467, saving inputs in /var/lib/pacemaker/pengine/pe-input-1573.bz2
Jun 09 17:50:54 node2.company.corp pengine[4720]:   notice: On loss of CCM Quorum: Ignore
Jun 09 17:50:54 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:50:54 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:50:54 node2.company.corp pengine[4720]:  warning: Forcing FTP_SERVICE away from node2.company.corp after 1000000 failures (max=1000000)
Jun 09 17:50:54 node2.company.corp pengine[4720]:   notice: Calculated transition 5468, saving inputs in /var/lib/pacemaker/pengine/pe-input-1574.bz2
Jun 09 17:50:54 node2.company.corp crmd[4721]:   notice: Transition 5468 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-1574.bz2): Complete
Jun 09 17:50:54 node2.company.corp crmd[4721]:   notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd
Jun 09 17:51:03 node2.company.corp crmd[4721]:  warning: No reason to expect node 2 to be down
Jun 09 17:51:03 node2.company.corp crmd[4721]:   notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph
Jun 09 17:51:03 node2.company.corp pengine[4720]:   notice: On loss of CCM Quorum: Ignore
Jun 09 17:51:03 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:51:03 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:51:03 node2.company.corp pengine[4720]:   notice: Calculated transition 5469, saving inputs in /var/lib/pacemaker/pengine/pe-input-1575.bz2
Jun 09 17:51:03 node2.company.corp crmd[4721]:   notice: Transition 5469 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-1575.bz2): Complete
Jun 09 17:51:03 node2.company.corp crmd[4721]:   notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd
Jun 09 17:51:38 node2.company.corp crmd[4721]:   notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph
Jun 09 17:51:38 node2.company.corp pengine[4720]:   notice: On loss of CCM Quorum: Ignore
Jun 09 17:51:38 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:51:38 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:51:38 node2.company.corp pengine[4720]:   notice: Calculated transition 5470, saving inputs in /var/lib/pacemaker/pengine/pe-input-1576.bz2
Jun 09 17:51:39 node2.company.corp pengine[4720]:   notice: On loss of CCM Quorum: Ignore
Jun 09 17:51:39 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:51:39 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:51:39 node2.company.corp pengine[4720]:   notice: Calculated transition 5471, saving inputs in /var/lib/pacemaker/pengine/pe-input-1577.bz2
Jun 09 17:51:39 node2.company.corp crmd[4721]:   notice: Transition 5471 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-1577.bz2): Complete
Jun 09 17:51:39 node2.company.corp crmd[4721]:   notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd
Jun 09 17:52:00 node2.company.corp crmd[4721]:  warning: No reason to expect node 2 to be down
Jun 09 17:52:00 node2.company.corp crmd[4721]:   notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph
Jun 09 17:52:01 node2.company.corp pengine[4720]:   notice: On loss of CCM Quorum: Ignore
Jun 09 17:52:01 node2.company.corp pengine[4720]:   notice: Start   FTP_SERVICE        (node2.company.corp)
Jun 09 17:52:01 node2.company.corp pengine[4720]:   notice: Calculated transition 5472, saving inputs in /var/lib/pacemaker/pengine/pe-input-1578.bz2
Jun 09 17:52:01 node2.company.corp crmd[4721]:   notice: Initiating monitor operation FTP_SERVICE_monitor_0 locally on node2.company.corp | action 5
Jun 09 17:52:01 node2.company.corp crmd[4721]:   notice: Result of probe operation for FTP_SERVICE on node2.company.corp: 196 (OCF_PENDING) | call=266 key=FTP_SERVICE_monitor_0 confirmed=true cib-update=6016
Jun 09 17:52:01 node2.company.corp crmd[4721]:  warning: Action 5 (FTP_SERVICE_monitor_0) on node2.company.corp failed (target: 7 vs. rc: 196): Error
Jun 09 17:52:01 node2.company.corp crmd[4721]:   notice: Transition aborted by operation FTP_SERVICE_monitor_0 'create' on node2.company.corp: Event failed | magic=0:196;5:5472:7:bf959758-83cf-4e4b-bd33-a360ffa08557 cib=0.454.1 source=match_graph_event:310 complete=false
Jun 09 17:52:01 node2.company.corp crmd[4721]:  warning: Action 5 (FTP_SERVICE_monitor_0) on node2.company.corp failed (target: 7 vs. rc: 196): Error
Jun 09 17:52:01 node2.company.corp crmd[4721]:   notice: Transition 5472 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=2, Source=/var/lib/pacemaker/pengine/pe-input-1578.bz2): Complete
Jun 09 17:52:01 node2.company.corp pengine[4720]:   notice: On loss of CCM Quorum: Ignore
Jun 09 17:52:01 node2.company.corp pengine[4720]:  warning: Processing failed op monitor for FTP_SERVICE on node2.company.corp: OCF_PENDING (196)
Jun 09 17:52:01 node2.company.corp pengine[4720]:  warning: Processing failed op monitor for FTP_SERVICE on node2.company.corp: OCF_PENDING (196)
Jun 09 17:52:01 node2.company.corp pengine[4720]:   notice: Recover FTP_SERVICE        (Started node2.company.corp)
Jun 09 17:52:01 node2.company.corp pengine[4720]:   notice: Calculated transition 5473, saving inputs in /var/lib/pacemaker/pengine/pe-input-1579.bz2
Jun 09 17:52:01 node2.company.corp crmd[4721]:   notice: Initiating stop operation FTP_SERVICE_stop_0 locally on node2.company.corp | action 4
Jun 09 17:52:01 node2.company.corp systemd[1]: Reloading.
Jun 09 17:52:21 node2.company.corp lrmd[4718]:   notice: Giving up on FTP_SERVICE stop (rc=196): timeout (elapsed=19988ms, remaining=12ms)
Jun 09 17:52:21 node2.company.corp crmd[4721]:    error: Result of stop operation for FTP_SERVICE on node2.company.corp: Timed Out | call=267 key=FTP_SERVICE_stop_0 timeout=20000ms
Jun 09 17:52:21 node2.company.corp crmd[4721]:  warning: Action 4 (FTP_SERVICE_stop_0) on node2.company.corp failed (target: 0 vs. rc: 198): Error
Jun 09 17:52:21 node2.company.corp crmd[4721]:   notice: Transition aborted by operation FTP_SERVICE_stop_0 'modify' on node2.company.corp: Event failed | magic=2:198;4:5473:0:bf959758-83cf-4e4b-bd33-a360ffa08557 cib=0.454.2 source=match_graph_event:310 complete=false
Jun 09 17:52:21 node2.company.corp crmd[4721]:  warning: Action 4 (FTP_SERVICE_stop_0) on node2.company.corp failed (target: 0 vs. rc: 198): Error
Jun 09 17:52:21 node2.company.corp crmd[4721]:   notice: Transition 5473 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=3, Source=/var/lib/pacemaker/pengine/pe-input-1579.bz2): Complete
Jun 09 17:52:21 node2.company.corp pengine[4720]:   notice: On loss of CCM Quorum: Ignore
Jun 09 17:52:21 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:52:21 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:52:21 node2.company.corp pengine[4720]:   notice: Calculated transition 5474, saving inputs in /var/lib/pacemaker/pengine/pe-input-1580.bz2
Jun 09 17:52:21 node2.company.corp pengine[4720]:   notice: On loss of CCM Quorum: Ignore
Jun 09 17:52:21 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:52:21 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:52:21 node2.company.corp pengine[4720]:  warning: Forcing FTP_SERVICE away from node2.company.corp after 1000000 failures (max=1000000)
Jun 09 17:52:21 node2.company.corp pengine[4720]:   notice: Calculated transition 5475, saving inputs in /var/lib/pacemaker/pengine/pe-input-1581.bz2
Jun 09 17:52:21 node2.company.corp crmd[4721]:   notice: Transition 5475 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-1581.bz2): Complete
Jun 09 17:52:21 node2.company.corp crmd[4721]:   notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd
Jun 09 17:52:58 node2.company.corp crmd[4721]:   notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph
Jun 09 17:52:58 node2.company.corp pengine[4720]:   notice: On loss of CCM Quorum: Ignore
Jun 09 17:52:58 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:52:58 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:52:58 node2.company.corp pengine[4720]:  warning: Forcing FTP_SERVICE away from node2.company.corp after 1000000 failures (max=1000000)
Jun 09 17:52:58 node2.company.corp pengine[4720]:   notice: Calculated transition 5476, saving inputs in /var/lib/pacemaker/pengine/pe-input-1582.bz2
Jun 09 17:52:58 node2.company.corp crmd[4721]:   notice: Transition 5476 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-1582.bz2): Complete
Jun 09 17:52:58 node2.company.corp crmd[4721]:   notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd
Jun 09 17:53:09 node2.company.corp crmd[4721]:   notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph
Jun 09 17:53:09 node2.company.corp pengine[4720]:   notice: On loss of CCM Quorum: Ignore
Jun 09 17:53:09 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:53:09 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:53:09 node2.company.corp pengine[4720]:  warning: Forcing FTP_SERVICE away from node2.company.corp after 1000000 failures (max=1000000)
Jun 09 17:53:09 node2.company.corp pengine[4720]:   notice: Calculated transition 5477, saving inputs in /var/lib/pacemaker/pengine/pe-input-1583.bz2
Jun 09 17:53:09 node2.company.corp crmd[4721]:   notice: Transition 5477 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-1583.bz2): Complete
Jun 09 17:53:09 node2.company.corp crmd[4721]:   notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd
Jun 09 17:53:28 node2.company.corp crmd[4721]:   notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph
Jun 09 17:53:28 node2.company.corp pengine[4720]:   notice: On loss of CCM Quorum: Ignore
Jun 09 17:53:28 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:53:28 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:53:28 node2.company.corp pengine[4720]:  warning: Forcing FTP_SERVICE away from node2.company.corp after 1000000 failures (max=1000000)
Jun 09 17:53:28 node2.company.corp pengine[4720]:   notice: Calculated transition 5478, saving inputs in /var/lib/pacemaker/pengine/pe-input-1584.bz2
Jun 09 17:53:28 node2.company.corp pengine[4720]:   notice: On loss of CCM Quorum: Ignore
Jun 09 17:53:28 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:53:28 node2.company.corp pengine[4720]:  warning: Processing failed op stop for FTP_SERVICE on node2.company.corp: OCF_TIMEOUT (198)
Jun 09 17:53:28 node2.company.corp pengine[4720]:  warning: Forcing FTP_SERVICE away from node2.company.corp after 1000000 failures (max=1000000)
Jun 09 17:53:28 node2.company.corp pengine[4720]:   notice: Calculated transition 5479, saving inputs in /var/lib/pacemaker/pengine/pe-input-1585.bz2
Jun 09 17:53:28 node2.company.corp crmd[4721]:   notice: Transition 5479 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-1585.bz2): Complete
Jun 09 17:53:28 node2.company.corp crmd[4721]:   notice: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd
Jun 09 17:53:31 node2.company.corp systemd[1]: vsftpd.service stop-final-sigterm timed out. Killing.
Jun 09 17:53:31 node2.company.corp systemd[1]: Stopped Vsftpd ftp daemon.
-- Subject: Unit vsftpd.service has finished shutting down
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit vsftpd.service has finished shutting down.
Jun 09 17:53:31 node2.company.corp systemd[1]: Unit vsftpd.service entered failed state.
Jun 09 17:53:31 node2.company.corp systemd[1]: vsftpd.service failed.

Pacemaker configuration

Cluster Name: FTP_CLUSTER
Corosync Nodes:
 node1 node2
Pacemaker Nodes:
 node1 node2

 

Resources:
 Resource: INTERNAL_FTP_VIRTUAL_IP (class=ocf provider=heartbeat type=IPaddr2)
  Attributes: ip=111.55.3.25 cidr_netmask=24
  Operations: start interval=0s timeout=20s (INTERNAL_FTP_VIRTUAL_IP-start-interval-0s)
              stop interval=0s timeout=20s (INTERNAL_FTP_VIRTUAL_IP-stop-interval-0s)
              monitor interval=10s (INTERNAL_FTP_VIRTUAL_IP-monitor-interval-10s)
 Resource: INTERNAL_FTP (class=systemd type=vsftpd)
  Operations: monitor interval=10s (INTERNAL_FTP-monitor-interval-10s)
 Resource: TOMCAT (class=ocf provider=heartbeat type=tomcat)
  Attributes: java_home=/usr/java/jdk1.8.0_181 catalina_home=/usr/local/tomcat
  Operations: start interval=0s timeout=60s (TOMCAT-start-interval-0s)
              stop interval=0s timeout=120s (TOMCAT-stop-interval-0s)
              monitor interval=15s (TOMCAT-monitor-interval-15s)
 Resource: INTERNAL_FTP_VIRTUAL_IP_APACHE (class=ocf provider=heartbeat type=IPaddr2)
  Attributes: ip=111.99.3.25 cidr_netmask=32 nic=bond0.1104
  Operations: start interval=0s timeout=20s (INTERNAL_FTP_VIRTUAL_IP_APACHE-start-interval-0s)
              stop interval=0s timeout=20s (INTERNAL_FTP_VIRTUAL_IP_APACHE-stop-interval-0s)
              monitor interval=30s (INTERNAL_FTP_VIRTUAL_IP_APACHE-monitor-interval-30s)

 

Stonith Devices:
Fencing Levels:

 

Location Constraints:
  Resource: INTERNAL_FTP
    Enabled on: node2 (score:INFINITY) (role: Started) (id:cli-prefer-INTERNAL_FTP)
  Resource: TOMCAT
    Enabled on: node2 (score:INFINITY) (role: Started) (id:cli-prefer-TOMCAT)
  Resource: INTERNAL_FTP_VIRTUAL_IP
    Enabled on: node1 (score:INFINITY) (id:location-INTERNAL_FTP_VIRTUAL_IP-node1-INFINITY)
    Enabled on: node2 (score:INFINITY) (role: Started) (id:cli-prefer-INTERNAL_FTP_VIRTUAL_IP)
  Resource: INTERNAL_FTP_VIRTUAL_IP_APACHE
    Enabled on: node2 (score:INFINITY) (role: Started) (id:cli-prefer-INTERNAL_FTP_VIRTUAL_IP_APACHE)
Ordering Constraints:
  start INTERNAL_FTP_VIRTUAL_IP then start INTERNAL_FTP (kind:Mandatory) (id:order-INTERNAL_FTP_VIRTUAL_IP-INTERNAL_FTP-mandatory)
Colocation Constraints:
  INTERNAL_FTP with INTERNAL_FTP_VIRTUAL_IP (score:INFINITY) (id:colocation-INTERNAL_FTP-INTERNAL_FTP_VIRTUAL_IP-INFINITY)
Ticket Constraints:

 

Alerts:
 No alerts defined

 

Resources Defaults:
 No defaults set
Operations Defaults:
 No defaults set

 

Cluster Properties:
 cluster-infrastructure: corosync
 cluster-name: FTP_CLUSTER
 dc-version: 1.1.15-11.el7-e174ec8
 have-watchdog: false
 last-lrm-refresh: 1624555372
 no-quorum-policy: ignore
 stonith-enabled: false

 

Quorum:
  Options:

Versions: Oracle Linux Server release 7.2 vsftpd: version 3.0.2 pacemaker-1.1.15-11.el7.x86_64

cape

Asked: 2020-04-27 05:06:19 +0800 CST

NFS server daemon fails to start at boot

I´m trying to export an external disk, so, I configured my nfs-server service to wait for disk1 to mount, however it fails.

This is the situation after boot:

$ systemctl status nfs-server.service
● nfs-server.service - NFS server and services
   Loaded: loaded (/etc/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sun 2020-04-26 14:46:28 CEST; 3min 7s ago
  Process: 307 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=1/FAILURE)
  Process: 312 ExecStopPost=/usr/sbin/exportfs -au (code=exited, status=0/SUCCESS)
  Process: 314 ExecStopPost=/usr/sbin/exportfs -f (code=exited, status=0/SUCCESS)

abr 26 14:46:28 raspberrypi systemd[1]: Starting NFS server and services...
abr 26 14:46:28 raspberrypi exportfs[307]: exportfs: Failed to stat /media/pi/disk1: No such file or directory
abr 26 14:46:28 raspberrypi systemd[1]: nfs-server.service: Control process exited, code=exited, status=1/FAILURE
abr 26 14:46:28 raspberrypi systemd[1]: nfs-server.service: Failed with result 'exit-code'.
abr 26 14:46:28 raspberrypi systemd[1]: Failed to start NFS server and services.

If I just restart the service it just works smoothly

$ sudo systemctl restart nfs-server.service
$ systemctl status nfs-server.service
● nfs-server.service - NFS server and services
   Loaded: loaded (/etc/systemd/system/nfs-server.service; enabled; vendor preset: enabled)
   Active: active (exited) since Sun 2020-04-26 14:59:51 CEST; 4s ago
  Process: 943 ExecStartPre=/usr/sbin/exportfs -r (code=exited, status=0/SUCCESS)
  Process: 944 ExecStart=/usr/sbin/rpc.nfsd $RPCNFSDARGS (code=exited, status=0/SUCCESS)
 Main PID: 944 (code=exited, status=0/SUCCESS)

abr 26 14:59:51 raspberrypi systemd[1]: Starting NFS server and services...
abr 26 14:59:51 raspberrypi systemd[1]: Started NFS server and services.

I configured the service to "Requires" and "After" disk1 mounts, but it didn´t work:

# /etc/systemd/system/nfs-server.service
[Unit]
Description=NFS server and services
DefaultDependencies=no
Requires=network.target proc-fs-nfsd.mount media-pi-disk1.mount
Requires=nfs-mountd.service
Wants=rpcbind.socket
Wants=nfs-idmapd.service

After=local-fs.target media-pi-disk1.mount
After=network.target proc-fs-nfsd.mount rpcbind.socket nfs-mountd.service
After=nfs-idmapd.service rpc-statd.service
Before=rpc-statd-notify.service

Extra info requested:


$ systemctl status media-pi-disk1.mount
● media-pi-disk1.mount - /media/pi/disk1
   Loaded: loaded
   Active: active (mounted) since Sun 2020-04-26 14:47:34 CEST; 3h 22min ago
    Where: /media/pi/disk1
     What: /dev/sda1



 $ egrep -v '^#|^$' /etc/fstab
proc            /proc           proc    defaults          0       0
/dev/mmcblk0p8  /boot           vfat    defaults          0       2
/dev/mmcblk0p9  /  


 $ lsblk
NAME        MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda           8:0    0 931,5G  0 disk
└─sda1        8:1    0 931,5G  0 part /media/pi/disk1
mmcblk0     179:0    0  29,7G  0 disk
├─mmcblk0p1 179:1    0   2,4G  0 part
├─mmcblk0p2 179:2    0     1K  0 part
├─mmcblk0p5 179:5    0    32M  0 part
├─mmcblk0p6 179:6    0   512M  0 part /media/pi/System
├─mmcblk0p7 179:7    0  12,1G  0 part /media/pi/Storage
├─mmcblk0p8 179:8    0   256M  0 part /boot
└─mmcblk0p9 179:9    0  14,5G  0 part /



$ mount
/dev/mmcblk0p9 on / type ext4 (rw,noatime)
devtmpfs on /dev type devtmpfs (rw,relatime,size=217076k,nr_inodes=54269,mode=755)
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime)
proc on /proc type proc (rw,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,mode=755)
tmpfs on /run/lock type tmpfs (rw,nosuid,nodev,noexec,relatime,size=5120k)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,mode=755)
cgroup2 on /sys/fs/cgroup/unified type cgroup2 (rw,nosuid,nodev,noexec,relatime,nsdelegate)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,xattr,name=systemd)
bpf on /sys/fs/bpf type bpf (rw,nosuid,nodev,noexec,relatime,mode=700)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,memory)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,cpu,cpuacct)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,devices)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,blkio)
cgroup on /sys/fs/cgroup/net_cls type cgroup (rw,nosuid,nodev,noexec,relatime,net_cls)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=28,pgrp=1,timeout=0,minproto=5,maxproto=5,direct)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
mqueue on /dev/mqueue type mqueue (rw,relatime)
sunrpc on /run/rpc_pipefs type rpc_pipefs (rw,relatime)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/mmcblk0p8 on /boot type vfat (rw,relatime,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,errors=remount-ro)
tmpfs on /run/user/1000 type tmpfs (rw,nosuid,nodev,relatime,size=44280k,mode=700,uid=1000,gid=1000)
gvfsd-fuse on /run/user/1000/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=1000,group_id=1000)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
/dev/sda1 on /media/pi/disk1 type ext4 (rw,nosuid,nodev,relatime,uhelper=udisks2)
/dev/mmcblk0p7 on /media/pi/Storage type ext4 (rw,nosuid,nodev,relatime,uhelper=udisks2)
/dev/mmcblk0p6 on /media/pi/System type vfat (rw,nosuid,nodev,relatime,uid=1000,gid=1000,fmask=0022,dmask=0022,codepage=437,iocharset=ascii,shortname=mixed,showexec,utf8,flush,errors=remount-ro,uhelper=udisks2)

Clustered vsftpd service stops working with no error log

NFS server daemon fails to start at boot

Can you pass user/pass for HTTP Basic Authentication in URL parameters?

Ping a Specific Port

Check if port is open or closed on a Linux server?

How to automate SSH login with password?

How do I tell Git for Windows where to find my private RSA key?

What's the default superuser username/password for postgres after a new install?

What port does SFTP use?

Command line to list users in a Windows Active Directory group?

What is a Pem file and how does it differ from other OpenSSL Generated Key File Formats?

How to determine if a bash variable is empty?