<html dir="ltr">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="text-align:left; direction:ltr;">
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
Hello users, I would like to ask for assistance on the below setup please, mainly on the monitor fence timeout:</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
<br>
</div>
<div style="white-space: normal;">#pcs --version</div>
<div style="white-space: normal;">0.9.167</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
<br>
</div>
<div style="white-space: normal;">#pacemakerd --version</div>
<div style="white-space: normal;">Pacemaker 1.1.20-5.el7_7.2</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
<br>
</div>
<div style="white-space: normal;">#corosync -v</div>
<div style="white-space: normal;">Corosync Cluster Engine, version '2.4.3'</div>
<div style="white-space: normal;">Copyright (c) 2006-2009 Red Hat, Inc.</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
<br>
</div>
<div style="white-space: normal;"># cat /etc/redhat-release </div>
<div style="white-space: normal;">CentOS Linux release 7.7.1908 (Core)</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
<br>
</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
I have setup a 2-node axigen mail server setup with 1 resource group (and 3 resources within) and 2 fence devices (1 for each node).</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
<br>
</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
the hosts file on the nodes is as follows:</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
<br>
</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
#KVM Management nodes</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
10.1.4.31 zc-infra-mgmt-node-1</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
10.1.4.20 zc-infra-mgmt-node-2</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
<br>
</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
#Service Network</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
10.1.4.22 zc-mail-1.domain.com zc-mail-1 </div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
10.1.4.23 zc-mail-2.domain.com zc-mail-2</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
<br>
</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
#High-Availability Network (cross-over link)</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
192.168.1.22 zc-mail-1-ha.domain.local zc-mail-1-ha</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
192.168.1.23 zc-mail-2-ha.domain.local zc-mail-2-ha</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
<br>
</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
the routable network is 10.1.4.0/24. A VIP is setup as part of the HA Cluster resources (IPaddr2) which is 10.1.4.24</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
<br>
</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
the resources are described as follows:</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
<br>
</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
# pcs resource show --full</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
<br>
</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
Group: zc-mail-res-group</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
<br>
</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
Resource: zc-mail-ha-Cfs (class=ocf provider=heartbeat type=Filesystem)</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
Attributes: device=10.1.3.11:6789,10.1.3.12:6789,10.1.3.13:6789:/ directory=/var/clusterfs/data/axigen fstype=ceph options=name=email,secretfile=/etc/ceph/ceph.key statusfile_prefix=ceph_fs_checks_</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
Operations: monitor interval=120s (zc-mail-ha-Cfs-monitor-interval-120s)</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
notify interval=0s timeout=120s (zc-mail-ha-Cfs-notify-interval-0s)</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
start interval=0s timeout=120s (zc-mail-ha-Cfs-start-interval-0s)</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
stop interval=0s timeout=120s (zc-mail-ha-Cfs-stop-interval-0s)</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
Resource: zc-mail-ha-vip (class=ocf provider=heartbeat type=IPaddr2)</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
Attributes: cidr_netmask=24 ip=10.1.4.24</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
Operations: monitor interval=120s (zc-mail-ha-vip-monitor-interval-120s)</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
start interval=0s timeout=120s (zc-mail-ha-vip-start-interval-0s)</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
stop interval=0s timeout=120s (zc-mail-ha-vip-stop-interval-0s)</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
Resource: zc-mail-ha-svc (class=lsb type=axigen)</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
Meta Attrs: is-managed=true target-role=Started</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
Operations: force-reload interval=0s timeout=60 (zc-mail-ha-svc-force-reload-interval-0s)</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
monitor interval=30s timeout=120s OCF_CHECK_LEVEL=20 (zc-mail-ha-svc-monitor-interval-30s)</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
restart interval=0s timeout=120s (zc-mail-ha-svc-restart-interval-0s)</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
start interval=0s timeout=120s (zc-mail-ha-svc-start-interval-0s)</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
stop interval=0s timeout=120s (zc-mail-ha-svc-stop-interval-0s)</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
<br>
</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
# pcs stonith show --full</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
<br>
</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
Resource: fence_zc-mail-1_virsh (class=stonith type=fence_virsh)</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
Attributes: delay=0 identity_file=/home/lcami/.ssh/id_rsa ipaddr=zc-infra-mgmt-node-1 login=lcami login_timeout=20 pcmk_host_check=static-list pcmk_host_list=zc-mail-1-ha pcmk_host_map=zc-mail-1-ha:zc-infra-mgmt-node-1 port=Axigen-Mail-1 sudo=1</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
Operations: monitor interval=60s (fence_zc-mail-1_virsh-monitor-interval-60s)</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
<br>
</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
Resource: fence_zc-mail-2_virsh (class=stonith type=fence_virsh)</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
Attributes: identity_file=/home/lcami/.ssh/id_rsa ipaddr=zc-infra-mgmt-node-2 login=lcami login_timeout=20 pcmk_host_check=static-list pcmk_host_list=zc-mail-2-ha pcmk_host_map=zc-mail-2-ha:zc-infra-mgmt-node-2 port=Axigen-Mail-2 sudo=1</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
Operations: monitor interval=60s (fence_zc-mail-2_virsh-monitor-interval-60s)</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
<br>
</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
Every couple of days I used to receive the following error:</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
<br>
</div>
<div><font color="#ff0000">Feb 16 00:00:24 [2051] zc-mail-2.zylacloud.com stonith-ng: notice: operation_finished: fence_virsh_monitor_1:21995:stderr [ 2020-02-16 00:00:23,996 ERROR: Connection timed out ]</font></div>
<div><font color="#ff0000">Feb 16 00:00:24 [2051] zc-mail-2.zylacloud.com stonith-ng: notice: operation_finished: fence_virsh_monitor_1:21995:stderr [ ]</font></div>
<div><font color="#ff0000">Feb 16 00:00:24 [2051] zc-mail-2.zylacloud.com stonith-ng: notice: operation_finished: fence_virsh_monitor_1:21995:stderr [ ]</font></div>
<div><font color="#ff0000">Feb 16 00:00:24 [2051] zc-mail-2.zylacloud.com stonith-ng: warning: log_action: fence_virsh[21995] stderr: [ 2020-02-16 00:00:23,996 ERROR: Connection timed out ]</font></div>
<div><font color="#ff0000">Feb 16 00:00:24 [2051] zc-mail-2.zylacloud.com stonith-ng: warning: log_action: fence_virsh[21995] stderr: [ ]</font></div>
<div><font color="#ff0000">Feb 16 00:00:24 [2051] zc-mail-2.zylacloud.com stonith-ng: warning: log_action: fence_virsh[21995] stderr: [ ]</font></div>
<div><font color="#ff0000">Feb 16 00:00:24 [2051] zc-mail-2.zylacloud.com stonith-ng: notice: log_operation: Operation 'monitor' [21995] for device 'fence_zc-mail-1_virsh' returned: -62 (Timer expired)</font></div>
<div><font color="#ff0000">Feb 16 00:00:24 [2052] zc-mail-2.zylacloud.com lrmd: info: log_finished: finished - rsc:fence_zc-mail-1_virsh action:start call_id:85 exit-code:1 exec-time:5449ms queue-time:0ms</font></div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
<br>
</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
which I concluded was a problem with the login timeout (which was 5 seconds)</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
<br>
</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
I have therefore incresed this timeut to 20 seconds but the timeout persisted:</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
<br>
</div>
<div style="caret-color: rgb(0, 0, 0); white-space: normal;">
<div><font color="#ff0000">Feb 23 00:00:21 [24633] zc-mail-2.zylacloud.com stonith-ng: notice: operation_finished: fence_virsh_monitor_1:20006:stderr [ 2020-02-23 00:00:21,102 ERROR: Connection timed out ]</font></div>
<div><font color="#ff0000">Feb 23 00:00:21 [24633] zc-mail-2.zylacloud.com stonith-ng: notice: operation_finished: fence_virsh_monitor_1:20006:stderr [ ]</font></div>
<div><font color="#ff0000">Feb 23 00:00:21 [24633] zc-mail-2.zylacloud.com stonith-ng: notice: operation_finished: fence_virsh_monitor_1:20006:stderr [ ]</font></div>
<div><font color="#ff0000">Feb 23 00:00:21 [24633] zc-mail-2.zylacloud.com stonith-ng: warning: log_action: fence_virsh[20006] stderr: [ 2020-02-23 00:00:21,102 ERROR: Connection timed out ]</font></div>
<div><font color="#ff0000">Feb 23 00:00:21 [24633] zc-mail-2.zylacloud.com stonith-ng: warning: log_action: fence_virsh[20006] stderr: [ ]</font></div>
<div><font color="#ff0000">Feb 23 00:00:21 [24633] zc-mail-2.zylacloud.com stonith-ng: warning: log_action: fence_virsh[20006] stderr: [ ]</font></div>
<div><font color="#ff0000">Feb 23 00:00:21 [24633] zc-mail-2.zylacloud.com stonith-ng: notice: log_operation: Operation 'monitor' [20006] for device 'fence_zc-mail-1_virsh' returned: -62 (Timer expired)</font></div>
<div><font color="#ff0000">Feb 23 00:00:21 [24637] zc-mail-2.zylacloud.com crmd: error: process_lrm_event: Result of monitor operation for fence_zc-mail-1_virsh on zc-mail-2-ha: Timed Out | call=30 key=fence_zc-mail-1_virsh_monitor_60000 timeout=20000ms</font></div>
<div></div>
</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
<br>
</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
There is also a constraint as shown below so that the fencing "agent" runs on the opposite node to be restarted:</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
<br>
</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
# pcs constraint show --full</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
<br>
</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
Location Constraints:</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
<br>
</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
Resource: fence_zc-mail-1_virsh</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
Enabled on: zc-mail-2-ha (score:INFINITY) (role: Started) (id:cli-prefer-fence_zc-mail-1_virsh)</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
<br>
</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
Resource: fence_zc-mail-2_virsh</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
Enabled on: zc-mail-1-ha (score:INFINITY) (role: Started) (id:cli-prefer-fence_zc-mail-2_virsh)</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
<br>
</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
Ordering Constraints:</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
<br>
</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
Colocation Constraints:</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
<br>
</div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
Ticket Constraints:</div>
<div>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); white-space: normal;">
</div>
</div>
</body>
</html>