<div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr"><div dir="ltr">Here is configuration (Note that we have put node01 in standby since then, in order to keep the services stable for the moment):</div><div dir="ltr">===</div><div dir="ltr"><div dir="ltr">node 1: <a href="http://node01.example.com">node01.example.com</a> \</div><div dir="ltr"> attributes standby=on</div><div dir="ltr">node 2: <a href="http://node02.example.com">node02.example.com</a> \</div><div dir="ltr"> attributes standby=off</div><div dir="ltr">primitive app_ourApp lsb:ourUser \</div><div dir="ltr"> meta target-role=Started \</div><div dir="ltr"> op stop interval=0s timeout=90s</div><div dir="ltr">primitive daemon_httpd apache \</div><div dir="ltr"> params configfile="/etc/httpd/conf/httpd.conf" port=80 \</div><div dir="ltr"> op start interval=0s timeout=60s \</div><div dir="ltr"> op monitor interval=5s timeout=20s \</div><div dir="ltr"> op stop interval=0s timeout=60s \</div><div dir="ltr"> meta target-role=Started</div><div dir="ltr">primitive drbd_ourApp ocf:linbit:drbd \</div><div dir="ltr"> params drbd_resource=ourApp \</div><div dir="ltr"> op monitor interval=15s role=Master \</div><div dir="ltr"> op monitor interval=30s role=Slave</div><div dir="ltr">primitive fs_ourApp Filesystem \</div><div dir="ltr"> params device="/dev/drbd0" directory="/data" fstype=xfs \</div><div dir="ltr"> op stop interval=0s timeout=90s</div><div dir="ltr">primitive ip_ourApp IPaddr2 \</div><div dir="ltr"> params ip=10.6.21.100 nic=bond0 cidr_netmask=24 iflabel=1</div><div dir="ltr">primitive pingd ocf:pacemaker:ping \</div><div dir="ltr"> params host_list=10.6.21.1 multiplier=100 \</div><div dir="ltr"> op monitor interval=30s timeout=20s</div><div dir="ltr">group httpd daemon_httpd \</div><div dir="ltr"> meta target-role=Started</div><div dir="ltr">group ourApp fs_ourApp ip_ourApp app_ourApp \</div><div dir="ltr"> meta target-role=Started</div><div dir="ltr">ms ms_drbd_ourApp drbd_ourApp \</div><div dir="ltr"> meta master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true target-role=Started</div><div dir="ltr">clone pingdclone pingd \</div><div dir="ltr"> meta globally-unique=false target-role=Started</div><div dir="ltr">location <a href="http://cli-ban-httpd-on-node01.example.com">cli-ban-httpd-on-node01.example.com</a> httpd role=Started -inf: <a href="http://node01.example.com">node01.example.com</a></div><div dir="ltr">location <a href="http://cli-ban-ms_drbd_ourApp-on-node01.example.com">cli-ban-ms_drbd_ourApp-on-node01.example.com</a> ms_drbd_ourApp role=Master -inf: <a href="http://node01.example.com">node01.example.com</a></div><div dir="ltr">location <a href="http://cli-ban-ourApp-on-node01.example.com">cli-ban-ourApp-on-node01.example.com</a> ourApp role=Started -inf: <a href="http://node01.example.com">node01.example.com</a></div><div dir="ltr">colocation httpd-with-ip inf: daemon_httpd ip_ourApp</div><div dir="ltr">order httpd_after_ourApp inf: ourApp:start daemon_httpd</div><div dir="ltr">order ourApp_after_drbd inf: ms_drbd_ourApp:promote ourApp:start</div><div dir="ltr">colocation ourApp_on_drbd inf: ourApp ms_drbd_ourApp:Master</div><div dir="ltr">property cib-bootstrap-options: \</div><div dir="ltr"> have-watchdog=false \</div><div dir="ltr"> dc-version=1.1.18-11.el7_5.3-2b07d5c5a9 \</div><div dir="ltr"> cluster-infrastructure=corosync \</div><div dir="ltr"> stonith-enabled=false \</div><div dir="ltr"> no-quorum-policy=stop \</div><div dir="ltr"> cluster-name=ourAppapp \</div><div dir="ltr"> last-lrm-refresh=1611024747</div><div><div>rsc_defaults rsc-options: \</div><div> resource-stickiness=100</div></div><div>===</div><div><br></div><div>Here are constraints (if any) on all resources:</div><div>===</div><div><div>* httpd</div><div> : Node <a href="http://node01.example.com">node01.example.com</a> (score=-INFINITY, id=<a href="http://cli-ban-httpd-on-node01.example.com">cli-ban-httpd-on-node01.example.com</a>)</div><div>* ourApp</div><div> : Node <a href="http://node01.example.com">node01.example.com</a> (score=-INFINITY, id=<a href="http://cli-ban-ourApp-on-node01.example.com">cli-ban-ourApp-on-node01.example.com</a>)</div><div> ms_drbd_ourApp (score=INFINITY, with role=Master, id=ourApp_on_drbd)</div><div> : Node <a href="http://node01.example.com">node01.example.com</a> (score=-INFINITY, id=<a href="http://cli-ban-ms_drbd_ourApp-on-node01.example.com">cli-ban-ms_drbd_ourApp-on-node01.example.com</a>)</div><div>* ourApp</div><div> : Node <a href="http://node01.example.com">node01.example.com</a> (score=-INFINITY, id=<a href="http://cli-ban-ourApp-on-node01.example.com">cli-ban-ourApp-on-node01.example.com</a>)</div><div> ms_drbd_ourApp (score=INFINITY, with role=Master, id=ourApp_on_drbd)</div><div> : Node <a href="http://node01.example.com">node01.example.com</a> (score=-INFINITY, id=<a href="http://cli-ban-ms_drbd_ourApp-on-node01.example.com">cli-ban-ms_drbd_ourApp-on-node01.example.com</a>)</div><div>* ourApp</div><div> : Node <a href="http://node01.example.com">node01.example.com</a> (score=-INFINITY, id=<a href="http://cli-ban-ourApp-on-node01.example.com">cli-ban-ourApp-on-node01.example.com</a>)</div><div> ms_drbd_ourApp (score=INFINITY, with role=Master, id=ourApp_on_drbd)</div><div> : Node <a href="http://node01.example.com">node01.example.com</a> (score=-INFINITY, id=<a href="http://cli-ban-ms_drbd_ourApp-on-node01.example.com">cli-ban-ms_drbd_ourApp-on-node01.example.com</a>)</div><div> ourApp (score=INFINITY, needs role=Master, id=ourApp_on_drbd)</div><div> : Node <a href="http://node01.example.com">node01.example.com</a> (score=-INFINITY, id=<a href="http://cli-ban-ourApp-on-node01.example.com">cli-ban-ourApp-on-node01.example.com</a>)</div><div>* ms_drbd_ourApp</div><div> : Node <a href="http://node01.example.com">node01.example.com</a> (score=-INFINITY, id=<a href="http://cli-ban-ms_drbd_ourApp-on-node01.example.com">cli-ban-ms_drbd_ourApp-on-node01.example.com</a>)</div><div>* pingdclone</div><div>===<br></div></div><div><br></div><div>Here are all of the LogAction messages in the 30 minute period centered on 16:55 (i.e. this particular example):</div><div>===</div><div><div>Jan 18 16:52:19 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: notice: LogAction: * Stop daemon_httpd ( <a href="http://node02.example.com">node02.example.com</a> ) due to node availability</div><div>Jan 18 16:52:19 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: notice: LogAction: * Stop fs_ourApp ( <a href="http://node02.example.com">node02.example.com</a> ) due to node availability</div><div>Jan 18 16:52:19 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: notice: LogAction: * Stop ip_ourApp ( <a href="http://node02.example.com">node02.example.com</a> ) due to node availability</div><div>Jan 18 16:52:19 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: notice: LogAction: * Stop app_ourApp ( <a href="http://node02.example.com">node02.example.com</a> ) due to node availability</div><div>Jan 18 16:52:19 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: info: LogActions: Leave drbd_ourApp:0 (Slave <a href="http://node01.example.com">node01.example.com</a>)</div><div>Jan 18 16:52:19 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: notice: LogAction: * Demote drbd_ourApp:1 ( Master -> Slave <a href="http://node02.example.com">node02.example.com</a> ) </div><div>Jan 18 16:52:19 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: info: LogActions: Leave pingd:0 (Started <a href="http://node01.example.com">node01.example.com</a>)</div><div>Jan 18 16:52:19 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: info: LogActions: Leave pingd:1 (Started <a href="http://node02.example.com">node02.example.com</a>)</div><div>Jan 18 16:52:25 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: notice: LogAction: * Start daemon_httpd ( <a href="http://node02.example.com">node02.example.com</a> ) </div><div>Jan 18 16:52:25 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: info: LogActions: Leave fs_ourApp (Started <a href="http://node02.example.com">node02.example.com</a>)</div><div>Jan 18 16:52:25 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: info: LogActions: Leave ip_ourApp (Started <a href="http://node02.example.com">node02.example.com</a>)</div><div>Jan 18 16:52:25 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: info: LogActions: Leave app_ourApp (Started <a href="http://node02.example.com">node02.example.com</a>)</div><div>Jan 18 16:52:25 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: info: LogActions: Leave drbd_ourApp:0 (Slave <a href="http://node01.example.com">node01.example.com</a>)</div><div>Jan 18 16:52:25 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: info: LogActions: Leave drbd_ourApp:1 (Master <a href="http://node02.example.com">node02.example.com</a>)</div><div>Jan 18 16:52:25 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: info: LogActions: Leave pingd:0 (Started <a href="http://node01.example.com">node01.example.com</a>)</div><div>Jan 18 16:52:25 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: info: LogActions: Leave pingd:1 (Started <a href="http://node02.example.com">node02.example.com</a>)</div><div>Jan 18 16:52:44 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: info: LogActions: Leave daemon_httpd (Started <a href="http://node02.example.com">node02.example.com</a>)</div><div>Jan 18 16:52:44 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: info: LogActions: Leave fs_ourApp (Started <a href="http://node02.example.com">node02.example.com</a>)</div><div>Jan 18 16:52:44 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: info: LogActions: Leave ip_ourApp (Started <a href="http://node02.example.com">node02.example.com</a>)</div><div>Jan 18 16:52:44 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: info: LogActions: Leave app_ourApp (Started <a href="http://node02.example.com">node02.example.com</a>)</div><div>Jan 18 16:52:44 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: info: LogActions: Leave drbd_ourApp:0 (Slave <a href="http://node01.example.com">node01.example.com</a>)</div><div>Jan 18 16:52:44 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: info: LogActions: Leave drbd_ourApp:1 (Master <a href="http://node02.example.com">node02.example.com</a>)</div><div>Jan 18 16:52:44 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: info: LogActions: Leave pingd:0 (Started <a href="http://node01.example.com">node01.example.com</a>)</div><div>Jan 18 16:52:44 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: info: LogActions: Leave pingd:1 (Started <a href="http://node02.example.com">node02.example.com</a>)</div><div>Jan 18 16:53:37 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: notice: LogAction: * Stop daemon_httpd ( <a href="http://node02.example.com">node02.example.com</a> ) due to node availability</div><div>Jan 18 16:53:37 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: notice: LogAction: * Stop fs_ourApp ( <a href="http://node02.example.com">node02.example.com</a> ) due to node availability</div><div>Jan 18 16:53:37 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: notice: LogAction: * Stop ip_ourApp ( <a href="http://node02.example.com">node02.example.com</a> ) due to node availability</div><div>Jan 18 16:53:37 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: notice: LogAction: * Stop app_ourApp ( <a href="http://node02.example.com">node02.example.com</a> ) due to node availability</div><div>Jan 18 16:53:37 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: info: LogActions: Leave drbd_ourApp:0 (Slave <a href="http://node01.example.com">node01.example.com</a>)</div><div>Jan 18 16:53:37 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: notice: LogAction: * Demote drbd_ourApp:1 ( Master -> Slave <a href="http://node02.example.com">node02.example.com</a> ) </div><div>Jan 18 16:53:37 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: info: LogActions: Leave pingd:0 (Started <a href="http://node01.example.com">node01.example.com</a>)</div><div>Jan 18 16:53:37 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: info: LogActions: Leave pingd:1 (Started <a href="http://node02.example.com">node02.example.com</a>)</div><div>Jan 18 16:53:50 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: info: LogActions: Leave daemon_httpd (Stopped)</div><div>Jan 18 16:53:50 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: info: LogActions: Leave fs_ourApp (Stopped)</div><div>Jan 18 16:53:50 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: info: LogActions: Leave ip_ourApp (Stopped)</div><div>Jan 18 16:53:50 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: info: LogActions: Leave app_ourApp (Stopped)</div><div>Jan 18 16:53:50 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: info: LogActions: Leave drbd_ourApp:0 (Slave <a href="http://node01.example.com">node01.example.com</a>)</div><div>Jan 18 16:53:50 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: notice: LogAction: * Demote drbd_ourApp:1 ( Master -> Slave <a href="http://node02.example.com">node02.example.com</a> ) </div><div>Jan 18 16:53:50 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: info: LogActions: Leave pingd:0 (Started <a href="http://node01.example.com">node01.example.com</a>)</div><div>Jan 18 16:53:50 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: info: LogActions: Leave pingd:1 (Started <a href="http://node02.example.com">node02.example.com</a>)</div><div>Jan 18 16:53:51 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: notice: LogAction: * Start daemon_httpd ( <a href="http://node02.example.com">node02.example.com</a> ) </div><div>Jan 18 16:53:51 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: notice: LogAction: * Start fs_ourApp ( <a href="http://node02.example.com">node02.example.com</a> ) </div><div>Jan 18 16:53:51 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: notice: LogAction: * Start ip_ourApp ( <a href="http://node02.example.com">node02.example.com</a> ) </div><div>Jan 18 16:53:51 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: notice: LogAction: * Start app_ourApp ( <a href="http://node02.example.com">node02.example.com</a> ) </div><div>Jan 18 16:53:51 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: info: LogActions: Leave drbd_ourApp:0 (Slave <a href="http://node01.example.com">node01.example.com</a>)</div><div>Jan 18 16:53:51 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: notice: LogAction: * Promote drbd_ourApp:1 ( Slave -> Master <a href="http://node02.example.com">node02.example.com</a> ) </div><div>Jan 18 16:53:51 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: info: LogActions: Leave pingd:0 (Started <a href="http://node01.example.com">node01.example.com</a>)</div><div>Jan 18 16:53:51 [21937] <a href="http://node02.example.com">node02.example.com</a> pengine: info: LogActions: Leave pingd:1 (Started <a href="http://node02.example.com">node02.example.com</a>)</div></div><div>===</div></div></div></div></div></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Jan 19, 2021 at 2:27 AM Reid Wahl <<a href="mailto:nwahl@redhat.com">nwahl@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr">Can you share the cluster configuration (e.g., `pcs config` or the CIB)? And are there any additional LogAction messages after that one (e.g., Promote for node01)?<br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Jan 18, 2021 at 7:47 PM Stuart Massey <<a href="mailto:djangoschef@gmail.com" target="_blank">djangoschef@gmail.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div dir="ltr">So, we have a 2-node cluster with a quorum device. One of the nodes (node1) is having some trouble, so we have added constraints to prevent any resources migrating to it, but have not put it in standby, so that drbd in secondary on that node stays in sync. The problems it is having lead to OS lockups that eventually resolve themselves - but that causes it to be temporarily dropped from the cluster by the current master (node2). <div>Sometimes when node1 rejoins, then node2 will demote the drbd ms resource. That causes all resources that depend on it to be stopped, leading to a service outage. They are then restarted on node2, since they can't run on node1 (due to constraints).</div><div>We are having a hard time understanding why this happens. It seems like there may be some sort of DC contention happening. Does anyone have any idea how we might prevent this from happening?</div><div>Selected messages (de-identified) from pacemaker.log that illustrate suspicion re DC confusion are below. The update_dc and abort_transition_graph re deletion of lrm seem to always precede the demotion, and a demotion seems to always follow (when not already demoted).</div><div><br></div><div><div>Jan 18 16:52:17 [21938] <a href="http://node02.example.com" target="_blank">node02.example.com</a> crmd: info: do_dc_takeover: Taking over DC status for this partition</div><div>Jan 18 16:52:17 [21938] <a href="http://node02.example.com" target="_blank">node02.example.com</a> crmd: info: update_dc: Set DC to <a href="http://node02.example.com" target="_blank">node02.example.com</a> (3.0.14)</div><div>Jan 18 16:52:17 [21938] <a href="http://node02.example.com" target="_blank">node02.example.com</a> crmd: info: abort_transition_graph: Transition aborted by deletion of lrm[@id='1']: Resource state removal | cib=0.89.327 source=abort_unless_down:357 path=/cib/status/node_state[@id='1']/lrm[@id='1'] complete=true</div><div>Jan 18 16:52:19 [21937] <a href="http://node02.example.com" target="_blank">node02.example.com</a> pengine: info: master_color: ms_drbd_ourApp: Promoted 0 instances of a possible 1 to master</div><div>Jan 18 16:52:19 [21937] <a href="http://node02.example.com" target="_blank">node02.example.com</a> pengine: notice: LogAction: * Demote drbd_ourApp:1 ( Master -> Slave <a href="http://node02.example.com" target="_blank">node02.example.com</a> ) </div></div><div><br></div></div></div>
_______________________________________________<br>
Manage your subscription:<br>
<a href="https://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.clusterlabs.org/mailman/listinfo/users</a><br>
<br>
ClusterLabs home: <a href="https://www.clusterlabs.org/" rel="noreferrer" target="_blank">https://www.clusterlabs.org/</a><br>
</blockquote></div><br clear="all"><br>-- <br><div dir="ltr"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div>Regards,<br><br></div>Reid Wahl, RHCA<br></div><div>Senior Software Maintenance Engineer, Red Hat<br></div>CEE - Platform Support Delivery - ClusterHA</div></div></div></div></div></div></div></div></div></div></div></div></div></div>
_______________________________________________<br>
Manage your subscription:<br>
<a href="https://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.clusterlabs.org/mailman/listinfo/users</a><br>
<br>
ClusterLabs home: <a href="https://www.clusterlabs.org/" rel="noreferrer" target="_blank">https://www.clusterlabs.org/</a><br>
</blockquote></div>