<div dir="ltr"><div>Hi all,</div><div><br></div><div>we recently experienced an outage in our pacemaker cluster and I would like to understand how we can configure the cluster to avoid this problem in the future.</div><div><br></div><div>First our basic setup:</div><div>- CentOS7</div><div>- Pacemaker 1.1.23</div><div>- Corosync 2.4.5</div><div>- Resource-Agents 4.1.1</div><div><br></div><div>Our cluster is composed of multiple active/passive nodes. Each software component runs on two nodes simultaneously and all traffic is routed to the active node via Virtual IP.</div><div>If the active node fails, the passive node grabs the Virtual IP and immediately takes over all work of the failed node. Since the software is already up and running on the passive node, there should be virtually no downtime.</div><div>We have tried achieved this in pacemaker by configuring clone-sets for each software component.</div><div><br></div><div>Now the problem:</div><div>When a software component fails on the active node, the Virtual-IP is correctly grabbed by the passive node. BUT the software component is also immediately restarted on the passive Node.</div><div>That unfortunately defeats the purpose of the whole setup, since we now have a downtime until the software component is restarted on the passive node and the restart might even fail and lead to a complete outage.</div><div>After some investigating I now understand that the cloned resource is restarted on all nodes after a monitoring failure because the default "on-fail" of "monitor" is restart. But that is not what I want.</div><div><br></div><div>I have created a minimal setup that reproduces the problem:</div><div><br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div><configuration><br> <crm_config><br> <cluster_property_set id="cib-bootstrap-options"><br> <nvpair id="cib-bootstrap-options-have-watchdog" name="have-watchdog" value="false"/><br> <nvpair id="cib-bootstrap-options-dc-version" name="dc-version" value="1.1.23-1.el7_9.1-9acf116022"/><br> <nvpair id="cib-bootstrap-options-cluster-infrastructure" name="cluster-infrastructure" value="corosync"/><br> <nvpair id="cib-bootstrap-options-cluster-name" name="cluster-name" value="pacemaker-test"/><br> <nvpair id="cib-bootstrap-options-stonith-enabled" name="stonith-enabled" value="false"/><br> <nvpair id="cib-bootstrap-options-symmetric-cluster" name="symmetric-cluster" value="false"/><br> </cluster_property_set><br> </crm_config><br> <nodes><br> <node id="1" uname="active-node"/><br> <node id="2" uname="passive-node"/><br> </nodes><br> <resources><br> <primitive class="ocf" id="vip" provider="heartbeat" type="IPaddr2"><br> <instance_attributes id="vip-instance_attributes"><br> <nvpair id="vip-instance_attributes-ip" name="ip" value="{{infrastructure.virtual_ip}}"/><br> </instance_attributes><br> <operations><br> <op id="psa-vip-monitor-interval-10s" interval="10s" name="monitor" timeout="20s"/><br> <op id="psa-vip-start-interval-0s" interval="0s" name="start" timeout="20s"/><br> <op id="psa-vip-stop-interval-0s" interval="0s" name="stop" timeout="20s"/><br> </operations><br> </primitive><br> <clone id="apache-clone"><br> <primitive class="ocf" id="apache" provider="heartbeat" type="apache"><br> <instance_attributes id="apache-instance_attributes"><br> <nvpair id="apache-instance_attributes-port" name="port" value="80"/><br> <nvpair id="apache-instance_attributes-statusurl" name="statusurl" value="<a href="http://localhost/server-status">http://localhost/server-status</a>"/><br> </instance_attributes><br> <operations><br> <op id="apache-monitor-interval-10s" interval="10s" name="monitor" timeout="20s"/><br> <op id="apache-start-interval-0s" interval="0s" name="start" timeout="40s"/><br> <op id="apache-stop-interval-0s" interval="0s" name="stop" timeout="60s"/><br> </operations><br> </primitive><br> <meta_attributes id="apache-meta_attributes"><br> <nvpair id="apache-clone-meta_attributes-clone-max" name="clone-max" value="2"/><br> <nvpair id="apache-clone-meta_attributes-clone-node-max" name="clone-node-max" value="1"/><br> </meta_attributes><br> </clone><br> </resources><br> <constraints><br> <rsc_location id="location-apache-clone-active-node-100" node="active-node" rsc="apache-clone" score="100" resource-discovery="exclusive"/><br> <rsc_location id="location-apache-clone-passive-node-0" node="passive-node" rsc="apache-clone" score="0" resource-discovery="exclusive"/><br> <rsc_location id="location-vip-clone-active-node-100" node="active-node" rsc="vip" score="100" resource-discovery="exclusive"/><br> <rsc_location id="location-vip-clone-passive-node-0" node="passive-node" rsc="vip" score="0" resource-discovery="exclusive"/><br> <rsc_colocation id="colocation-vip-apache-clone-INFINITY" rsc="vip" score="INFINITY" with-rsc="apache-clone"/><br> </constraints><br> <rsc_defaults><br> <meta_attributes id="rsc_defaults-options"><br> <nvpair id="rsc_defaults-options-resource-stickiness" name="resource-stickiness" value="50"/><br> </meta_attributes><br> </rsc_defaults><br></configuration><br></div></blockquote><div><br></div><div><br></div><div>When this configuration is started, httpd will be running on active-node and passive-node. The VIP runs only on active-node.</div><div>When crashing the httpd on active-node (with killall httpd), passive-node immediately grabs the VIP and restarts its own httpd.</div><div><br></div><div>How can I change this configuration so that when the resource fails on active-node:</div><div>- passive-node immediately grabs the VIP (as it does now).<br></div><div>- active-node tries to restart the failed resource, giving up after x attempts.</div><div>- passive-node does NOT restart the resource.</div><div><br></div><div>Regards</div><div><br></div><div>Andreas Janning<br></div><br><div><div><br></div><div><br>-- <br><div dir="ltr" class="gmail_signature" data-smartmail="gmail_signature"><div dir="ltr"><div>
<hr style="height:1px;background-color:#ccc;border:none" width="100%" size="1" align="center">
<div style="font-size:8pt;font-family:sans-serif">
<p style="margin:0pt 1pt 0pt">
<b>Beste Arbeitgeber ITK 2021 - 1. Platz für QAware</b><br>
ausgezeichnet von
<a href="https://www.qaware.de/news/platz-1-bei-beste-arbeitgeber-in-der-itk-2021/" rel="nofollow" target="_blank">Great Place to Work</a>
</p>
<hr style="height:1px;background-color:#ccc;border:none" width="100%" size="1" align="center">
<p style="margin:0pt 1pt 8pt">
Andreas Janning<br>
Expert Software Engineer<br>
</p>
<p style="margin:0pt 1pt 0pt">
QAware GmbH<br>
Aschauer Straße 32<br>
81549 München, Germany<br>
Mobil <span>+49 160 1492426</span><br>
<a href="mailto:andreas.janning@qaware.de" rel="nofollow" target="_blank">andreas.janning@qaware.de</a><br>
<a href="https://www.qaware.de" rel="nofollow" target="_blank">www.qaware.de</a><br>
</p>
</div>
<hr style="height:1px;background-color:#ccc;border:none" width="100%" size="1" align="center">
<div style="font-size:7pt;font-family:sans-serif">
<p style="margin:0pt 1pt 14pt">
Geschäftsführer: Christian Kamm, Johannes Weigend, Dr. Josef Adersberger<br>
Registergericht: München<br>
Handelsregisternummer: HRB 163761<br>
</p>
</div>
</div></div></div></div></div></div>