[ClusterLabs] Cloned ressource is restarted on all nodes if one node fails

[Andreas Janning]

Hi all,

we recently experienced an outage in our pacemaker cluster and I would like
to understand how we can configure the cluster to avoid this problem in the
future.

First our basic setup:
- CentOS7
- Pacemaker 1.1.23
- Corosync 2.4.5
- Resource-Agents 4.1.1

Our cluster is composed of multiple active/passive nodes. Each software
component runs on two nodes simultaneously and all traffic is routed to the
active node via Virtual IP.
If the active node fails, the passive node grabs the Virtual IP and
immediately takes over all work of the failed node. Since the software is
already up and running on the passive node, there should be virtually no
downtime.
We have tried achieved this in pacemaker by configuring clone-sets for each
software component.

Now the problem:
When a software component fails on the active node, the Virtual-IP is
correctly grabbed by the passive node. BUT the software component is also
immediately restarted on the passive Node.
That unfortunately defeats the purpose of the whole setup, since we now
have a downtime until the software component is restarted on the passive
node and the restart might even fail and lead to a complete outage.
After some investigating I now understand that the cloned resource is
restarted on all nodes after a monitoring failure because the default
"on-fail" of "monitor" is restart. But that is not what I want.

I have created a minimal setup that reproduces the problem:

<configuration>
>  <crm_config>
>  <cluster_property_set id="cib-bootstrap-options">
>  <nvpair id="cib-bootstrap-options-have-watchdog" name="have-watchdog"
> value="false"/>
>  <nvpair id="cib-bootstrap-options-dc-version" name="dc-version"
> value="1.1.23-1.el7_9.1-9acf116022"/>
>  <nvpair id="cib-bootstrap-options-cluster-infrastructure"
> name="cluster-infrastructure" value="corosync"/>
>  <nvpair id="cib-bootstrap-options-cluster-name" name="cluster-name"
> value="pacemaker-test"/>
>  <nvpair id="cib-bootstrap-options-stonith-enabled" name="stonith-enabled"
> value="false"/>
>  <nvpair id="cib-bootstrap-options-symmetric-cluster"
> name="symmetric-cluster" value="false"/>
>  </cluster_property_set>
>  </crm_config>
>  <nodes>
>  <node id="1" uname="active-node"/>
>  <node id="2" uname="passive-node"/>
>  </nodes>
>  <resources>
>  <primitive class="ocf" id="vip" provider="heartbeat" type="IPaddr2">
>  <instance_attributes id="vip-instance_attributes">
>  <nvpair id="vip-instance_attributes-ip" name="ip"
> value="{{infrastructure.virtual_ip}}"/>
>  </instance_attributes>
>  <operations>
>  <op id="psa-vip-monitor-interval-10s" interval="10s" name="monitor"
> timeout="20s"/>
>  <op id="psa-vip-start-interval-0s" interval="0s" name="start"
> timeout="20s"/>
>  <op id="psa-vip-stop-interval-0s" interval="0s" name="stop"
> timeout="20s"/>
>  </operations>
>  </primitive>
>  <clone id="apache-clone">
>  <primitive class="ocf" id="apache" provider="heartbeat" type="apache">
>  <instance_attributes id="apache-instance_attributes">
>  <nvpair id="apache-instance_attributes-port" name="port" value="80"/>
>  <nvpair id="apache-instance_attributes-statusurl" name="statusurl" value="
> http://localhost/server-status"/>
>  </instance_attributes>
>  <operations>
>  <op id="apache-monitor-interval-10s" interval="10s" name="monitor"
> timeout="20s"/>
>  <op id="apache-start-interval-0s" interval="0s" name="start"
> timeout="40s"/>
>  <op id="apache-stop-interval-0s" interval="0s" name="stop" timeout="60s"/>
>  </operations>
>  </primitive>
>  <meta_attributes id="apache-meta_attributes">
>  <nvpair id="apache-clone-meta_attributes-clone-max" name="clone-max"
> value="2"/>
>  <nvpair id="apache-clone-meta_attributes-clone-node-max"
> name="clone-node-max" value="1"/>
>  </meta_attributes>
>  </clone>
>  </resources>
>  <constraints>
>  <rsc_location id="location-apache-clone-active-node-100"
> node="active-node" rsc="apache-clone" score="100"
> resource-discovery="exclusive"/>
>  <rsc_location id="location-apache-clone-passive-node-0"
> node="passive-node" rsc="apache-clone" score="0"
> resource-discovery="exclusive"/>
>  <rsc_location id="location-vip-clone-active-node-100" node="active-node"
> rsc="vip" score="100" resource-discovery="exclusive"/>
>  <rsc_location id="location-vip-clone-passive-node-0" node="passive-node"
> rsc="vip" score="0" resource-discovery="exclusive"/>
>  <rsc_colocation id="colocation-vip-apache-clone-INFINITY" rsc="vip"
> score="INFINITY" with-rsc="apache-clone"/>
>  </constraints>
>  <rsc_defaults>
>  <meta_attributes id="rsc_defaults-options">
>  <nvpair id="rsc_defaults-options-resource-stickiness"
> name="resource-stickiness" value="50"/>
>  </meta_attributes>
>  </rsc_defaults>
> </configuration>
>


When this configuration is started, httpd will be running on active-node
and passive-node. The VIP runs only on active-node.
When crashing the httpd on active-node (with killall httpd), passive-node
immediately grabs the VIP and restarts its own httpd.

How can I change this configuration so that when the resource fails on
active-node:
- passive-node immediately grabs the VIP (as it does now).
- active-node tries to restart the failed resource, giving up after x
attempts.
- passive-node does NOT restart the resource.

Regards

Andreas Janning



-- 
------------------------------

*Beste Arbeitgeber ITK 2021 - 1. Platz für QAware*
ausgezeichnet von Great Place to Work
<https://www.qaware.de/news/platz-1-bei-beste-arbeitgeber-in-der-itk-2021/>
------------------------------

Andreas Janning
Expert Software Engineer

QAware GmbH
Aschauer Straße 32
81549 München, Germany
Mobil +49 160 1492426
andreas.janning at qaware.de
www.qaware.de
------------------------------

Geschäftsführer: Christian Kamm, Johannes Weigend, Dr. Josef Adersberger
Registergericht: München
Handelsregisternummer: HRB 163761
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20210809/d778cb77/attachment.htm>