<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<div dir="auto">
<div>I've been experiencing exactly the same issue. Pacemaker prioritises restarting the failed resource over maintaining a master instance. In my case I used crm_simulate to analyse the actions planned and taken by pacemaker during resource recovery. It showed
that the system did plan to failover the master instance, but it was near the bottom of the action list. Higher priority was given to restarting the failed instance, consequently when that had occurred, it was easier just to promote the same instance rather
than failing over. </div>
<div dir="auto"><br>
</div>
<div dir="auto">This particular behaviour caused me a lot of headaches. In the end I had to use a workaround by setting max failures for the resource to 1, and clearing the failure after 10 seconds. This forces it to failover, but there is then a window (longer
than 10 seconds due to the cluster check timer which is used to clear failures) where the resource can't fail back if there happened to be a second failure. It also means that there is no slave running during this time, which causes a performance hit in my
case. </div>
<div dir="auto"><br>
</div>
<div dir="auto">Regards, </div>
<div dir="auto">Harvey <br>
<div dir="auto"><br>
<div class="elided-text">On 13 Aug 2019 9:17 am, Michael Powell <Michael.Powell@harmonicinc.com> wrote:<br type="attribution">
<blockquote style="margin:0 0 0 0.8ex;border-left:1px #ccc solid;padding-left:1ex">
<div>
<div>
<p>Yes, I have tried that. I used <b>crm_resource --meta -p resource-stickiness -v 0 -r SS16201289RN00023
</b>to disable resource stickiness and then <b>kill -9 <pid></b> to kill the application associated with the master resource. The results are the same: the slave resource remains a slave while the failed resource is restarted and becomes master again.</p>
<p> </p>
<p>One approach that seems to work is to run <b>crm_resource -M -r ms-SS16201289RN00023 -H mgraid-16201289RN00023-1</b> to move the resource to the other node (assuming that the master is running on node mgraid-16201289RN00023-0<b>.)
</b>My original understanding was that this would “restart” the resource on the destination node, but that was apparently a misunderstanding. I can change our scripts to use this approach, but a) thought that maintain the approach of demoting the master resource
and promoting the slave to master was more generic and b) I am unsure of any potential side effects of moving the resource. Given what I’m trying to accomplish, is this in fact the preferred approach?</p>
<p> </p>
<p>Regards,</p>
<p> Michael</p>
<p> </p>
<p> </p>
<p>-----Original Message-----<br>
From: Users <users-bounces@clusterlabs.org> On Behalf Of users-request@clusterlabs.org<br>
Sent: Monday, August 12, 2019 1:10 PM<br>
To: users@clusterlabs.org<br>
Subject: [EXTERNAL] Users Digest, Vol 55, Issue 19</p>
<p> </p>
<p>Send Users mailing list submissions to</p>
<p> <a href="mailto:users@clusterlabs.org"><span style="text-decoration:none">users@clusterlabs.org</span></a></p>
<p> </p>
<p>To subscribe or unsubscribe via the World Wide Web, visit</p>
<p> <a href="https://lists.clusterlabs.org/mailman/listinfo/users">
<span style="text-decoration:none">https://lists.clusterlabs.org/mailman/listinfo/users</span></a></p>
<p>or, via email, send a message with subject or body 'help' to</p>
<p> <a href="mailto:users-request@clusterlabs.org"><span style="text-decoration:none">users-request@clusterlabs.org</span></a></p>
<p> </p>
<p>You can reach the person managing the list at</p>
<p> <a href="mailto:users-owner@clusterlabs.org"><span style="text-decoration:none">users-owner@clusterlabs.org</span></a></p>
<p> </p>
<p>When replying, please edit your Subject line so it is more specific than "Re: Contents of Users digest..."</p>
<p> </p>
<p> </p>
<p>Today's Topics:</p>
<p> </p>
<p> 1. why is node fenced ? (Lentes, Bernd)</p>
<p> 2. Postgres HA - pacemaker RA do not support auto failback (Shital A)</p>
<p> 3. Re: why is node fenced ? (Chris Walker)</p>
<p> 4. Re: Master/slave failover does not work as expected</p>
<p> (Andrei Borzenkov)</p>
<p> </p>
<p> </p>
<p>----------------------------------------------------------------------</p>
<p> </p>
<p>Message: 1</p>
<p>Date: Mon, 12 Aug 2019 18:09:24 +0200 (CEST)</p>
<p>From: "Lentes, Bernd" <<a href="mailto:bernd.lentes@helmholtz-muenchen.de"><span style="text-decoration:none">bernd.lentes@helmholtz-muenchen.de</span></a>></p>
<p>To: Pacemaker ML <<a href="mailto:users@clusterlabs.org"><span style="text-decoration:none">users@clusterlabs.org</span></a>></p>
<p>Subject: [ClusterLabs] why is node fenced ?</p>
<p>Message-ID:</p>
<p> <<a href="mailto:546330844.1686419.1565626164456.JavaMail.zimbra@helmholtz-muenchen.de"><span style="text-decoration:none">546330844.1686419.1565626164456.JavaMail.zimbra@helmholtz-muenchen.de</span></a>></p>
<p> </p>
<p>Content-Type: text/plain; charset=utf-8</p>
<p> </p>
<p>Hi,</p>
<p> </p>
<p>last Friday (9th of August) i had to install patches on my two-node cluster.</p>
<p>I put one of the nodes (ha-idg-2) into standby (crm node standby ha-idg-2), patched it, rebooted, started the cluster (systemctl start pacemaker) again, put the node again online, everything fine.</p>
<p> </p>
<p>Then i wanted to do the same procedure with the other node (ha-idg-1).</p>
<p>I put it in standby, patched it, rebooted, started pacemaker again.</p>
<p>But then ha-idg-1 fenced ha-idg-2, it said the node is unclean.</p>
<p>I know that nodes which are unclean need to be shutdown, that's logical.</p>
<p> </p>
<p>But i don't know from where the conclusion comes that the node is unclean respectively why it is unclean, i searched in the logs and didn't find any hint.</p>
<p> </p>
<p>I put the syslog and the pacemaker log on a seafile share, i'd be very thankful if you'll have a look.</p>
<p><a href="https://hmgubox.helmholtz-muenchen.de/d/53a10960932445fb9cfe/"><span style="text-decoration:none">https://hmgubox.helmholtz-muenchen.de/d/53a10960932445fb9cfe/</span></a></p>
<p> </p>
<p>Here the cli history of the commands:</p>
<p> </p>
<p>17:03:04 crm node standby ha-idg-2</p>
<p>17:07:15 zypper up (install Updates on ha-idg-2)</p>
<p>17:17:30 systemctl reboot</p>
<p>17:25:21 systemctl start pacemaker.service</p>
<p>17:25:47 crm node online ha-idg-2</p>
<p>17:26:35 crm node standby ha-idg1-</p>
<p>17:30:21 zypper up (install Updates on ha-idg-1)</p>
<p>17:37:32 systemctl reboot</p>
<p>17:43:04 systemctl start pacemaker.service</p>
<p>17:44:00 ha-idg-1 is fenced</p>
<p> </p>
<p>Thanks.</p>
<p> </p>
<p>Bernd</p>
<p> </p>
<p>OS is SLES 12 SP4, pacemaker 1.1.19, corosync 2.3.6-9.13.1</p>
<p> </p>
<p> </p>
<p>-- </p>
<p> </p>
<p>Bernd Lentes</p>
<p>Systemadministration</p>
<p>Institut f?r Entwicklungsgenetik</p>
<p>Geb?ude 35.34 - Raum 208</p>
<p>HelmholtzZentrum m?nchen</p>
<p><a href="mailto:bernd.lentes@helmholtz-muenchen.de"><span style="text-decoration:none">bernd.lentes@helmholtz-muenchen.de</span></a></p>
<p>phone: +49 89 3187 1241</p>
<p>phone: +49 89 3187 3827</p>
<p>fax: +49 89 3187 2294</p>
<p><a href="http://www.helmholtz-muenchen.de/idg"><span style="text-decoration:none">http://www.helmholtz-muenchen.de/idg</span></a>
</p>
<p> </p>
<p>Perfekt ist wer keine Fehler macht</p>
<p>Also sind Tote perfekt</p>
<p></p>
<p> </p>
<p>Helmholtz Zentrum Muenchen</p>
<p>Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)</p>
<p>Ingolstaedter Landstr. 1</p>
<p>85764 Neuherberg</p>
<p><a href="http://www.helmholtz-muenchen.de"><span style="text-decoration:none">www.helmholtz-muenchen.de</span></a></p>
<p>Aufsichtsratsvorsitzende: MinDir'in Prof. Dr. Veronika von Messling</p>
<p>Geschaeftsfuehrung: Prof. Dr. med. Dr. h.c. Matthias Tschoep, Heinrich Bassler, Kerstin Guenther</p>
<p>Registergericht: Amtsgericht Muenchen HRB 6466</p>
<p>USt-IdNr: DE 129521671</p>
<p> </p>
<p> </p>
<p> </p>
<p>------------------------------</p>
<p> </p>
<p>Message: 2</p>
<p>Date: Mon, 12 Aug 2019 12:24:02 +0530</p>
<p>From: Shital A <<a href="mailto:brightuser2019@gmail.com"><span style="text-decoration:none">brightuser2019@gmail.com</span></a>></p>
<p>To: <a href="mailto:pgsql-general@postgresql.com"><span style="text-decoration:none">pgsql-general@postgresql.com</span></a>,
<a href="mailto:Users@clusterlabs.org"><span style="text-decoration:none">Users@clusterlabs.org</span></a></p>
<p>Subject: [ClusterLabs] Postgres HA - pacemaker RA do not support auto</p>
<p> failback</p>
<p>Message-ID:</p>
<p> <<a href="mailto:CAMp7vw_KF2eM_Buh_fPbZNC9Z6PVvx+7rxjyMhfMcoZXuWGLKw@mail.gmail.com"><span style="text-decoration:none">CAMp7vw_KF2eM_Buh_fPbZNC9Z6PVvx+7rxjyMhfMcoZXuWGLKw@mail.gmail.com</span></a>></p>
<p>Content-Type: text/plain; charset="utf-8"</p>
<p> </p>
<p>Hello,</p>
<p> </p>
<p>Postgres version : 9.6</p>
<p>OS:Rhel 7.6</p>
<p> </p>
<p>We are working on HA setup for postgres cluster of two nodes in</p>
<p>active-passive mode.</p>
<p> </p>
<p>Installed:</p>
<p>Pacemaker 1.1.19</p>
<p>Corosync 2.4.3</p>
<p> </p>
<p>The pacemaker agent with this installation doesn't support automatic</p>
<p>failback. What I mean by that is explained below:</p>
<p>1. Cluster is setup like A - B with A as master.</p>
<p>2. Kill services on A, node B will come up as master.</p>
<p>3. node A is ready to join the cluster, we have to delete the lock file it</p>
<p>creates on any one of the node and execute the cleanup command to get the</p>
<p>node back as standby</p>
<p> </p>
<p>Step 3 is manual so HA is not achieved in real sense.</p>
<p> </p>
<p>Please help to check:</p>
<p>1. Is there any version of the resouce agent which supports automatic</p>
<p>failback? To avoid generation of lock file and deleting it.</p>
<p> </p>
<p>2. If there is no such support, if we need such functionality, do we have</p>
<p>to modify existing code?</p>
<p> </p>
<p>How this can be achieved. Please suggest.</p>
<p>Thanks.</p>
<p> </p>
<p>Thanks.</p>
<p>-------------- next part --------------</p>
<p>An HTML attachment was scrubbed...</p>
<p>URL: <<a href="https://lists.clusterlabs.org/pipermail/users/attachments/20190812/737a010e/attachment-0001.html"><span style="text-decoration:none">https://lists.clusterlabs.org/pipermail/users/attachments/20190812/737a010e/attachment-0001.html</span></a>></p>
<p> </p>
<p>------------------------------</p>
<p> </p>
<p>Message: 3</p>
<p>Date: Mon, 12 Aug 2019 17:47:02 +0000</p>
<p>From: Chris Walker <<a href="mailto:cwalker@cray.com"><span style="text-decoration:none">cwalker@cray.com</span></a>></p>
<p>To: Cluster Labs - All topics related to open-source clustering</p>
<p> welcomed <<a href="mailto:users@clusterlabs.org"><span style="text-decoration:none">users@clusterlabs.org</span></a>></p>
<p>Subject: Re: [ClusterLabs] why is node fenced ?</p>
<p>Message-ID: <<a href="mailto:EAFEF777-5A49-4C06-A2F6-8711F528B4B6@cray.com"><span style="text-decoration:none">EAFEF777-5A49-4C06-A2F6-8711F528B4B6@cray.com</span></a>></p>
<p>Content-Type: text/plain; charset="utf-8"</p>
<p> </p>
<p>When ha-idg-1 started Pacemaker around 17:43, it did not see ha-idg-2, for example,</p>
<p> </p>
<p>Aug 09 17:43:05 [6318] ha-idg-1 pacemakerd: info: pcmk_quorum_notification: Quorum retained | membership=1320 members=1</p>
<p> </p>
<p>after ~20s (dc-deadtime parameter), ha-idg-2 is marked 'unclean' and STONITHed as part of startup fencing.</p>
<p> </p>
<p>There is nothing in ha-idg-2's HA logs around 17:43 indicating that it saw ha-idg-1 either, so it appears that there was no communication at all between the two nodes.</p>
<p> </p>
<p>I'm not sure exactly why the nodes did not see one another, but there are indications of network issues around this time</p>
<p> </p>
<p>2019-08-09T17:42:16.427947+02:00 ha-idg-2 kernel: [ 1229.245533] bond1: now running without any active interface!</p>
<p> </p>
<p>so perhaps that's related.</p>
<p> </p>
<p>HTH,</p>
<p>Chris</p>
<p> </p>
<p> </p>
<p>?On 8/12/19, 12:09 PM, "Users on behalf of Lentes, Bernd" <<a href="mailto:users-bounces@clusterlabs.org%20on%20behalf%20of%20bernd.lentes@helmholtz-muenchen.de"><span style="text-decoration:none">users-bounces@clusterlabs.org on behalf of bernd.lentes@helmholtz-muenchen.de</span></a>>
wrote:</p>
<p> </p>
<p> Hi,</p>
<p> </p>
<p> last Friday (9th of August) i had to install patches on my two-node cluster.</p>
<p> I put one of the nodes (ha-idg-2) into standby (crm node standby ha-idg-2), patched it, rebooted,
</p>
<p> started the cluster (systemctl start pacemaker) again, put the node again online, everything fine.</p>
<p> </p>
<p> Then i wanted to do the same procedure with the other node (ha-idg-1).</p>
<p> I put it in standby, patched it, rebooted, started pacemaker again.</p>
<p> But then ha-idg-1 fenced ha-idg-2, it said the node is unclean.</p>
<p> I know that nodes which are unclean need to be shutdown, that's logical.</p>
<p> </p>
<p> But i don't know from where the conclusion comes that the node is unclean respectively why it is unclean,</p>
<p> i searched in the logs and didn't find any hint.</p>
<p> </p>
<p> I put the syslog and the pacemaker log on a seafile share, i'd be very thankful if you'll have a look.</p>
<p> <a href="https://hmgubox.helmholtz-muenchen.de/d/53a10960932445fb9cfe/"><span style="text-decoration:none">https://hmgubox.helmholtz-muenchen.de/d/53a10960932445fb9cfe/</span></a></p>
<p> </p>
<p> Here the cli history of the commands:</p>
<p> </p>
<p> 17:03:04 crm node standby ha-idg-2</p>
<p> 17:07:15 zypper up (install Updates on ha-idg-2)</p>
<p> 17:17:30 systemctl reboot</p>
<p> 17:25:21 systemctl start pacemaker.service</p>
<p> 17:25:47 crm node online ha-idg-2</p>
<p> 17:26:35 crm node standby ha-idg1-</p>
<p> 17:30:21 zypper up (install Updates on ha-idg-1)</p>
<p> 17:37:32 systemctl reboot</p>
<p> 17:43:04 systemctl start pacemaker.service</p>
<p> 17:44:00 ha-idg-1 is fenced</p>
<p> </p>
<p> Thanks.</p>
<p> </p>
<p> Bernd</p>
<p> </p>
<p> OS is SLES 12 SP4, pacemaker 1.1.19, corosync 2.3.6-9.13.1</p>
<p> </p>
<p> </p>
<p> -- </p>
<p> </p>
<p> Bernd Lentes </p>
<p> Systemadministration </p>
<p> Institut f?r Entwicklungsgenetik </p>
<p> Geb?ude 35.34 - Raum 208 </p>
<p> HelmholtzZentrum m?nchen </p>
<p> <a href="mailto:bernd.lentes@helmholtz-muenchen.de"><span style="text-decoration:none">bernd.lentes@helmholtz-muenchen.de</span></a>
</p>
<p> phone: +49 89 3187 1241 </p>
<p> phone: +49 89 3187 3827 </p>
<p> fax: +49 89 3187 2294 </p>
<p> <a href="http://www.helmholtz-muenchen.de/idg"><span style="text-decoration:none">http://www.helmholtz-muenchen.de/idg</span></a>
</p>
<p> </p>
<p> Perfekt ist wer keine Fehler macht </p>
<p> Also sind Tote perfekt</p>
<p> </p>
<p> </p>
<p> Helmholtz Zentrum Muenchen</p>
<p> Deutsches Forschungszentrum fuer Gesundheit und Umwelt (GmbH)</p>
<p> Ingolstaedter Landstr. 1</p>
<p> 85764 Neuherberg</p>
<p> <a href="http://www.helmholtz-muenchen.de"><span style="text-decoration:none">www.helmholtz-muenchen.de</span></a></p>
<p> Aufsichtsratsvorsitzende: MinDir'in Prof. Dr. Veronika von Messling</p>
<p> Geschaeftsfuehrung: Prof. Dr. med. Dr. h.c. Matthias Tschoep, Heinrich Bassler, Kerstin Guenther</p>
<p> Registergericht: Amtsgericht Muenchen HRB 6466</p>
<p> USt-IdNr: DE 129521671</p>
<p> </p>
<p> _______________________________________________</p>
<p> Manage your subscription:</p>
<p> <a href="https://lists.clusterlabs.org/mailman/listinfo/users"><span style="text-decoration:none">https://lists.clusterlabs.org/mailman/listinfo/users</span></a></p>
<p> </p>
<p> ClusterLabs home: <a href="https://www.clusterlabs.org/"><span style="text-decoration:none">https://www.clusterlabs.org/</span></a></p>
<p> </p>
<p> </p>
<p>------------------------------</p>
<p> </p>
<p>Message: 4</p>
<p>Date: Mon, 12 Aug 2019 23:09:31 +0300</p>
<p>From: Andrei Borzenkov <<a href="mailto:arvidjaar@gmail.com"><span style="text-decoration:none">arvidjaar@gmail.com</span></a>></p>
<p>To: Cluster Labs - All topics related to open-source clustering</p>
<p> welcomed <<a href="mailto:users@clusterlabs.org"><span style="text-decoration:none">users@clusterlabs.org</span></a>></p>
<p>Cc: Venkata Reddy Chappavarapu <<a href="mailto:Venkata.Chappavarapu@harmonicinc.com"><span style="text-decoration:none">Venkata.Chappavarapu@harmonicinc.com</span></a>></p>
<p>Subject: Re: [ClusterLabs] Master/slave failover does not work as</p>
<p> expected</p>
<p>Message-ID:</p>
<p> <<a href="mailto:CAA91j0WxSxt_eVmUvXgJ_0goBkBw69r3o-VesRvGc6atg6o=jQ@mail.gmail.com"><span style="text-decoration:none">CAA91j0WxSxt_eVmUvXgJ_0goBkBw69r3o-VesRvGc6atg6o=jQ@mail.gmail.com</span></a>></p>
<p>Content-Type: text/plain; charset="utf-8"</p>
<p> </p>
<p>On Mon, Aug 12, 2019 at 4:12 PM Michael Powell <</p>
<p><a href="mailto:Michael.Powell@harmonicinc.com"><span style="text-decoration:none">Michael.Powell@harmonicinc.com</span></a>> wrote:</p>
<p> </p>
<p>> At 07:44:49, the ss agent discovers that the master instance has failed on</p>
<p>> node *mgraid?-0* as a result of a failed *ssadm* request in response to</p>
<p>> an *ss_monitor()* operation. It issues a *crm_master -Q -D* command with</p>
<p>> the intent of demoting the master and promoting the slave, on the other</p>
<p>> node, to master. The *ss_demote()* function finds that the application</p>
<p>> is no longer running and returns *OCF_NOT_RUNNING* (7). In the older</p>
<p>> product, this was sufficient to promote the other instance to master, but</p>
<p>> in the current product, that does not happen. Currently, the failed</p>
<p>> application is restarted, as expected, and is promoted to master, but this</p>
<p>> takes 10?s of seconds.</p>
<p>> </p>
<p>> </p>
<p>> </p>
<p> </p>
<p>Did you try to disable resource stickiness for this ms?</p>
<p>-------------- next part --------------</p>
<p>An HTML attachment was scrubbed...</p>
<p>URL: <<a href="https://lists.clusterlabs.org/pipermail/users/attachments/20190812/12978d55/attachment.html"><span style="text-decoration:none">https://lists.clusterlabs.org/pipermail/users/attachments/20190812/12978d55/attachment.html</span></a>></p>
<p>-------------- next part --------------</p>
<p>A non-text attachment was scrubbed...</p>
<p>Name: image001.gif</p>
<p>Type: image/gif</p>
<p>Size: 1854 bytes</p>
<p>Desc: not available</p>
<p>URL: <<a href="https://lists.clusterlabs.org/pipermail/users/attachments/20190812/12978d55/attachment.gif"><span style="text-decoration:none">https://lists.clusterlabs.org/pipermail/users/attachments/20190812/12978d55/attachment.gif</span></a>></p>
<p> </p>
<p>------------------------------</p>
<p> </p>
<p>Subject: Digest Footer</p>
<p> </p>
<p>_______________________________________________</p>
<p>Manage your subscription:</p>
<p><a href="https://lists.clusterlabs.org/mailman/listinfo/users"><span style="text-decoration:none">https://lists.clusterlabs.org/mailman/listinfo/users</span></a></p>
<p> </p>
<p>ClusterLabs home: <a href="https://www.clusterlabs.org/"><span style="text-decoration:none">https://www.clusterlabs.org/</span></a></p>
<p> </p>
<p>------------------------------</p>
<p> </p>
<p>End of Users Digest, Vol 55, Issue 19</p>
<p>*************************************</p>
</div>
</div>
</blockquote>
</div>
<br>
</div>
</div>
</div>
</body>
</html>