<div dir="ltr"><div dir="ltr">Hi,<div>Thank you for your response.</div><div><br></div><div><div>Ok, It seems that fencing resources and secondary timed out at the same time, together with ldap.</div></div><div>I understand that because of "migration-threshold=1", standby tried to recover just once and then was stopped. Is this ok, or the threshold should be increased?<br></div><div><br></div><div>Master server is started with systmectl, then pacemaker is started on master, which detects master and then when starting pacemaker on secondary it brings up postgres service in slave mode.</div><div>I didn't manage to start postgres master over pacemaker. I tested failover with setup like this and it works. I will try to setup postgres to be run with pacemaker, but I am concerned about those timeouts which  caused cluster to crash. Can you help me investigate why this happened or what should I change in order to avoid it? For aws virtual ip is used AWS secondary IP.</div><div>Link to the awsvip resource:</div><div><span id="m_-1337925362275024796gmail-docs-internal-guid-c19131e9-7fff-d223-96a5-9e8e806a18a1"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="text-decoration-line:underline;font-size:11pt;font-family:"Courier New";background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"><a href="https://raw.githubusercontent.com/ClusterLabs/resource-agents/master/heartbeat/awsvip" style="text-decoration-line:none" target="_blank">https://raw.githubusercontent.com/ClusterLabs/resource-agents/master/heartbeat/awsvip</a></span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><br></p><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt">Link to the ec2 stonith reosurce agent:</p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><br></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><a href="https://raw.githubusercontent.com/ClusterLabs/cluster-glue/master/lib/plugins/stonith/external/ec2" style="text-decoration-line:none" target="_blank"><span style="font-size:11pt;font-family:"Courier New";background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;text-decoration-line:underline;vertical-align:baseline;white-space:pre-wrap">https://raw.githubusercontent.com/ClusterLabs/cluster-glue/master/lib/plugins/stonith/external/ec2</span></a></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><br></p></span></div><div>Command output when cluster works:</div><div><br></div><div><span id="m_-1337925362275024796gmail-docs-internal-guid-e0541263-7fff-9ac2-73e3-9217ae984a0b"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:"Courier New";color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">crm status</span></p></span><br class="m_-1337925362275024796gmail-Apple-interchange-newline"></div><div><span id="m_-1337925362275024796gmail-docs-internal-guid-8639d71c-7fff-0a47-c6d6-6bc0ac89df4c"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:"Courier New";color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">Output:</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:"Courier New";color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">Stack: corosync</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:"Courier New";color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">Current DC: postgres-ha-1 (version 1.1.14-70404b0) - partition with quorum</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:"Courier New";color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">2 nodes and 5 resources configured</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:"Courier New";color:rgb(106,168,79);background-color:transparent;font-weight:700;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">Online</span><span style="font-size:11pt;font-family:"Courier New";color:rgb(106,168,79);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">:</span><span style="font-size:11pt;font-family:"Courier New";color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"> [ postgres-ha-1 postgres-ha-2 ]</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:"Courier New";color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">Full list of resources:</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:"Courier New";color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"> </span><span style="font-size:11pt;font-family:"Courier New";color:rgb(0,0,0);background-color:transparent;font-weight:700;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">AWSVIP</span><span style="font-size:11pt;font-family:"Courier New";color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"><span class="m_-1337925362275024796gmail-Apple-tab-span" style="white-space:pre-wrap">   </span></span><span style="font-size:11pt;font-family:"Courier New";color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">(ocf::heartbeat:awsvip):</span><span style="font-size:11pt;font-family:"Courier New";color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"><span class="m_-1337925362275024796gmail-Apple-tab-span" style="white-space:pre-wrap">   </span></span><span style="font-size:11pt;font-family:"Courier New";color:rgb(106,168,79);background-color:transparent;font-weight:700;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">Started</span><span style="font-size:11pt;font-family:"Courier New";color:rgb(106,168,79);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"> </span><span style="font-size:11pt;font-family:"Courier New";color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">postgres-ha-1</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:"Courier New";color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"> </span><span style="font-size:11pt;font-family:"Courier New";color:rgb(106,168,79);background-color:transparent;font-weight:700;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">Master</span><span style="font-size:11pt;font-family:"Courier New";color:rgb(106,168,79);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">/</span><span style="font-size:11pt;font-family:"Courier New";color:rgb(106,168,79);background-color:transparent;font-weight:700;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">Slave</span><span style="font-size:11pt;font-family:"Courier New";color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"> Set: PGSQL-HA [PGSQL]</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:"Courier New";color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">     </span><span style="font-size:11pt;font-family:"Courier New";color:rgb(106,168,79);background-color:transparent;font-weight:700;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">Master</span><span style="font-size:11pt;font-family:"Courier New";color:rgb(106,168,79);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">s</span><span style="font-size:11pt;font-family:"Courier New";color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">: [ postgres-ha-1 ]</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:"Courier New";color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">     </span><span style="font-size:11pt;font-family:"Courier New";color:rgb(106,168,79);background-color:transparent;font-weight:700;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">Slave</span><span style="font-size:11pt;font-family:"Courier New";color:rgb(106,168,79);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">s</span><span style="font-size:11pt;font-family:"Courier New";color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">: [ postgres-ha-2 ]</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:"Courier New";color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"> </span><span style="font-size:11pt;font-family:"Courier New";color:rgb(0,0,0);background-color:transparent;font-weight:700;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">fencing-postgres-ha-1</span><span style="font-size:11pt;font-family:"Courier New";color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"><span class="m_-1337925362275024796gmail-Apple-tab-span" style="white-space:pre-wrap">   </span></span><span style="font-size:11pt;font-family:"Courier New";color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">(stonith:external/ec2):</span><span style="font-size:11pt;font-family:"Courier New";color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"><span class="m_-1337925362275024796gmail-Apple-tab-span" style="white-space:pre-wrap">    </span></span><span style="font-size:11pt;font-family:"Courier New";color:rgb(106,168,79);background-color:transparent;font-weight:700;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">Started</span><span style="font-size:11pt;font-family:"Courier New";color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"> postgres-ha-2</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:"Courier New";color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"> </span><span style="font-size:11pt;font-family:"Courier New";color:rgb(0,0,0);background-color:transparent;font-weight:700;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">fencing-postgres-ha-2</span><span style="font-size:11pt;font-family:"Courier New";color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"><span class="m_-1337925362275024796gmail-Apple-tab-span" style="white-space:pre-wrap">    </span></span><span style="font-size:11pt;font-family:"Courier New";color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">(stonith:external/ec2):</span><span style="font-size:11pt;font-family:"Courier New";color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"><span class="m_-1337925362275024796gmail-Apple-tab-span" style="white-space:pre-wrap">    </span></span><span style="font-size:11pt;font-family:"Courier New";color:rgb(106,168,79);background-color:transparent;font-weight:700;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap">Started</span><span style="font-size:11pt;font-family:"Courier New";color:rgb(0,0,0);background-color:transparent;font-variant-numeric:normal;font-variant-east-asian:normal;vertical-align:baseline;white-space:pre-wrap"> postgres-ha-1</span></p></span><br class="m_-1337925362275024796gmail-Apple-interchange-newline"></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Thu, 18 Apr 2019 at 18:24, Jehan-Guillaume de Rorthais <<a href="mailto:jgdr@dalibo.com" target="_blank">jgdr@dalibo.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Thu, 18 Apr 2019 14:19:44 +0200<br>
Danka Ivanović <<a href="mailto:danka.ivanovic@gmail.com" target="_blank">danka.ivanovic@gmail.com</a>> wrote:<br>
<br>
<br>
<br>
It seems you had timeout for both fencing resources and your standby in the same<br>
time here:<br>
<br>
> Apr 17 10:03:34 master pengine[12480]: warning: Processing failed op<br>
>   monitor for fencing-secondary on master: unknown error (1)<br>
> Apr 17 10:03:34 master pengine[12480]: warning: Processing failed op<br>
>   monitor for fencing-master on secondary: unknown error (1)<br>
> Apr 17 10:03:34 master pengine[12480]: warning: Processing failed op<br>
>   monitor for PGSQL:1 on secondary: unknown error (1)<br>
> Apr 17 10:03:34 master pengine[12480]: warning: Forcing fencing-secondary<br>
>   away from master after 1 failures (max=1)<br>
> Apr 17 10:03:34 master pengine[12480]: warning: Forcing fencing-master away<br>
>   from secondary after 1 failures (max=1)<br>
> Apr 17 10:03:34 master pengine[12480]: warning: Forcing PGSQL-HA away from<br>
>   secondary after 1 failures (max=1)<br>
> Apr 17 10:03:34 master pengine[12480]: warning: Forcing PGSQL-HA away from<br>
>   secondary after 1 failures (max=1)<br>
<br>
Because you have "migration-threshold=1", the standby will be shut down:<br>
<br>
> Apr 17 10:03:34 master pengine[12480]: notice: Stop PGSQL:1 (secondary)<br>
<br>
The transition is stopped because the pgsql master timed out in the meantime<br>
:<br>
<br>
> Apr 17 10:03:40 master crmd[12481]: notice: Transition 3462 (Complete=5,<br>
> Pending=0, Fired=0, Skipped=1, Incomplete=6,<br>
> Source=/var/lib/pacemaker/pengine/pe-input-59.bz2): Stopped<br>
<br>
and as you mentioned, your ldap as well:<br>
<br>
> Apr 17 10:03:40 master nslcd[1518]: [d7e446] <group(all)> ldap_result()<br>
> timed out<br>
<br>
Here are the four timeout errors (2 fencings and 2 pgsql instances):<br>
<br>
> Apr 17 10:03:40 master pengine[12480]: warning: Processing failed op<br>
>   monitor for fencing-secondary on master: unknown error (1)<br>
> Apr 17 10:03:40 master pengine[12480]: warning: Processing failed op<br>
>   monitor for PGSQL:0 on master: unknown error (1)<br>
> Apr 17 10:03:40 master pengine[12480]: warning: Processing failed op<br>
>   monitor for fencing-master on secondary: unknown error (1)<br>
> Apr 17 10:03:40 master pengine[12480]: warning: Processing failed op<br>
>   monitor for PGSQL:1 on secondary: unknown error (1)<br>
<br>
As a reaction, Pacemaker decide to stop everything because it can not move<br>
resources anywhere:<br>
<br>
> Apr 17 10:03:40 master pengine[12480]: warning: Forcing PGSQL-HA away from<br>
> master after 1 failures (max=1)<br>
> Apr 17 10:03:40 master pengine[12480]: warning: Forcing PGSQL-HA away from<br>
> master after 1 failures (max=1)<br>
> Apr 17 10:03:40 master pengine[12480]: warning: Forcing fencing-secondary<br>
> away from master after 1 failures (max=1)<br>
> Apr 17 10:03:40 master pengine[12480]: warning: Forcing fencing-master away<br>
> from secondary after 1 failures (max=1)<br>
> Apr 17 10:03:40 master pengine[12480]: warning: Forcing PGSQL-HA away from<br>
> secondary after 1 failures (max=1)<br>
> Apr 17 10:03:40 master pengine[12480]: warning: Forcing PGSQL-HA away from<br>
> secondary after 1 failures (max=1)<br>
> Apr 17 10:03:40 master pengine[12480]: notice: Stop AWSVIP (master)<br>
> Apr 17 10:03:40 master pengine[12480]: notice: Demote PGSQL:0 (Master -><br>
> Stopped master)<br>
> Apr 17 10:03:40 master pengine[12480]: notice: Stop PGSQL:1 (secondary)<br>
<br>
Now, following lines are really not expected. Why systemd detects PostgreSQL<br>
stopped?<br>
<br>
> Apr 17 10:03:40 master postgresql@9.5-main[32458]: Cluster is not running.<br>
> Apr 17 10:03:40 master systemd[1]: postgresql@9.5-main.service: Control<br>
> process exited, code=exited status=2<br>
> Apr 17 10:03:40 master systemd[1]: postgresql@9.5-main.service: Unit<br>
> entered failed state.<br>
> Apr 17 10:03:40 master systemd[1]: postgresql@9.5-main.service: Failed with<br>
> result 'exit-code'.<br>
<br>
I suspect the service is still enabled or has been started by hand.<br>
<br>
As soon as you setup a resource in Pacemaker, admin show **always** ask<br>
Pacemaker to start/stop it. Never use systemctl to handle the resource yourself.<br>
<br>
You must disable this service in systemd.<br>
<br>
++</blockquote></div>
</div>