<div dir="ltr"><div>Ken,</div><div><br></div><div>I have another set of logs : </div><div><br></div><div><font face="monospace" size="2">Sep 01 09:10:05 [1328] <a href="http://TPC-F9-26.phaedrus.sandvine.com">TPC-F9-26.phaedrus.sandvine.com</a> crmd: info: do_lrm_rsc_op: Performing key=5:50864:0:86160921-abd7-4e14-94d4-f53cee278858 op=SVSDEHA_monitor_2000<br>SvsdeStateful(SVSDEHA)[6174]: 2017/09/01_09:10:06 ERROR: Resource is in failed state<br>Sep 01 09:10:06 [1328] <a href="http://TPC-F9-26.phaedrus.sandvine.com">TPC-F9-26.phaedrus.sandvine.com</a> crmd: info: action_synced_wait: Managed SvsdeStateful_meta-data_0 process 6274 exited with rc=4<br>Sep 01 09:10:06 [1328] <a href="http://TPC-F9-26.phaedrus.sandvine.com">TPC-F9-26.phaedrus.sandvine.com</a> crmd: error: generic_get_metadata: Failed to receive meta-data for ocf:pacemaker:SvsdeStateful<br>Sep 01 09:10:06 [1328] <a href="http://TPC-F9-26.phaedrus.sandvine.com">TPC-F9-26.phaedrus.sandvine.com</a> crmd: error: build_operation_update: No metadata for ocf::pacemaker:SvsdeStateful<br>Sep 01 09:10:06 [1328] <a href="http://TPC-F9-26.phaedrus.sandvine.com">TPC-F9-26.phaedrus.sandvine.com</a> crmd: info: process_lrm_event: Result of monitor operation for SVSDEHA on <a href="http://TPC-F9-26.phaedrus.sandvine.com">TPC-F9-26.phaedrus.sandvine.com</a>: 0 (ok) | call=939 key=SVSDEHA_monitor_2000 confirmed=false cib-update=476<br>Sep 01 09:10:06 [1325] <a href="http://TPC-F9-26.phaedrus.sandvine.com">TPC-F9-26.phaedrus.sandvine.com</a> cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/476)<br>Sep 01 09:10:06 [1325] <a href="http://TPC-F9-26.phaedrus.sandvine.com">TPC-F9-26.phaedrus.sandvine.com</a> cib: info: cib_perform_op: Diff: --- 0.37.4054 2<br>Sep 01 09:10:06 [1325] <a href="http://TPC-F9-26.phaedrus.sandvine.com">TPC-F9-26.phaedrus.sandvine.com</a> cib: info: cib_perform_op: Diff: +++ 0.37.4055 (null)<br>Sep 01 09:10:06 [1325] <a href="http://TPC-F9-26.phaedrus.sandvine.com">TPC-F9-26.phaedrus.sandvine.com</a> cib: info: cib_perform_op: + /cib: @num_updates=4055<br>Sep 01 09:10:06 [1325] <a href="http://TPC-F9-26.phaedrus.sandvine.com">TPC-F9-26.phaedrus.sandvine.com</a> cib: info: cib_perform_op: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='SVSDEHA']: <lrm_rsc_op id="SVSDEHA_monitor_2000" operation_key="SVSDEHA_monitor_2000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.10" transition-key="5:50864:0:86160921-abd7-4e14-94d4-f53cee278858" transition-magic="0:0;5:50864:0:86160921-abd7-4e14-94d4-f53cee278858" on_node="TPC-F9-26.phaedrus.sandvi<br>Sep 01 09:10:06 [1325] <a href="http://TPC-F9-26.phaedrus.sandvine.com">TPC-F9-26.phaedrus.sandvine.com</a> cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=<a href="http://TPC-F9-26.phaedrus.sandvine.com/crmd/476">TPC-F9-26.phaedrus.sandvine.com/crmd/476</a>, version=0.37.4055)<br><b>Sep 01 09:10:12 [1325] <a href="http://TPC-F9-26.phaedrus.sandvine.com">TPC-F9-26.phaedrus.sandvine.com</a> cib: info: cib_process_ping: Reporting our current digest to <a href="http://TPC-E9-23.phaedrus.sandvine.com">TPC-E9-23.phaedrus.sandvine.com</a>: 74bbb7e9f35fabfdb624300891e32018 for 0.37.4055 (0x7f5719954560 0)<br>Sep 01 09:15:33 [1325] <a href="http://TPC-F9-26.phaedrus.sandvine.com">TPC-F9-26.phaedrus.sandvine.com</a> cib: info: cib_perform_op: Diff: --- 0.37.4055 2</b><br>Sep 01 09:15:33 [1325] <a href="http://TPC-F9-26.phaedrus.sandvine.com">TPC-F9-26.phaedrus.sandvine.com</a> cib: info: cib_perform_op: Diff: +++ 0.37.4056 (null)<br>Sep 01 09:15:33 [1325] <a href="http://TPC-F9-26.phaedrus.sandvine.com">TPC-F9-26.phaedrus.sandvine.com</a> cib: info: cib_perform_op: + /cib: @num_updates=4056<br>Sep 01 09:15:33 [1325] <a href="http://TPC-F9-26.phaedrus.sandvine.com">TPC-F9-26.phaedrus.sandvine.com</a> cib: info: cib_perform_op: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='SVSDEHA']: <lrm_rsc_op id="SVSDEHA_last_failure_0" operation_key="SVSDEHA_monitor_1000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.10" transition-key="7:50662:8:86160921-abd7-4e14-94d4-f53cee278858" transition-magic="2:1;7:50662:8:86160921-abd7-4e14-94d4-f53cee278858" on_node="TPC-E9-23.phaedrus.sand<br>Sep 01 09:15:33 [1325] <a href="http://TPC-F9-26.phaedrus.sandvine.com">TPC-F9-26.phaedrus.sandvine.com</a> cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=<a href="http://TPC-E9-23.phaedrus.sandvine.com/crmd/53508">TPC-E9-23.phaedrus.sandvine.com/crmd/53508</a>, version=0.37.4056)<br>Sep 01 09:15:33 [1327] <a href="http://TPC-F9-26.phaedrus.sandvine.com">TPC-F9-26.phaedrus.sandvine.com</a> attrd: info: attrd_peer_update: Setting fail-count-SVSDEHA[<a href="http://TPC-E9-23.phaedrus.sandvine.com">TPC-E9-23.phaedrus.sandvine.com</a>]: (null) -> 1 from <a href="http://TPC-E9-23.phaedrus.sandvine.com">TPC-E9-23.phaedrus.sandvine.com</a><br>Sep 01 09:15:33 [1327] <a href="http://TPC-F9-26.phaedrus.sandvine.com">TPC-F9-26.phaedrus.sandvine.com</a> attrd: info: attrd_peer_update: Setting last-failure-SVSDEHA[<a href="http://TPC-E9-23.phaedrus.sandvine.com">TPC-E9-23.phaedrus.sandvine.com</a>]: (null) -> 1504271733 from <a href="http://TPC-E9-23.phaedrus.sandvine.com">TPC-E9-23.phaedrus.sandvine.com</a><br>Sep 01 09:15:33 [1325] <a href="http://TPC-F9-26.phaedrus.sandvine.com">TPC-F9-26.phaedrus.sandvine.com</a> cib: info: cib_perform_op: Diff: --- 0.37.4056 2<br>Sep 01 09:15:33 [1325] <a href="http://TPC-F9-26.phaedrus.sandvine.com">TPC-F9-26.phaedrus.sandvine.com</a> cib: info: cib_perform_op: Diff: +++ 0.37.4057 (null)<br>Sep 01 09:15:33 [1325] <a href="http://TPC-F9-26.phaedrus.sandvine.com">TPC-F9-26.phaedrus.sandvine.com</a> cib: info: cib_perform_op: + /cib: @num_updates=4057<br>Sep 01 09:15:33 [1325] <a href="http://TPC-F9-26.phaedrus.sandvine.com">TPC-F9-26.phaedrus.sandvine.com</a> cib: info: cib_perform_op: ++ /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']: <nvpair id="status-2-fail-count-SVSDEHA" name="fail-count-SVSDEHA" </font></div><div><font face="monospace" size="2">value="1"/></font></div><div><font face="Consolas" size="2"><br></font></div><div>I was suspecting around the highlighted parts of the logs above. </div><div><font face="sans-serif" size="2">After 09:10:12 the next log is at 09:15:33. During this time other node failed several times but was not migrated here.</font></div><div><font size="2"><br></font></div><div><font size="2">I am yet to check with sbd fencing with the patch shared by Klaus.</font></div><div><font size="2">I am on CentOS. </font></div><div><font face="monospace" size="2"><br></font></div><div><font face="monospace" size="2"># cat /etc/centos-release<br>CentOS Linux release 7.3.1611 (Core)</font></div><div><font face="sans-serif" size="2"><br></font></div><div><font face="sans-serif" size="2">Regards,</font></div><div><font face="sans-serif" size="2">Abhay</font></div><div><font face="sans-serif" size="2"><br></font></div><div><br></div><div><b></b><i></i><u></u><sub></sub><sup></sup><strike></strike><b></b><i></i><u></u><sub></sub><sup></sup><strike></strike><b></b><b></b><font face="sans-serif"></font><br></div></div><br><div class="gmail_quote"><div dir="ltr">On Sat, 2 Sep 2017 at 15:23 Klaus Wenninger <<a href="mailto:kwenning@redhat.com">kwenning@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 09/01/2017 11:45 PM, Ken Gaillot wrote:<br>
> On Fri, 2017-09-01 at 15:06 +0530, Abhay B wrote:<br>
>> Are you sure the monitor stopped? Pacemaker only logs<br>
>> recurring monitors<br>
>> when the status changes. Any successful monitors after this<br>
>> wouldn't be<br>
>> logged.<br>
>><br>
>> Yes. Since there were no logs which said "RecurringOp: Start<br>
>> recurring monitor" on the node after it had failed.<br>
>> Also there were no logs for any actions pertaining to<br>
>> The problem was that even though the one node was failing, the<br>
>> resources were never moved to the other node(the node on which I<br>
>> suspect monitoring had stopped).<br>
>><br>
>><br>
>> There are a lot of resource action failures, so I'm not sure<br>
>> where the<br>
>> issue is, but I'm guessing it has to do with<br>
>> migration-threshold=1 --<br>
>> once a resource has failed once on a node, it won't be allowed<br>
>> back on<br>
>> that node until the failure is cleaned up. Of course you also<br>
>> have<br>
>> failure-timeout=1s, which should clean it up immediately, so<br>
>> I'm not<br>
>> sure.<br>
>><br>
>><br>
>> migration-threshold=1<br>
>> failure-timeout=1s<br>
>><br>
>> cluster-recheck-interval=2s<br>
>><br>
>><br>
>> first, set "two_node:<br>
>> 1" in corosync.conf and let no-quorum-policy default in<br>
>> pacemaker<br>
>><br>
>><br>
>> This is already configured.<br>
>> # cat /etc/corosync/corosync.conf<br>
>> totem {<br>
>> version: 2<br>
>> secauth: off<br>
>> cluster_name: SVSDEHA<br>
>> transport: udpu<br>
>> token: 5000<br>
>> }<br>
>><br>
>><br>
>> nodelist {<br>
>> node {<br>
>> ring0_addr: 2.0.0.10<br>
>> nodeid: 1<br>
>> }<br>
>><br>
>><br>
>> node {<br>
>> ring0_addr: 2.0.0.11<br>
>> nodeid: 2<br>
>> }<br>
>> }<br>
>><br>
>><br>
>> quorum {<br>
>> provider: corosync_votequorum<br>
>> two_node: 1<br>
>> }<br>
>><br>
>><br>
>> logging {<br>
>> to_logfile: yes<br>
>> logfile: /var/log/cluster/corosync.log<br>
>> to_syslog: yes<br>
>> }<br>
>><br>
>><br>
>> let no-quorum-policy default in pacemaker; then,<br>
>> get stonith configured, tested, and enabled<br>
>><br>
>><br>
>> By not configuring no-quorum-policy, would it ignore quorum for a 2<br>
>> node cluster?<br>
> With two_node, corosync always provides quorum to pacemaker, so<br>
> pacemaker doesn't see any quorum loss. The only significant difference<br>
> from ignoring quorum is that corosync won't form a cluster from a cold<br>
> start unless both nodes can reach each other (a safety feature).<br>
><br>
>> For my use case I don't need stonith enabled. My intention is to have<br>
>> a highly available system all the time.<br>
> Stonith is the only way to recover from certain types of failure, such<br>
> as the "split brain" scenario, and a resource that fails to stop.<br>
><br>
> If your nodes are physical machines with hardware watchdogs, you can set<br>
> up sbd for fencing without needing any extra equipment.<br>
<br>
Small caveat here:<br>
If I get it right you have a 2-node-setup. In this case the watchdog-only<br>
sbd-setup would not be usable as it relies on 'real' quorum.<br>
In 2-node-setups sbd needs at least a single shared disk.<br>
For the sbd-single-disk-setup working with 2-node<br>
you need the patch from <a href="https://github.com/ClusterLabs/sbd/pull/23" rel="noreferrer" target="_blank">https://github.com/ClusterLabs/sbd/pull/23</a><br>
in place. (Saw you mentioning RHEL documentation - RHEL-7.4 has<br>
it in since GA)<br>
<br>
Regards,<br>
Klaus<br>
<br>
><br>
>> I will test my RA again as suggested with no-quorum-policy=default.<br>
>><br>
>><br>
>> One more doubt.<br>
>> Why do we see this is 'pcs property' ?<br>
>> last-lrm-refresh: 1504090367<br>
>><br>
>><br>
>><br>
>> Never seen this on a healthy cluster.<br>
>> From RHEL documentation:<br>
>> last-lrm-refresh<br>
>><br>
>> Last refresh of the<br>
>> Local Resource Manager,<br>
>> given in units of<br>
>> seconds since epoca.<br>
>> Used for diagnostic<br>
>> purposes; not<br>
>> user-configurable.<br>
>><br>
>><br>
>> Doesn't explain much.<br>
> Whenever a cluster property changes, the cluster rechecks the current<br>
> state to see if anything needs to be done. last-lrm-refresh is just a<br>
> dummy property that the cluster uses to trigger that. It's set in<br>
> certain rare circumstances when a resource cleanup is done. You should<br>
> see a line in your logs like "Triggering a refresh after ... deleted ...<br>
> from the LRM". That might give some idea of why.<br>
><br>
>> Also. does avg. CPU load impact resource monitoring ?<br>
>><br>
>><br>
>> Regards,<br>
>> Abhay<br>
> Well, it could cause the monitor to take so long that it times out. The<br>
> only direct effect of load on pacemaker is that the cluster might lower<br>
> the number of agent actions that it can execute simultaneously.<br>
><br>
><br>
>> On Thu, 31 Aug 2017 at 20:11 Ken Gaillot <<a href="mailto:kgaillot@redhat.com" target="_blank">kgaillot@redhat.com</a>> wrote:<br>
>><br>
>> On Thu, 2017-08-31 at 06:41 +0000, Abhay B wrote:<br>
>> > Hi,<br>
>> ><br>
>> ><br>
>> > I have a 2 node HA cluster configured on CentOS 7 with pcs<br>
>> command.<br>
>> ><br>
>> ><br>
>> > Below are the properties of the cluster :<br>
>> ><br>
>> ><br>
>> > # pcs property<br>
>> > Cluster Properties:<br>
>> > cluster-infrastructure: corosync<br>
>> > cluster-name: SVSDEHA<br>
>> > cluster-recheck-interval: 2s<br>
>> > dc-deadtime: 5<br>
>> > dc-version: 1.1.15-11.el7_3.5-e174ec8<br>
>> > have-watchdog: false<br>
>> > last-lrm-refresh: 1504090367<br>
>> > no-quorum-policy: ignore<br>
>> > start-failure-is-fatal: false<br>
>> > stonith-enabled: false<br>
>> ><br>
>> ><br>
>> > PFA the cib.<br>
>> > Also attached is the corosync.log around the time the below<br>
>> issue<br>
>> > happened.<br>
>> ><br>
>> ><br>
>> > After around 10 hrs and multiple failures, pacemaker stops<br>
>> monitoring<br>
>> > resource on one of the nodes in the cluster.<br>
>> ><br>
>> ><br>
>> > So even though the resource on other node fails, it is never<br>
>> migrated<br>
>> > to the node on which the resource is not monitored.<br>
>> ><br>
>> ><br>
>> > Wanted to know what could have triggered this and how to<br>
>> avoid getting<br>
>> > into such scenarios.<br>
>> > I am going through the logs and couldn't find why this<br>
>> happened.<br>
>> ><br>
>> ><br>
>> > After this log the monitoring stopped.<br>
>> ><br>
>> > Aug 29 11:01:44 [16500] <a href="http://TPC-D12-10-002.phaedrus.sandvine.com" rel="noreferrer" target="_blank">TPC-D12-10-002.phaedrus.sandvine.com</a><br>
>> > crmd: info: process_lrm_event: Result of monitor<br>
>> operation for<br>
>> > SVSDEHA on <a href="http://TPC-D12-10-002.phaedrus.sandvine.com" rel="noreferrer" target="_blank">TPC-D12-10-002.phaedrus.sandvine.com</a>: 0 (ok) |<br>
>> call=538<br>
>> > key=SVSDEHA_monitor_2000 confirmed=false cib-update=50013<br>
>><br>
>> Are you sure the monitor stopped? Pacemaker only logs<br>
>> recurring monitors<br>
>> when the status changes. Any successful monitors after this<br>
>> wouldn't be<br>
>> logged.<br>
>><br>
>> > Below log says the resource is leaving the cluster.<br>
>> > Aug 29 11:01:44 [16499] <a href="http://TPC-D12-10-002.phaedrus.sandvine.com" rel="noreferrer" target="_blank">TPC-D12-10-002.phaedrus.sandvine.com</a><br>
>> > pengine: info: LogActions: Leave SVSDEHA:0<br>
>> (Slave<br>
>> > <a href="http://TPC-D12-10-002.phaedrus.sandvine.com" rel="noreferrer" target="_blank">TPC-D12-10-002.phaedrus.sandvine.com</a>)<br>
>><br>
>> This means that the cluster will leave the resource where it<br>
>> is (i.e. it<br>
>> doesn't need a start, stop, move, demote, promote, etc.).<br>
>><br>
>> > Let me know if anything more is needed.<br>
>> ><br>
>> ><br>
>> > Regards,<br>
>> > Abhay<br>
>> ><br>
>> ><br>
>> > PS:'pcs resource cleanup' brought the cluster back into good<br>
>> state.<br>
>><br>
>> There are a lot of resource action failures, so I'm not sure<br>
>> where the<br>
>> issue is, but I'm guessing it has to do with<br>
>> migration-threshold=1 --<br>
>> once a resource has failed once on a node, it won't be allowed<br>
>> back on<br>
>> that node until the failure is cleaned up. Of course you also<br>
>> have<br>
>> failure-timeout=1s, which should clean it up immediately, so<br>
>> I'm not<br>
>> sure.<br>
>><br>
>> My gut feeling is that you're trying to do too many things at<br>
>> once. I'd<br>
>> start over from scratch and proceed more slowly: first, set<br>
>> "two_node:<br>
>> 1" in corosync.conf and let no-quorum-policy default in<br>
>> pacemaker; then,<br>
>> get stonith configured, tested, and enabled; then, test your<br>
>> resource<br>
>> agent manually on the command line to make sure it conforms to<br>
>> the<br>
>> expected return values<br>
>> ( <a href="http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#ap-ocf" rel="noreferrer" target="_blank">http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html-single/Pacemaker_Explained/index.html#ap-ocf</a> ); then add your resource to the cluster without migration-threshold or failure-timeout, and work out any issues with frequent failures; then finally set migration-threshold and failure-timeout to reflect how you want recovery to proceed.<br>
>> --<br>
>> Ken Gaillot <<a href="mailto:kgaillot@redhat.com" target="_blank">kgaillot@redhat.com</a>><br>
>><br>
>><br>
>><br>
>><br>
>><br>
>> _______________________________________________<br>
>> Users mailing list: <a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
>> <a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/mailman/listinfo/users</a><br>
>><br>
>> Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
>> Getting started:<br>
>> <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
>> Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
<br>
<br>
_______________________________________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/mailman/listinfo/users</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
</blockquote></div>