<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Klaus,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks for the infor!</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Did you mean i should compile sbd from github source to include the fixs you mentioned by myself?</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
The <span style="font-family: Calibri, Arial, Helvetica, sans-serif; background-color: rgb(255, 255, 255); display: inline !important">
corosync, </span>pacemaker and sbd version in my setup is as below:</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
corosync: 2.3.6-9.13.1</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt;">pacemaker: 1.1.16-6.5.1</span><br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
sbd: 1.3.1+20180507</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Regards</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Fulong</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Klaus Wenninger <kwenning@redhat.com><br>
<b>Sent:</b> Monday, February 11, 2019 18:51<br>
<b>To:</b> Cluster Labs - All topics related to open-source clustering welcomed; Fulong Wang; Gao,Yan<br>
<b>Subject:</b> Re: [ClusterLabs] SuSE12SP3 HAE SBD Communication Issue</font>
<div> </div>
</div>
<div style="background-color:#FFFFFF">
<div class="x_moz-cite-prefix">On 02/11/2019 09:49 AM, Fulong Wang wrote:<br>
</div>
<blockquote type="cite">
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
Thanks Yan,</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<div style="margin:0px; font-size:12pt; font-family:Calibri,Arial,Helvetica,sans-serif">
You gave me more valuable hints on the SBD operation!</div>
<div style="margin:0px; font-size:12pt; font-family:Calibri,Arial,Helvetica,sans-serif">
Now, i can see the verbose output after service restart.</div>
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="">>Be aware since pacemaker integration (-P) is enabled by default, which<span> </span></span><br style="">
<span style="">>means despite the sbd failure, if the node itself was clean and<span> </span></span><br style="">
<span style="">>"healthy" from pacemaker's point of view and if it's in the cluster<span> </span></span><br style="">
<span style="">>partition with the quorum, it wouldn't self-fence -- meaning a node just<span> </span></span><br style="">
<span style="">>being unable to fence doesn't necessarily need to be fenced.</span><br style="">
<br style="">
<span style="">>As described in sbd man page, "this allows sbd to survive temporary<span> </span></span><br style="">
<span style="">>outages of the majority of devices. However, while the cluster is in<span> </span></span><br style="">
<span style="">>such a degraded state, it can neither successfully fence nor be shutdown<span> </span></span><br style="">
<span style="">>cleanly (as taking the cluster below the quorum threshold will<span> </span></span><br style="">
<span style="">>immediately cause all remaining nodes to self-fence). In short, it will<span> </span></span><br style="">
<span style="">>not tolerate any further faults. Please repair the system before<span> </span></span><br style="">
<span style="">>continuing."</span><br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style=""><br>
</span></div>
<div style=""><span style="background-color:rgb(255,255,255); display:inline!important">
<div style="margin:0px"><font color="#000000" face="Calibri, Arial, Helvetica, sans-serif" style="color:rgb(0,0,0); font-family:Calibri,Arial,Helvetica,sans-serif; font-size:inherit; font-style:inherit; font-variant-ligatures:inherit; font-variant-caps:inherit; font-weight:inherit">Yes,
I can see the "<span style="">pacemaker integration</span>" was enabled in my sbd config file by default.</font></div>
<div style="margin:0px"><font color="#000000" face="Calibri, Arial, Helvetica, sans-serif" style="color:rgb(0,0,0); font-family:Calibri,Arial,Helvetica,sans-serif; font-size:inherit; font-style:inherit; font-variant-ligatures:inherit; font-variant-caps:inherit; font-weight:inherit">So,
you mean in some sbd failure cases, if the node was considered as "healthy" from pacemaker's poinit of view, it still wouldn't sel-fence. </font></div>
<div style="margin:0px"><font color="#000000" face="Calibri, Arial, Helvetica, sans-serif" style="color:rgb(0,0,0); font-family:Calibri,Arial,Helvetica,sans-serif; font-size:inherit; font-style:inherit; font-variant-ligatures:inherit; font-variant-caps:inherit; font-weight:inherit"><br>
</font></div>
<div style="margin:0px"><font color="#000000" face="Calibri, Arial, Helvetica, sans-serif" style="color:rgb(0,0,0); font-family:Calibri,Arial,Helvetica,sans-serif; font-size:inherit; font-style:inherit; font-variant-ligatures:inherit; font-variant-caps:inherit; font-weight:inherit">H</font><span style="color:rgb(0,0,0); font-family:Calibri,Arial,Helvetica,sans-serif">onestly
speaking, i didn't get you at this point. </span><span style="color:rgb(0,0,0); font-family:Calibri,Arial,Helvetica,sans-serif">I have "</span><span style="color:rgb(0,0,0); font-family:Calibri,Arial,Helvetica,sans-serif">no-quorum-policy=ignore" setting in
my setup and it's a two node cluster. </span></div>
<div style="margin:0px"><font color="#000000" face="Calibri, Arial, Helvetica, sans-serif" style="color:rgb(0,0,0); font-family:Calibri,Arial,Helvetica,sans-serif; font-size:inherit; font-style:inherit; font-variant-ligatures:inherit; font-variant-caps:inherit; font-weight:inherit">Can
you show me a sample situation for this?</font><br>
</div>
</span></div>
</blockquote>
<br>
<font face="Calibri, Arial, Helvetica, sans-serif">When using sbd with 2-node-clusters and pacemaker-integration you might<br>
check <a class="x_moz-txt-link-freetext" href="https://github.com/ClusterLabs/sbd/commit/4bd0a66da3ac9c9afaeb8a2468cdd3ed51ad3377">
https://github.com/ClusterLabs/sbd/commit/4bd0a66da3ac9c9afaeb8a2468cdd3ed51ad3377</a><br>
to be included in your sbd-version.<br>
This is relevant when 2-node is configured in corosync.<br>
<br>
Regards,<br>
Klaus<br>
<br>
</font>
<blockquote type="cite">
<div style=""><span style="background-color:rgb(255,255,255); display:inline!important">
<div style="margin:0px"></div>
<div style="margin:0px"><font color="#000000" face="Calibri, Arial, Helvetica, sans-serif" style="color:rgb(0,0,0); font-family:Calibri,Arial,Helvetica,sans-serif; font-size:inherit; font-style:inherit; font-variant-ligatures:inherit; font-variant-caps:inherit; font-weight:inherit"><br>
</font></div>
<div style="margin:0px"><font color="#000000" face="Calibri, Arial, Helvetica, sans-serif" style="color:rgb(0,0,0); font-family:Calibri,Arial,Helvetica,sans-serif; font-size:inherit; font-style:inherit; font-variant-ligatures:inherit; font-variant-caps:inherit; font-weight:inherit">Many
Thanks!!!</font></div>
<div style="margin:0px"><font color="#000000" face="Calibri, Arial, Helvetica, sans-serif" style="color:rgb(0,0,0); font-family:Calibri,Arial,Helvetica,sans-serif; font-size:inherit; font-style:inherit; font-variant-ligatures:inherit; font-variant-caps:inherit; font-weight:inherit"><br>
</font></div>
<div style="margin:0px"><font color="#000000" face="Calibri, Arial, Helvetica, sans-serif" style="color:rgb(0,0,0); font-family:Calibri,Arial,Helvetica,sans-serif; font-size:inherit; font-style:inherit; font-variant-ligatures:inherit; font-variant-caps:inherit; font-weight:inherit"><br>
</font></div>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style=""><br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style=""><br>
</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="">Reagards</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<span style="">Fulong</span></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="x_divRplyFwdMsg" dir="ltr"><font color="#000000" face="Calibri, sans-serif" style="font-size:11pt"><b>From:</b> Gao,Yan
<a class="x_moz-txt-link-rfc2396E" href="mailto:ygao@suse.com"><ygao@suse.com></a><br>
<b>Sent:</b> Thursday, January 3, 2019 20:43<br>
<b>To:</b> Fulong Wang; Cluster Labs - All topics related to open-source clustering welcomed<br>
<b>Subject:</b> Re: [ClusterLabs] SuSE12SP3 HAE SBD Communication Issue</font>
<div> </div>
</div>
<div class="x_BodyFragment"><font size="2"><span style="font-size:11pt">
<div class="x_PlainText">On 12/24/18 7:10 AM, Fulong Wang wrote:<br>
> Yan, klaus and Everyone,<br>
> <br>
> <br>
> Merry Christmas!!!<br>
> <br>
> <br>
> <br>
> Many thanks for your advice!<br>
> I added the "-v" param in "SBD_OPTS", but didn't see any apparent change <br>
> in the system message log, am i looking at a wrong place?<br>
Did you restart all cluster services, for example by "crm cluster stop" <br>
and then "crm cluster start"? Basically sbd.service needs to be <br>
restarted. Be aware "systemctl restart pacemaker" only restarts pacemaker.<br>
<br>
SBD daemons log into syslog. When a sbd watcher receives a "test" <br>
command, there should be a syslog like this showing up:<br>
<br>
"servant: Received command test from ..."<br>
<br>
sbd won't actually do anything about a "test" command but logging a message.<br>
<br>
If you are not running a late version of sbd (maintenance update) yet, a <br>
single "-v" will make sbd too verbose already. But of course you could <br>
use grep.<br>
<br>
> <br>
> By the way, we want to test when the disk access paths (multipath <br>
> devices) lost, the sbd can fence the node automatically.<br>
Be aware since pacemaker integration (-P) is enabled by default, which <br>
means despite the sbd failure, if the node itself was clean and <br>
"healthy" from pacemaker's point of view and if it's in the cluster <br>
partition with the quorum, it wouldn't self-fence -- meaning a node just <br>
being unable to fence doesn't necessarily need to be fenced.<br>
<br>
As described in sbd man page, "this allows sbd to survive temporary <br>
outages of the majority of devices. However, while the cluster is in <br>
such a degraded state, it can neither successfully fence nor be shutdown <br>
cleanly (as taking the cluster below the quorum threshold will <br>
immediately cause all remaining nodes to self-fence). In short, it will <br>
not tolerate any further faults. Please repair the system before <br>
continuing."<br>
<br>
Regards,<br>
Yan<br>
<br>
<br>
> what's your recommendation for this scenario?<br>
> <br>
> <br>
> <br>
> <br>
> <br>
> <br>
> <br>
> The "crm node fence" did the work.<br>
> <br>
> <br>
> <br>
> <br>
> <br>
> <br>
> <br>
> <br>
> <br>
> <br>
> <br>
> <br>
> <br>
> Regards<br>
> Fulong<br>
> <br>
> ------------------------------------------------------------------------<br>
> *From:* Gao,Yan <a class="x_moz-txt-link-rfc2396E" href="mailto:ygao@suse.com">
<ygao@suse.com></a><br>
> *Sent:* Friday, December 21, 2018 20:43<br>
> *To:* <a class="x_moz-txt-link-abbreviated" href="mailto:kwenning@redhat.com">kwenning@redhat.com</a>; Cluster Labs - All topics related to
<br>
> open-source clustering welcomed; Fulong Wang<br>
> *Subject:* Re: [ClusterLabs] SuSE12SP3 HAE SBD Communication Issue<br>
> First thanks for your reply, Klaus!<br>
> <br>
> On 2018/12/21 10:09, Klaus Wenninger wrote:<br>
>> On 12/21/2018 08:15 AM, Fulong Wang wrote:<br>
>>> Hello Experts,<br>
>>><br>
>>> I'm New to this mail lists.<br>
>>> Pls kindlyforgive me if this mail has disturb you!<br>
>>><br>
>>> Our Company recently is evaluating the usage of the SuSE HAE on x86 <br>
>>> platform.<br>
>>> Wen simulating the storage disaster fail-over, i finally found that <br>
>>> the SBD communication functioned normal on SuSE11 SP4 but abnormal on <br>
>>> SuSE12 SP3.<br>
>> <br>
>> I have no experience with SBD on SLES but I know that handling of the<br>
>> logging verbosity-levels has changed recently in the upstream-repo.<br>
>> Given that it was done by Yan Gao iirc I'd assume it went into SLES.<br>
>> So changing the verbosity of the sbd-daemon might get you back<br>
>> these logs.<br>
> Yes, I think it's the issue. Could you please retrieve the latest<br>
> maintenance update for SLE12SP3 and try? Otherwise of course you could<br>
> temporarily enable verbose/debug logging by adding a couple of "-v" into<br>
> "SBD_OPTS" in /etc/sysconfig/sbd.<br>
> <br>
> But frankly, it makes more sense to manually trigger fencing for example<br>
> by "crm node fence" and see if it indeed works correctly.<br>
> <br>
>> And of course you can use the list command on the other node<br>
>> to verify as well.<br>
> The "test" message in the slot might get overwritten soon by a "clear"<br>
> if the sbd daemon is running.<br>
> <br>
> Regards,<br>
> Yan<br>
> <br>
> <br>
>> <br>
>> Klaus<br>
>> <br>
>>> The SBD device was added during the initialization of the first <br>
>>> cluster node.<br>
>>><br>
>>> I have requested help from SuSE guys, but they didn't give me any <br>
>>> valuable feedback yet now!<br>
>>><br>
>>><br>
>>> Below are some screenshots to explain what i have encountered.<br>
>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~<br>
>>><br>
>>> on a SuSE11 SP4 HAE cluster, i run the sbd test command as below:<br>
>>><br>
>>><br>
>>> then there will be some information showed up in the local system <br>
>>> message log<br>
>>><br>
>>><br>
>>><br>
>>> on the second node, we can found that the communication is normal by<br>
>>><br>
>>><br>
>>><br>
>>> but when i turn to a SuSE12 SP3 HAE cluster, ran the same command as <br>
>>> above:<br>
>>><br>
>>><br>
>>><br>
>>> I didn't get any response in the system message log.<br>
>>><br>
>>><br>
>>> "systemctl status sbd" also doesn't give me any clue on this.<br>
>>><br>
>>><br>
>>><br>
>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~<br>
>>><br>
>>> What could be the reason for this abnormal behavior? Is there any <br>
>>> problems with my setup?<br>
>>> Any suggestions are appreciate!<br>
>>><br>
>>> Thanks!<br>
>>><br>
>>><br>
>>> Regards<br>
>>> FuLong<br>
>>><br>
>>><br>
>>> _______________________________________________<br>
>>> Users mailing <a class="x_moz-txt-link-abbreviated" href="mailto:list:Users@clusterlabs.org">
list:Users@clusterlabs.org</a><br>
>>> <a href="https://lists.clusterlabs.org/mailman/listinfo/users">https://lists.clusterlabs.org/mailman/listinfo/users</a><br>
>>><br>
>>> Project Home:<a class="x_moz-txt-link-freetext" href="http://www.clusterlabs.org">http://www.clusterlabs.org</a><br>
>>> Getting started:<a class="x_moz-txt-link-freetext" href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
>>> Bugs:<a class="x_moz-txt-link-freetext" href="http://bugs.clusterlabs.org">http://bugs.clusterlabs.org</a><br>
>> <br>
>> <br>
>> <br>
>> _______________________________________________<br>
>> Users mailing list: <a class="x_moz-txt-link-abbreviated" href="mailto:Users@clusterlabs.org">
Users@clusterlabs.org</a><br>
>> <a href="https://lists.clusterlabs.org/mailman/listinfo/users">https://lists.clusterlabs.org/mailman/listinfo/users</a><br>
>> <br>
>> Project Home: <a href="http://www.clusterlabs.org">http://www.clusterlabs.org</a><br>
>> Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf">
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
>> Bugs: <a href="http://bugs.clusterlabs.org">http://bugs.clusterlabs.org</a><br>
>> <br>
</div>
</span></font></div>
<br>
<fieldset class="x_mimeAttachmentHeader"></fieldset> <br>
<pre>_______________________________________________
Users mailing list: <a class="x_moz-txt-link-abbreviated" href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a>
<a class="x_moz-txt-link-freetext" href="https://lists.clusterlabs.org/mailman/listinfo/users">https://lists.clusterlabs.org/mailman/listinfo/users</a>
Project Home: <a class="x_moz-txt-link-freetext" href="http://www.clusterlabs.org">http://www.clusterlabs.org</a>
Getting started: <a class="x_moz-txt-link-freetext" href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a>
Bugs: <a class="x_moz-txt-link-freetext" href="http://bugs.clusterlabs.org">http://bugs.clusterlabs.org</a>
</pre>
</blockquote>
<br>
</div>
</body>
</html>