<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style type="text/css" style="display:none;"> P {margin-top:0;margin-bottom:0;} </style>
</head>
<body dir="ltr">
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
Thanks Yan,</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<div style="margin: 0px; font-size: 12pt; font-family: Calibri, Arial, Helvetica, sans-serif">
You gave me more valuable hints on the SBD operation!</div>
<div style="margin: 0px; font-size: 12pt; font-family: Calibri, Arial, Helvetica, sans-serif">
Now, i can see the verbose output after service restart.</div>
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(51, 51, 51); font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; background-color: rgb(255, 255, 255); display: inline !important">>Be
aware since pacemaker integration (-P) is enabled by default, which<span> </span></span><br style="color: rgb(51, 51, 51); font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; background-color: rgb(255, 255, 255)">
<span style="color: rgb(51, 51, 51); font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; background-color: rgb(255, 255, 255); display: inline !important">>means
despite the sbd failure, if the node itself was clean and<span> </span></span><br style="color: rgb(51, 51, 51); font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; background-color: rgb(255, 255, 255)">
<span style="color: rgb(51, 51, 51); font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; background-color: rgb(255, 255, 255); display: inline !important">>"healthy"
from pacemaker's point of view and if it's in the cluster<span> </span></span><br style="color: rgb(51, 51, 51); font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; background-color: rgb(255, 255, 255)">
<span style="color: rgb(51, 51, 51); font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; background-color: rgb(255, 255, 255); display: inline !important">>partition
with the quorum, it wouldn't self-fence -- meaning a node just<span> </span></span><br style="color: rgb(51, 51, 51); font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; background-color: rgb(255, 255, 255)">
<span style="color: rgb(51, 51, 51); font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; background-color: rgb(255, 255, 255); display: inline !important">>being
unable to fence doesn't necessarily need to be fenced.</span><br style="color: rgb(51, 51, 51); font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; background-color: rgb(255, 255, 255)">
<br style="color: rgb(51, 51, 51); font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; background-color: rgb(255, 255, 255)">
<span style="color: rgb(51, 51, 51); font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; background-color: rgb(255, 255, 255); display: inline !important">>As
described in sbd man page, "this allows sbd to survive temporary<span> </span></span><br style="color: rgb(51, 51, 51); font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; background-color: rgb(255, 255, 255)">
<span style="color: rgb(51, 51, 51); font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; background-color: rgb(255, 255, 255); display: inline !important">>outages
of the majority of devices. However, while the cluster is in<span> </span></span><br style="color: rgb(51, 51, 51); font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; background-color: rgb(255, 255, 255)">
<span style="color: rgb(51, 51, 51); font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; background-color: rgb(255, 255, 255); display: inline !important">>such
a degraded state, it can neither successfully fence nor be shutdown<span> </span></span><br style="color: rgb(51, 51, 51); font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; background-color: rgb(255, 255, 255)">
<span style="color: rgb(51, 51, 51); font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; background-color: rgb(255, 255, 255); display: inline !important">>cleanly
(as taking the cluster below the quorum threshold will<span> </span></span><br style="color: rgb(51, 51, 51); font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; background-color: rgb(255, 255, 255)">
<span style="color: rgb(51, 51, 51); font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; background-color: rgb(255, 255, 255); display: inline !important">>immediately
cause all remaining nodes to self-fence). In short, it will<span> </span></span><br style="color: rgb(51, 51, 51); font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; background-color: rgb(255, 255, 255)">
<span style="color: rgb(51, 51, 51); font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; background-color: rgb(255, 255, 255); display: inline !important">>not
tolerate any further faults. Please repair the system before<span> </span></span><br style="color: rgb(51, 51, 51); font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; background-color: rgb(255, 255, 255)">
<span style="color: rgb(51, 51, 51); font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; background-color: rgb(255, 255, 255); display: inline !important">>continuing."</span><br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(51, 51, 51); font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; background-color: rgb(255, 255, 255); display: inline !important"><br>
</span></div>
<div style=""><span style="background-color: rgb(255, 255, 255); display: inline !important;">
<div style="margin: 0px;"><font color="#000000" face="Calibri, Arial, Helvetica, sans-serif" style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; font-size: inherit; font-style: inherit; font-variant-ligatures: inherit; font-variant-caps: inherit; font-weight: inherit;">Yes,
I can see the "<span style="color: rgb(51, 51, 51); font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px;">pacemaker integration</span>" was enabled
in my sbd config file by default.</font></div>
<div style="margin: 0px;"><font color="#000000" face="Calibri, Arial, Helvetica, sans-serif" style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; font-size: inherit; font-style: inherit; font-variant-ligatures: inherit; font-variant-caps: inherit; font-weight: inherit;">So,
you mean in some sbd failure cases, if the node was considered as "healthy" from pacemaker's poinit of view, it still wouldn't sel-fence. </font></div>
<div style="margin: 0px;"><font color="#000000" face="Calibri, Arial, Helvetica, sans-serif" style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; font-size: inherit; font-style: inherit; font-variant-ligatures: inherit; font-variant-caps: inherit; font-weight: inherit;"><br>
</font></div>
<div style="margin: 0px;"><font color="#000000" face="Calibri, Arial, Helvetica, sans-serif" style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; font-size: inherit; font-style: inherit; font-variant-ligatures: inherit; font-variant-caps: inherit; font-weight: inherit;">H</font><span style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif;">onestly
speaking, i didn't get you at this point. </span><span style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif;">I have "</span><span style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif;">no-quorum-policy=ignore"
setting in my setup and it's a two node cluster. </span></div>
<div style="margin: 0px;"><font color="#000000" face="Calibri, Arial, Helvetica, sans-serif" style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; font-size: inherit; font-style: inherit; font-variant-ligatures: inherit; font-variant-caps: inherit; font-weight: inherit;">Can
you show me a sample situation for this?</font><br>
</div>
<div style="margin: 0px;"><font color="#000000" face="Calibri, Arial, Helvetica, sans-serif" style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; font-size: inherit; font-style: inherit; font-variant-ligatures: inherit; font-variant-caps: inherit; font-weight: inherit;"><br>
</font></div>
<div style="margin: 0px;"><font color="#000000" face="Calibri, Arial, Helvetica, sans-serif" style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; font-size: inherit; font-style: inherit; font-variant-ligatures: inherit; font-variant-caps: inherit; font-weight: inherit;">Many
Thanks!!!</font></div>
<div style="margin: 0px;"><font color="#000000" face="Calibri, Arial, Helvetica, sans-serif" style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; font-size: inherit; font-style: inherit; font-variant-ligatures: inherit; font-variant-caps: inherit; font-weight: inherit;"><br>
</font></div>
<div style="margin: 0px;"><font color="#000000" face="Calibri, Arial, Helvetica, sans-serif" style="color: rgb(0, 0, 0); font-family: Calibri, Arial, Helvetica, sans-serif; font-size: inherit; font-style: inherit; font-variant-ligatures: inherit; font-variant-caps: inherit; font-weight: inherit;"><br>
</font></div>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(51, 51, 51); font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; background-color: rgb(255, 255, 255); display: inline !important"><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(51, 51, 51); font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; background-color: rgb(255, 255, 255); display: inline !important"><br>
</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(51, 51, 51); font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; background-color: rgb(255, 255, 255); display: inline !important">Reagards</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<span style="color: rgb(51, 51, 51); font-family: "Segoe UI", "Segoe UI Web (West European)", "Segoe UI", -apple-system, BlinkMacSystemFont, Roboto, "Helvetica Neue", sans-serif; font-size: 14.6667px; background-color: rgb(255, 255, 255); display: inline !important">Fulong</span></div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div style="font-family: Calibri, Arial, Helvetica, sans-serif; font-size: 12pt; color: rgb(0, 0, 0);">
<br>
</div>
<div id="appendonsend"></div>
<div style="font-family:Calibri,Arial,Helvetica,sans-serif; font-size:12pt; color:rgb(0,0,0)">
<br>
</div>
<hr tabindex="-1" style="display:inline-block; width:98%">
<div id="divRplyFwdMsg" dir="ltr"><font face="Calibri, sans-serif" color="#000000" style="font-size:11pt"><b>From:</b> Gao,Yan <ygao@suse.com><br>
<b>Sent:</b> Thursday, January 3, 2019 20:43<br>
<b>To:</b> Fulong Wang; Cluster Labs - All topics related to open-source clustering welcomed<br>
<b>Subject:</b> Re: [ClusterLabs] SuSE12SP3 HAE SBD Communication Issue</font>
<div> </div>
</div>
<div class="BodyFragment"><font size="2"><span style="font-size:11pt">
<div class="PlainText">On 12/24/18 7:10 AM, Fulong Wang wrote:<br>
> Yan, klaus and Everyone,<br>
> <br>
> <br>
> Merry Christmas!!!<br>
> <br>
> <br>
> <br>
> Many thanks for your advice!<br>
> I added the "-v" param in "SBD_OPTS", but didn't see any apparent change <br>
> in the system message log, am i looking at a wrong place?<br>
Did you restart all cluster services, for example by "crm cluster stop" <br>
and then "crm cluster start"? Basically sbd.service needs to be <br>
restarted. Be aware "systemctl restart pacemaker" only restarts pacemaker.<br>
<br>
SBD daemons log into syslog. When a sbd watcher receives a "test" <br>
command, there should be a syslog like this showing up:<br>
<br>
"servant: Received command test from ..."<br>
<br>
sbd won't actually do anything about a "test" command but logging a message.<br>
<br>
If you are not running a late version of sbd (maintenance update) yet, a <br>
single "-v" will make sbd too verbose already. But of course you could <br>
use grep.<br>
<br>
> <br>
> By the way, we want to test when the disk access paths (multipath <br>
> devices) lost, the sbd can fence the node automatically.<br>
Be aware since pacemaker integration (-P) is enabled by default, which <br>
means despite the sbd failure, if the node itself was clean and <br>
"healthy" from pacemaker's point of view and if it's in the cluster <br>
partition with the quorum, it wouldn't self-fence -- meaning a node just <br>
being unable to fence doesn't necessarily need to be fenced.<br>
<br>
As described in sbd man page, "this allows sbd to survive temporary <br>
outages of the majority of devices. However, while the cluster is in <br>
such a degraded state, it can neither successfully fence nor be shutdown <br>
cleanly (as taking the cluster below the quorum threshold will <br>
immediately cause all remaining nodes to self-fence). In short, it will <br>
not tolerate any further faults. Please repair the system before <br>
continuing."<br>
<br>
Regards,<br>
Yan<br>
<br>
<br>
> what's your recommendation for this scenario?<br>
> <br>
> <br>
> <br>
> <br>
> <br>
> <br>
> <br>
> The "crm node fence" did the work.<br>
> <br>
> <br>
> <br>
> <br>
> <br>
> <br>
> <br>
> <br>
> <br>
> <br>
> <br>
> <br>
> <br>
> Regards<br>
> Fulong<br>
> <br>
> ------------------------------------------------------------------------<br>
> *From:* Gao,Yan <ygao@suse.com><br>
> *Sent:* Friday, December 21, 2018 20:43<br>
> *To:* kwenning@redhat.com; Cluster Labs - All topics related to <br>
> open-source clustering welcomed; Fulong Wang<br>
> *Subject:* Re: [ClusterLabs] SuSE12SP3 HAE SBD Communication Issue<br>
> First thanks for your reply, Klaus!<br>
> <br>
> On 2018/12/21 10:09, Klaus Wenninger wrote:<br>
>> On 12/21/2018 08:15 AM, Fulong Wang wrote:<br>
>>> Hello Experts,<br>
>>><br>
>>> I'm New to this mail lists.<br>
>>> Pls kindlyforgive me if this mail has disturb you!<br>
>>><br>
>>> Our Company recently is evaluating the usage of the SuSE HAE on x86 <br>
>>> platform.<br>
>>> Wen simulating the storage disaster fail-over, i finally found that <br>
>>> the SBD communication functioned normal on SuSE11 SP4 but abnormal on <br>
>>> SuSE12 SP3.<br>
>> <br>
>> I have no experience with SBD on SLES but I know that handling of the<br>
>> logging verbosity-levels has changed recently in the upstream-repo.<br>
>> Given that it was done by Yan Gao iirc I'd assume it went into SLES.<br>
>> So changing the verbosity of the sbd-daemon might get you back<br>
>> these logs.<br>
> Yes, I think it's the issue. Could you please retrieve the latest<br>
> maintenance update for SLE12SP3 and try? Otherwise of course you could<br>
> temporarily enable verbose/debug logging by adding a couple of "-v" into<br>
> "SBD_OPTS" in /etc/sysconfig/sbd.<br>
> <br>
> But frankly, it makes more sense to manually trigger fencing for example<br>
> by "crm node fence" and see if it indeed works correctly.<br>
> <br>
>> And of course you can use the list command on the other node<br>
>> to verify as well.<br>
> The "test" message in the slot might get overwritten soon by a "clear"<br>
> if the sbd daemon is running.<br>
> <br>
> Regards,<br>
> Yan<br>
> <br>
> <br>
>> <br>
>> Klaus<br>
>> <br>
>>> The SBD device was added during the initialization of the first <br>
>>> cluster node.<br>
>>><br>
>>> I have requested help from SuSE guys, but they didn't give me any <br>
>>> valuable feedback yet now!<br>
>>><br>
>>><br>
>>> Below are some screenshots to explain what i have encountered.<br>
>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~<br>
>>><br>
>>> on a SuSE11 SP4 HAE cluster, i run the sbd test command as below:<br>
>>><br>
>>><br>
>>> then there will be some information showed up in the local system <br>
>>> message log<br>
>>><br>
>>><br>
>>><br>
>>> on the second node, we can found that the communication is normal by<br>
>>><br>
>>><br>
>>><br>
>>> but when i turn to a SuSE12 SP3 HAE cluster, ran the same command as <br>
>>> above:<br>
>>><br>
>>><br>
>>><br>
>>> I didn't get any response in the system message log.<br>
>>><br>
>>><br>
>>> "systemctl status sbd" also doesn't give me any clue on this.<br>
>>><br>
>>><br>
>>><br>
>>> ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~<br>
>>><br>
>>> What could be the reason for this abnormal behavior? Is there any <br>
>>> problems with my setup?<br>
>>> Any suggestions are appreciate!<br>
>>><br>
>>> Thanks!<br>
>>><br>
>>><br>
>>> Regards<br>
>>> FuLong<br>
>>><br>
>>><br>
>>> _______________________________________________<br>
>>> Users mailing list:Users@clusterlabs.org<br>
>>> <a href="https://lists.clusterlabs.org/mailman/listinfo/users">https://lists.clusterlabs.org/mailman/listinfo/users</a><br>
>>><br>
>>> Project Home:http://www.clusterlabs.org<br>
>>> Getting started:http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf<br>
>>> Bugs:http://bugs.clusterlabs.org<br>
>> <br>
>> <br>
>> <br>
>> _______________________________________________<br>
>> Users mailing list: Users@clusterlabs.org<br>
>> <a href="https://lists.clusterlabs.org/mailman/listinfo/users">https://lists.clusterlabs.org/mailman/listinfo/users</a><br>
>> <br>
>> Project Home: <a href="http://www.clusterlabs.org">http://www.clusterlabs.org</a><br>
>> Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf">
http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
>> Bugs: <a href="http://bugs.clusterlabs.org">http://bugs.clusterlabs.org</a><br>
>> <br>
</div>
</span></font></div>
</body>
</html>