<div dir="ltr">Thanks Aswathi. <div><br></div><div>(My account had stopped working due to mail bounces, never seen that occur on gmail accounts)<div><br></div><div>Ken,</div><div><br></div><div>Answers to your questions are below:</div><div><br></div><div><i>1. Using force option</i></div></div><div>A) During our testing we had observed that in some instances the resource deletion would fail and that's why we added the force option. With the force option we never saw the problem again.</div><div><br><div><i>2. "Maybe in this particular instance, you actually did "crm_resource -C"?"</i></div><div>A) This step is done through code so there is no human involvement. We are printing the full command and we always see resource name is included. So this cannot happen.</div><div><br></div><div><i>3. $ crm_node -R 0005B94238BC --force</i></div><div>A) Yes, we want to remove the node completely. We are not specifying the node information in corosync.conf so there is nothing to be removed there. </div><div>I need to go back and check but I vaguely remember that because of some issue we had switched from using "pcs cluster node remove" command to crm_node -R command. Perhaps because it gave us the option to use force. </div><div><br></div><div><i>4. "No STONITH and QUORUM"</i></div><div>A) As I have mentioned earlier, split-brain doesn't pose a problem for us since we have a second line of defense based on our architecture. Hence we have made a conscious decision to disable it. The config IS for production.</div><div><div class="gmail_extra"><br></div><div class="gmail_extra">BTW, we also issue a "<span style="color:rgb(0,0,0);font-family:"Segoe UI";font-size:10pt">pcs resource disable</span>" command before doing a "<span style="color:rgb(0,0,0);font-family:"Segoe UI";font-size:10pt">pcs resource delete</span>". Not sure if that makes any difference.</div><div class="gmail_extra"><br></div><div class="gmail_extra">We will play around with those 4-5 commands that we execute to see whether the resource restart happens as a reaction to any of those command.</div><div class="gmail_extra"><br></div><div class="gmail_extra">-Thanks & Regards</div><div class="gmail_extra">Nikhil</div><div class="gmail_extra"><br><div class="gmail_quote">On Wed, May 24, 2017 at 11:28 AM, Anu Pillai <span dir="ltr"><<a href="mailto:anu.pillai.subscrib@gmail.com" target="_blank">anu.pillai.subscrib@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir="ltr"><div id="gmail-m_5030438798138421009gmail-imcontent"><span style="word-wrap:break-word;font-size:10pt;font-family:"Segoe UI";color:rgb(0,0,0);direction:ltr">blank response for thread to appear in mailbox..pls ignore</span></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, May 23, 2017 at 4:21 AM, Ken Gaillot <span dir="ltr"><<a href="mailto:kgaillot@redhat.com" target="_blank">kgaillot@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><span>On 05/16/2017 04:34 AM, Anu Pillai wrote:<br>
> Hi,<br>
><br>
</span><span>> Please find attached debug logs for the stated problem as well as<br>
> crm_mon command outputs.<br>
> In this case we are trying to remove/delete res3 and system/node<br>
> (0005B94238BC) from the cluster.<br>
><br>
</span>> *_Test reproduction steps_*<br>
<span>><br>
> Current Configuration of the cluster:<br>
> 0005B9423910 - res2<br>
> 0005B9427C5A - res1<br>
> 0005B94238BC - res3<br>
><br>
</span>> *crm_mon output:*<br>
<span>><br>
> Defaulting to one-shot mode<br>
> You need to have curses available at compile time to enable console mode<br>
> Stack: corosync<br>
> Current DC: 0005B9423910 (version 1.1.14-5a6cdd1) - partition with quorum<br>
> Last updated: Tue May 16 12:21:23 2017 Last change: Tue May 16<br>
> 12:13:40 2017 by root via crm_attribute on 0005B9423910<br>
><br>
> 3 nodes and 3 resources configured<br>
><br>
> Online: [ 0005B94238BC 0005B9423910 0005B9427C5A ]<br>
><br>
> res2 (ocf::redundancy:<wbr>RedundancyRA): Started 0005B9423910<br>
> res1 (ocf::redundancy:<wbr>RedundancyRA): Started 0005B9427C5A<br>
> res3 (ocf::redundancy:<wbr>RedundancyRA): Started 0005B94238BC<br>
><br>
><br>
> Trigger the delete operation for res3 and node 0005B94238BC.<br>
><br>
> Following commands applied from node 0005B94238BC<br>
> $ pcs resource delete res3 --force<br>
> $ crm_resource -C res3<br>
> $ pcs cluster stop --force<br>
<br>
</span>I don't think "pcs resource delete" or "pcs cluster stop" does anything<br>
with the --force option. In any case, --force shouldn't be needed here.<br>
<br>
The crm_mon output you see is actually not what it appears. It starts with:<br>
<br>
May 16 12:21:27 [4661] 0005B9423910 crmd: notice: do_lrm_invoke:<br>
Forcing the status of all resources to be redetected<br>
<br>
This is usually the result of a "cleanup all" command. It works by<br>
erasing the resource history, causing pacemaker to re-probe all nodes to<br>
get the current state. The history erasure makes it appear to crm_mon<br>
that the resources are stopped, but they actually are not.<br>
<br>
In this case, I'm not sure why it's doing a "cleanup all", since you<br>
only asked it to cleanup res3. Maybe in this particular instance, you<br>
actually did "crm_resource -C"?<br>
<span><br>
> Following command applied from DC(0005B9423910)<br>
> $ crm_node -R 0005B94238BC --force<br>
<br>
</span>This can cause problems. This command shouldn't be run unless the node<br>
is removed from both pacemaker's and corosync's configuration. If you<br>
actually are trying to remove the node completely, a better alternative<br>
would be "pcs cluster node remove 0005B94238BC", which will handle all<br>
of that for you. If you're not trying to remove the node completely,<br>
then you shouldn't need this command at all.<br>
<span class="gmail-m_5030438798138421009im gmail-m_5030438798138421009HOEnZb"><br>
><br>
><br>
> *crm_mon output:*<br>
> *<br>
> *<br>
</span><span class="gmail-m_5030438798138421009im gmail-m_5030438798138421009HOEnZb">> Defaulting to one-shot mode<br>
> You need to have curses available at compile time to enable console mode<br>
> Stack: corosync<br>
> Current DC: 0005B9423910 (version 1.1.14-5a6cdd1) - partition with quorum<br>
> Last updated: Tue May 16 12:21:27 2017 Last change: Tue May 16<br>
> 12:21:26 2017 by root via cibadmin on 0005B94238BC<br>
><br>
> 3 nodes and 2 resources configured<br>
><br>
> Online: [ 0005B94238BC 0005B9423910 0005B9427C5A ]<br>
><br>
><br>
> Observation is remaining two resources res2 and res1 were stopped and<br>
> started.<br>
><br>
><br>
> Regards,<br>
> Aswathi<br>
><br>
> On Mon, May 15, 2017 at 8:11 PM, Ken Gaillot <<a href="mailto:kgaillot@redhat.com" target="_blank">kgaillot@redhat.com</a><br>
</span><div class="gmail-m_5030438798138421009HOEnZb"><div class="gmail-m_5030438798138421009h5">> <mailto:<a href="mailto:kgaillot@redhat.com" target="_blank">kgaillot@redhat.com</a>>> wrote:<br>
><br>
> On 05/15/2017 06:59 AM, Klaus Wenninger wrote:<br>
> > On 05/15/2017 12:25 PM, Anu Pillai wrote:<br>
> >> Hi Klaus,<br>
> >><br>
> >> Please find attached cib.xml as well as corosync.conf.<br>
><br>
> Maybe you're only setting this while testing, but having<br>
> stonith-enabled=false and no-quorum-policy=ignore is highly dangerous in<br>
> any kind of network split.<br>
><br>
> FYI, default-action-timeout is deprecated in favor of setting a timeout<br>
> in op_defaults, but it doesn't hurt anything.<br>
><br>
> > Why wouldn't you keep placement-strategy with default<br>
> > to keep things simple. You aren't using any load-balancing<br>
> > anyway as far as I understood it.<br>
><br>
> It looks like the intent is to use placement-strategy to limit each node<br>
> to 1 resource. The configuration looks good for that.<br>
><br>
> > Haven't used resource-stickiness=INF. No idea which strange<br>
> > behavior that triggers. Try to have it just higher than what<br>
> > the other scores might some up to.<br>
><br>
> Either way would be fine. Using INFINITY ensures that no other<br>
> combination of scores will override it.<br>
><br>
> > I might have overseen something in your scores but otherwise<br>
> > there is nothing obvious to me.<br>
> ><br>
> > Regards,<br>
> > Klaus<br>
><br>
> I don't see anything obvious either. If you have logs around the time of<br>
> the incident, that might help.<br>
><br>
> >> Regards,<br>
> >> Aswathi<br>
> >><br>
> >> On Mon, May 15, 2017 at 2:46 PM, Klaus Wenninger <<a href="mailto:kwenning@redhat.com" target="_blank">kwenning@redhat.com</a> <mailto:<a href="mailto:kwenning@redhat.com" target="_blank">kwenning@redhat.com</a>><br>
</div></div><div class="gmail-m_5030438798138421009HOEnZb"><div class="gmail-m_5030438798138421009h5">> >> <mailto:<a href="mailto:kwenning@redhat.com" target="_blank">kwenning@redhat.com</a> <mailto:<a href="mailto:kwenning@redhat.com" target="_blank">kwenning@redhat.com</a>>>> wrote:<br>
> >><br>
> >> On 05/15/2017 09:36 AM, Anu Pillai wrote:<br>
> >> > Hi,<br>
> >> ><br>
> >> > We are running pacemaker cluster for managing our resources. We<br>
> >> have 6<br>
> >> > system running 5 resources and one is acting as standby. We have a<br>
> >> > restriction that, only one resource can run in one node. But our<br>
> >> > observation is whenever we add or delete a resource from cluster all<br>
> >> > the remaining resources in the cluster are stopped and started back.<br>
> >> ><br>
> >> > Can you please guide us whether this normal behavior or we are<br>
> >> missing<br>
> >> > any configuration that is leading to this issue.<br>
> >><br>
> >> It should definitely be possible to prevent this behavior.<br>
> >> If you share your config with us we might be able to<br>
> >> track that down.<br>
> >><br>
> >> Regards,<br>
> >> Klaus<br>
> >><br>
> >> ><br>
> >> > Regards<br>
> >> > Aswathi<br>
</div></div></blockquote></div><br></div>
<br>______________________________<wbr>_________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a><br>
<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/<wbr>doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
<br></blockquote></div><br></div></div></div></div>