<div dir="ltr"><div id="gmail-imcontent"><span style="word-wrap:break-word;font-size:10pt;font-family:"Segoe UI";color:rgb(0,0,0);direction:ltr">blank response for thread to appear in mailbox..pls ignore</span></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Tue, May 23, 2017 at 4:21 AM, Ken Gaillot <span dir="ltr"><<a href="mailto:kgaillot@redhat.com" target="_blank">kgaillot@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">On 05/16/2017 04:34 AM, Anu Pillai wrote:<br>
> Hi,<br>
><br>
</span><span class="">> Please find attached debug logs for the stated problem as well as<br>
> crm_mon command outputs.<br>
> In this case we are trying to remove/delete res3 and system/node<br>
> (0005B94238BC) from the cluster.<br>
><br>
</span>> *_Test reproduction steps_*<br>
<span class="">><br>
> Current Configuration of the cluster:<br>
> 0005B9423910 - res2<br>
> 0005B9427C5A - res1<br>
> 0005B94238BC - res3<br>
><br>
</span>> *crm_mon output:*<br>
<span class="">><br>
> Defaulting to one-shot mode<br>
> You need to have curses available at compile time to enable console mode<br>
> Stack: corosync<br>
> Current DC: 0005B9423910 (version 1.1.14-5a6cdd1) - partition with quorum<br>
> Last updated: Tue May 16 12:21:23 2017 Last change: Tue May 16<br>
> 12:13:40 2017 by root via crm_attribute on 0005B9423910<br>
><br>
> 3 nodes and 3 resources configured<br>
><br>
> Online: [ 0005B94238BC 0005B9423910 0005B9427C5A ]<br>
><br>
> res2 (ocf::redundancy:RedundancyRA)<wbr>: Started 0005B9423910<br>
> res1 (ocf::redundancy:RedundancyRA)<wbr>: Started 0005B9427C5A<br>
> res3 (ocf::redundancy:RedundancyRA)<wbr>: Started 0005B94238BC<br>
><br>
><br>
> Trigger the delete operation for res3 and node 0005B94238BC.<br>
><br>
> Following commands applied from node 0005B94238BC<br>
> $ pcs resource delete res3 --force<br>
> $ crm_resource -C res3<br>
> $ pcs cluster stop --force<br>
<br>
</span>I don't think "pcs resource delete" or "pcs cluster stop" does anything<br>
with the --force option. In any case, --force shouldn't be needed here.<br>
<br>
The crm_mon output you see is actually not what it appears. It starts with:<br>
<br>
May 16 12:21:27 [4661] 0005B9423910 crmd: notice: do_lrm_invoke:<br>
Forcing the status of all resources to be redetected<br>
<br>
This is usually the result of a "cleanup all" command. It works by<br>
erasing the resource history, causing pacemaker to re-probe all nodes to<br>
get the current state. The history erasure makes it appear to crm_mon<br>
that the resources are stopped, but they actually are not.<br>
<br>
In this case, I'm not sure why it's doing a "cleanup all", since you<br>
only asked it to cleanup res3. Maybe in this particular instance, you<br>
actually did "crm_resource -C"?<br>
<span class=""><br>
> Following command applied from DC(0005B9423910)<br>
> $ crm_node -R 0005B94238BC --force<br>
<br>
</span>This can cause problems. This command shouldn't be run unless the node<br>
is removed from both pacemaker's and corosync's configuration. If you<br>
actually are trying to remove the node completely, a better alternative<br>
would be "pcs cluster node remove 0005B94238BC", which will handle all<br>
of that for you. If you're not trying to remove the node completely,<br>
then you shouldn't need this command at all.<br>
<span class="im HOEnZb"><br>
><br>
><br>
> *crm_mon output:*<br>
> *<br>
> *<br>
</span><span class="im HOEnZb">> Defaulting to one-shot mode<br>
> You need to have curses available at compile time to enable console mode<br>
> Stack: corosync<br>
> Current DC: 0005B9423910 (version 1.1.14-5a6cdd1) - partition with quorum<br>
> Last updated: Tue May 16 12:21:27 2017 Last change: Tue May 16<br>
> 12:21:26 2017 by root via cibadmin on 0005B94238BC<br>
><br>
> 3 nodes and 2 resources configured<br>
><br>
> Online: [ 0005B94238BC 0005B9423910 0005B9427C5A ]<br>
><br>
><br>
> Observation is remaining two resources res2 and res1 were stopped and<br>
> started.<br>
><br>
><br>
> Regards,<br>
> Aswathi<br>
><br>
> On Mon, May 15, 2017 at 8:11 PM, Ken Gaillot <<a href="mailto:kgaillot@redhat.com">kgaillot@redhat.com</a><br>
</span><div class="HOEnZb"><div class="h5">> <mailto:<a href="mailto:kgaillot@redhat.com">kgaillot@redhat.com</a>>> wrote:<br>
><br>
> On 05/15/2017 06:59 AM, Klaus Wenninger wrote:<br>
> > On 05/15/2017 12:25 PM, Anu Pillai wrote:<br>
> >> Hi Klaus,<br>
> >><br>
> >> Please find attached cib.xml as well as corosync.conf.<br>
><br>
> Maybe you're only setting this while testing, but having<br>
> stonith-enabled=false and no-quorum-policy=ignore is highly dangerous in<br>
> any kind of network split.<br>
><br>
> FYI, default-action-timeout is deprecated in favor of setting a timeout<br>
> in op_defaults, but it doesn't hurt anything.<br>
><br>
> > Why wouldn't you keep placement-strategy with default<br>
> > to keep things simple. You aren't using any load-balancing<br>
> > anyway as far as I understood it.<br>
><br>
> It looks like the intent is to use placement-strategy to limit each node<br>
> to 1 resource. The configuration looks good for that.<br>
><br>
> > Haven't used resource-stickiness=INF. No idea which strange<br>
> > behavior that triggers. Try to have it just higher than what<br>
> > the other scores might some up to.<br>
><br>
> Either way would be fine. Using INFINITY ensures that no other<br>
> combination of scores will override it.<br>
><br>
> > I might have overseen something in your scores but otherwise<br>
> > there is nothing obvious to me.<br>
> ><br>
> > Regards,<br>
> > Klaus<br>
><br>
> I don't see anything obvious either. If you have logs around the time of<br>
> the incident, that might help.<br>
><br>
> >> Regards,<br>
> >> Aswathi<br>
> >><br>
> >> On Mon, May 15, 2017 at 2:46 PM, Klaus Wenninger <<a href="mailto:kwenning@redhat.com">kwenning@redhat.com</a> <mailto:<a href="mailto:kwenning@redhat.com">kwenning@redhat.com</a>><br>
</div></div><div class="HOEnZb"><div class="h5">> >> <mailto:<a href="mailto:kwenning@redhat.com">kwenning@redhat.com</a> <mailto:<a href="mailto:kwenning@redhat.com">kwenning@redhat.com</a>>>> wrote:<br>
> >><br>
> >> On 05/15/2017 09:36 AM, Anu Pillai wrote:<br>
> >> > Hi,<br>
> >> ><br>
> >> > We are running pacemaker cluster for managing our resources. We<br>
> >> have 6<br>
> >> > system running 5 resources and one is acting as standby. We have a<br>
> >> > restriction that, only one resource can run in one node. But our<br>
> >> > observation is whenever we add or delete a resource from cluster all<br>
> >> > the remaining resources in the cluster are stopped and started back.<br>
> >> ><br>
> >> > Can you please guide us whether this normal behavior or we are<br>
> >> missing<br>
> >> > any configuration that is leading to this issue.<br>
> >><br>
> >> It should definitely be possible to prevent this behavior.<br>
> >> If you share your config with us we might be able to<br>
> >> track that down.<br>
> >><br>
> >> Regards,<br>
> >> Klaus<br>
> >><br>
> >> ><br>
> >> > Regards<br>
> >> > Aswathi<br>
</div></div></blockquote></div><br></div>