<div dir="ltr">Andrew,<div>1. Is there a workaround for this issue ? </div><div>2. Also, can let me know if there are more issues with old versions in deleting multistate resource as mentioned in <a href="http://www.gossamer-threads.com/lists/linuxha/pacemaker/91230">http://www.gossamer-threads.com/lists/linuxha/pacemaker/91230</a></div>
<div><br></div><div>Regards,</div><div> Kiran</div></div><div class="gmail_extra"><br><br><div class="gmail_quote">On Wed, May 21, 2014 at 9:44 AM, Andrew Beekhof <span dir="ltr"><<a href="mailto:andrew@beekhof.net" target="_blank">andrew@beekhof.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5"><br>
On 19 May 2014, at 5:43 pm, K Mehta <<a href="mailto:kiranmehta1981@gmail.com">kiranmehta1981@gmail.com</a>> wrote:<br>
<br>
> Please see my reply inline. Attached is the crm_report output.<br>
><br>
><br>
> On Thu, May 8, 2014 at 5:45 AM, Andrew Beekhof <<a href="mailto:andrew@beekhof.net">andrew@beekhof.net</a>> wrote:<br>
><br>
> On 8 May 2014, at 12:38 am, K Mehta <<a href="mailto:kiranmehta1981@gmail.com">kiranmehta1981@gmail.com</a>> wrote:<br>
><br>
> > I created a multi state resource ms-2be6c088-a1fa-464a-b00d-f4bccb4f5af2 (vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2).<br>
> ><br>
> > Here is the configuration:<br>
> > ==========================<br>
> > [root@vsanqa11 ~]# pcs config<br>
> > Cluster Name: vsanqa11_12<br>
> > Corosync Nodes:<br>
> ><br>
> > Pacemaker Nodes:<br>
> > vsanqa11 vsanqa12<br>
> ><br>
> > Resources:<br>
> > Master: ms-2be6c088-a1fa-464a-b00d-f4bccb4f5af2<br>
> > Meta Attrs: clone-max=2 globally-unique=false target-role=started<br>
> > Resource: vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2 (class=ocf provider=heartbeat type=vgc-cm-agent.ocf)<br>
> > Attributes: cluster_uuid=2be6c088-a1fa-464a-b00d-f4bccb4f5af2<br>
> > Operations: monitor interval=30s role=Master timeout=100s (vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2-monitor-interval-30s)<br>
> > monitor interval=31s role=Slave timeout=100s (vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2-monitor-interval-31s)<br>
> ><br>
> > Stonith Devices:<br>
> > Fencing Levels:<br>
> ><br>
> > Location Constraints:<br>
> > Resource: ms-2be6c088-a1fa-464a-b00d-f4bccb4f5af2<br>
> > Enabled on: vsanqa11 (score:INFINITY) (id:location-ms-2be6c088-a1fa-464a-b00d-f4bccb4f5af2-vsanqa11-INFINITY)<br>
> > Enabled on: vsanqa12 (score:INFINITY) (id:location-ms-2be6c088-a1fa-464a-b00d-f4bccb4f5af2-vsanqa12-INFINITY)<br>
> > Resource: vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2<br>
> > Enabled on: vsanqa11 (score:INFINITY) (id:location-vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2-vsanqa11-INFINITY)<br>
> > Enabled on: vsanqa12 (score:INFINITY) (id:location-vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2-vsanqa12-INFINITY)<br>
> > Ordering Constraints:<br>
> > Colocation Constraints:<br>
> ><br>
> > Cluster Properties:<br>
> > cluster-infrastructure: cman<br>
> > dc-version: 1.1.10-14.el6_5.2-368c726<br>
> > last-lrm-refresh: 1399466204<br>
> > no-quorum-policy: ignore<br>
> > stonith-enabled: false<br>
> ><br>
> > ==============================================<br>
> > When i try to create and delete this resource in a loop,<br>
><br>
> Why would you do that? :-)<br>
><br>
> Just to test if things are fine if resource is created and deleted in quick succession. But the issue is also seen arbitrarily. Issue is sometimes seen even in first iteration of the loop.<br>
><br>
> > after few iteration, delete fails as shown below. This can be reproduced easily. I make sure to unclone resource before deleting the resource. Unclone happens successfully<br>
<br>
</div></div>[snip]<br>
<div class=""><br>
> > May 7 07:20:13 vsanqa12 attrd[4317]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2 (<null>)<br>
> > May 7 07:20:13 vsanqa12 attrd[4317]: notice: attrd_perform_update: Sent delete 4404: node=vsanqa12, attr=master-vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2, id=<n/a>, set=(null), section=status<br>
> > May 7 07:20:13 vsanqa12 crmd[4319]: notice: process_lrm_event: LRM operation vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2_stop_0 (call=1379, rc=0, cib-update=1161, confirmed=true) ok<br>
> > May 7 07:20:13 vsanqa12 attrd[4317]: notice: attrd_perform_update: Sent delete 4406: node=vsanqa12, attr=master-vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2, id=<n/a>, set=(null), section=status<br>
> > May 7 07:20:13 vsanqa12 attrd[4317]: warning: attrd_cib_callback: Update 4404 for master-vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2=(null) failed: Application of an update diff failed<br>
> > May 7 07:20:13 vsanqa12 cib[4314]: warning: cib_process_diff: Diff 0.6804.2 -> 0.6804.3 from vsanqa11 not applied to 0.6804.2: Failed application of an update diff<br>
> > May 7 07:20:13 vsanqa12 cib[4314]: notice: cib_server_process_diff: Not applying diff 0.6804.3 -> 0.6804.4 (sync in progress)<br>
<br>
<br>
</div>Ah. Now I recognise this :-(<br>
<br>
First the good news, this will be fixed when the new CIB code arrives in 6.6<br>
<br>
The way the old cib works is that one node makes the change and sends it out as a patch to the other nodes.<br>
Great in theory except the old patch format wasn't real great at preserving ordering changes - but it can detect them, hence:<br>
<div class=""><br>
> May 7 07:20:13 vsanqa12 cib[4314]: warning: cib_process_diff: Diff 0.6804.2 -> 0.6804.3 from vsanqa11 not applied to 0.6804.2: Failed application of an update diff<br>
<br>
</div>The cib does recover, but the operation is reported as having failed to pcs.<br>
<br>
We are considering a couple of options that may make it into 6.5<br>
<br>
<br>_______________________________________________<br>
Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br>
<a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a><br>
<br></blockquote></div><br></div>