<div dir="ltr">Please see my reply inline. Attached is the crm_report output.<br><div class="gmail_extra"><br><br><div class="gmail_quote">On Thu, May 8, 2014 at 5:45 AM, Andrew Beekhof <span dir="ltr"><<a href="mailto:andrew@beekhof.net" target="_blank">andrew@beekhof.net</a>></span> wrote:<br>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex"><div><div><br>
On 8 May 2014, at 12:38 am, K Mehta <<a href="mailto:kiranmehta1981@gmail.com" target="_blank">kiranmehta1981@gmail.com</a>> wrote:<br>
<br>
> I created a multi state resource ms-2be6c088-a1fa-464a-b00d-f4bccb4f5af2 (vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2).<br>
><br>
> Here is the configuration:<br>
> ==========================<br>
> [root@vsanqa11 ~]# pcs config<br>
> Cluster Name: vsanqa11_12<br>
> Corosync Nodes:<br>
><br>
> Pacemaker Nodes:<br>
> vsanqa11 vsanqa12<br>
><br>
> Resources:<br>
> Master: ms-2be6c088-a1fa-464a-b00d-f4bccb4f5af2<br>
> Meta Attrs: clone-max=2 globally-unique=false target-role=started<br>
> Resource: vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2 (class=ocf provider=heartbeat type=vgc-cm-agent.ocf)<br>
> Attributes: cluster_uuid=2be6c088-a1fa-464a-b00d-f4bccb4f5af2<br>
> Operations: monitor interval=30s role=Master timeout=100s (vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2-monitor-interval-30s)<br>
> monitor interval=31s role=Slave timeout=100s (vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2-monitor-interval-31s)<br>
><br>
> Stonith Devices:<br>
> Fencing Levels:<br>
><br>
> Location Constraints:<br>
> Resource: ms-2be6c088-a1fa-464a-b00d-f4bccb4f5af2<br>
> Enabled on: vsanqa11 (score:INFINITY) (id:location-ms-2be6c088-a1fa-464a-b00d-f4bccb4f5af2-vsanqa11-INFINITY)<br>
> Enabled on: vsanqa12 (score:INFINITY) (id:location-ms-2be6c088-a1fa-464a-b00d-f4bccb4f5af2-vsanqa12-INFINITY)<br>
> Resource: vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2<br>
> Enabled on: vsanqa11 (score:INFINITY) (id:location-vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2-vsanqa11-INFINITY)<br>
> Enabled on: vsanqa12 (score:INFINITY) (id:location-vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2-vsanqa12-INFINITY)<br>
> Ordering Constraints:<br>
> Colocation Constraints:<br>
><br>
> Cluster Properties:<br>
> cluster-infrastructure: cman<br>
> dc-version: 1.1.10-14.el6_5.2-368c726<br>
> last-lrm-refresh: 1399466204<br>
> no-quorum-policy: ignore<br>
> stonith-enabled: false<br>
><br>
> ==============================================<br>
> When i try to create and delete this resource in a loop,<br>
<br>
</div></div>Why would you do that? :-)<br></blockquote><div><br></div><div>Just to test if things are fine if resource is created and deleted in quick succession. But the issue is also seen arbitrarily. Issue is sometimes seen even in first iteration of the loop. </div>
<blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div><br>
> after few iteration, delete fails as shown below. This can be reproduced easily. I make sure to unclone resource before deleting the resource. Unclone happens successfully<br>
<br>
</div>Can you tell us the exact commands you ran?<br></blockquote><div>Here is the list of commands used for creation</div><div><br></div><div><div> pcs cluster cib $CLUSTER_CREATE_LOG || exit 1</div><div><br></div>
<div> pcs -f $CLUSTER_CREATE_LOG property set stonith-enabled=false || exit 1</div><div> pcs -f $CLUSTER_CREATE_LOG property set no-quorum-policy=ignore || exit 1</div><div> #syntax for following command is different across pcs 9.26 and 9.90</div>
<div> pcs -f $CLUSTER_CREATE_LOG resource defaults resource-stickiness=100 > /dev/null 2>&1</div><div> if [ $? -ne 0 ]; then</div><div> pcs -f $CLUSTER_CREATE_LOG resource rsc defaults resource-stickiness=100 || exit 1</div>
<div> fi</div><div> pcs -f $CLUSTER_CREATE_LOG resource create vha-$uuid ocf:heartbeat:vgc-cm-agent.ocf\</div><div> cluster_uuid=$uuid \</div><div> op monitor role="Master" interval=30s timeout=100s\</div>
<div> op monitor role="Slave" interval=31s timeout=100s || exit 1</div><div> pcs -f $CLUSTER_CREATE_LOG resource master ms-${uuid} vha-${uuid} meta clone-max=2 \</div><div> globally-unique=false target-role=started || exit 1</div>
<div><br></div><div> pcs -f $CLUSTER_CREATE_LOG constraint location vha-${uuid} prefers $node1 || exit 1</div><div> pcs -f $CLUSTER_CREATE_LOG constraint location vha-${uuid} prefers $node2 || exit 1</div>
<div> pcs -f $CLUSTER_CREATE_LOG constraint location ms-${uuid} prefers $node1 || exit 1</div><div> pcs -f $CLUSTER_CREATE_LOG constraint location ms-${uuid} prefers $node2 || exit 1</div><div><br></div>
<div> #syntax for following command different across pcs 9.26 and 9.90</div><div> pcs cluster cib-push $CLUSTER_CREATE_LOG > /dev/null 2>&1</div><div> if [ $? -ne 0 ]; then</div><div> pcs cluster push cib $CLUSTER_CREATE_LOG</div>
<div> fi</div><div><br></div><div> if [ $? -eq 0 ]; then</div><div> echo "Success"</div><div> else</div><div> echo "Failure"</div><div> exit 1</div>
<div> fi</div></div><div><br></div><div><br></div><div>List of commands for deletion</div><div><br></div><div><div> pcs resource show vha-${uuid} > /dev/null 2>&1</div><div> if [ $? -eq 0 ]; then</div>
<div> pcs resource unclone ms-${uuid} > /dev/null 2>&1</div><div> if [ $? -ne 0 ]; then</div><div> echo "Failed to unclone resource with uuid: $uuid"</div>
<div> #do not exit because this command always fails</div><div> #in pcs 9.26 version. Attempt to delete</div><div> fi</div><div><br></div><div> pcs resource delete vha-${uuid} > /dev/null 2>&1</div>
<div> if [ $? -ne 0 ]; then</div><div> echo "Failed to delete resource with uuid: $uuid"</div><div> exit 1</div><div> fi</div><div> else</div>
<div> if [ $verbose -eq 1 ]; then</div><div> echo "Could not find resource with uuid ${uuid}"</div><div> fi</div><div> </div><div> fi</div>
</div><div><br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-color:rgb(204,204,204);border-left-style:solid;padding-left:1ex">
<div><br>
><br>
> Removing Constraint - location-vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2-vsanqa11-INFINITY<br>
> Removing Constraint - location-vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2-vsanqa12-INFINITY<br>
> Attempting to stop: vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2...Error: Unable to stop: vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2 before deleting (re-run with --force to force deletion)<br>
> Failed to delete resource with uuid: 2be6c088-a1fa-464a-b00d-f4bccb4f5af2<br>
><br>
> ==============================================<br>
><br>
> Log file snippet of relevant time<br>
<br>
</div>And on the other node? Did you configure a log file? That would also be interesting.<br>
Actually, better make that a crm_report... the PE files will likely be interesting too.<br>
<div><div><br>
> ============================================<br>
><br>
> May 7 07:20:12 vsanqa12 vgc-vha-config: /usr/bin/vgc-vha-config --stop /dev/vgca0_vha<br>
> May 7 07:20:12 vsanqa12 crmd[4319]: notice: do_state_transition: State transition S_NOT_DC -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL origin=do_election_count_vote ]<br>
> May 7 07:20:12 vsanqa12 kernel: VGC: [0000006711341b03:I] Stopped vHA/vShare instance /dev/vgca0_vha<br>
> May 7 07:20:12 vsanqa12 stonith-ng[4315]: notice: unpack_config: On loss of CCM Quorum: Ignore<br>
> May 7 07:20:12 vsanqa12 vgc-vha-config: Success<br>
> May 7 07:20:13 vsanqa12 stonith-ng[4315]: notice: unpack_config: On loss of CCM Quorum: Ignore<br>
> May 7 07:20:13 vsanqa12 attrd[4317]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2 (<null>)<br>
> May 7 07:20:13 vsanqa12 attrd[4317]: notice: attrd_perform_update: Sent delete 4404: node=vsanqa12, attr=master-vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2, id=<n/a>, set=(null), section=status<br>
> May 7 07:20:13 vsanqa12 crmd[4319]: notice: process_lrm_event: LRM operation vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2_stop_0 (call=1379, rc=0, cib-update=1161, confirmed=true) ok<br>
> May 7 07:20:13 vsanqa12 attrd[4317]: notice: attrd_perform_update: Sent delete 4406: node=vsanqa12, attr=master-vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2, id=<n/a>, set=(null), section=status<br>
> May 7 07:20:12 vsanqa12 kernel: VGC: [0000006711341b03:I] Stopped vHA/vShare instance /dev/vgca0_vh088-a1fa-464a-b00d-f4bccb4f5af2=(null) failed: Application of an updMay 7 07:20:12 vsanqa12 stonith-ng[4315]: notice: unpack_config: On loss of CCM Quorum: Ignore<br>
> May 7 07:20:12 vsanqa12 vgc-vha-config: Success<br>
> May 7 07:20:13 vsanqa12 stonith-ng[4315]: notice: unpack_config: On loss of CCM Quorum: Ignore<br>
> May 7 07:20:13 vsanqa12 attrd[4317]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2 (<null>)<br>
> May 7 07:20:13 vsanqa12 attrd[4317]: notice: attrd_perform_update: Sent delete 4404: node=vsanqa12, attr=master-vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2, id=<n/a>, set=(null), section=status<br>
> May 7 07:20:13 vsanqa12 crmd[4319]: notice: process_lrm_event: LRM operation vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2_stop_0 (call=1379, rc=0, cib-update=1161, confirmed=true) ok<br>
> May 7 07:20:13 vsanqa12 attrd[4317]: notice: attrd_perform_update: Sent delete 4406: node=vsanqa12, attr=master-vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2, id=<n/a>, set=(null), section=status<br>
> May 7 07:20:13 vsanqa12 attrd[4317]: warning: attrd_cib_callback: Update 4404 for master-vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2=(null) failed: Application of an update diff failed<br>
> May 7 07:20:13 vsanqa12 cib[4314]: warning: cib_process_diff: Diff 0.6804.2 -> 0.6804.3 from vsanqa11 not applied to 0.6804.2: Failed application of an update diff<br>
> May 7 07:20:13 vsanqa12 cib[4314]: notice: cib_server_process_diff: Not applying diff 0.6804.3 -> 0.6804.4 (sync in progress)<br>
><br>
><br>
> [root@vsanqa12 ~]# pcs status<br>
> Cluster name: vsanqa11_12<br>
> Last updated: Wed May 7 07:24:29 2014<br>
> Last change: Wed May 7 07:20:13 2014 via crm_resource on vsanqa11<br>
> Stack: cman<br>
> Current DC: vsanqa11 - partition with quorum<br>
> Version: 1.1.10-14.el6_5.2-368c726<br>
> 2 Nodes configured<br>
> 1 Resources configured<br>
><br>
><br>
> Online: [ vsanqa11 vsanqa12 ]<br>
><br>
> Full list of resources:<br>
><br>
> vha-2be6c088-a1fa-464a-b00d-f4bccb4f5af2 (ocf::heartbeat:vgc-cm-agent.ocf): Stopped<br>
><br>
><br>
</div></div>> _______________________________________________<br>
> Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org" target="_blank">Pacemaker@oss.clusterlabs.org</a><br>
> <a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
><br>
> Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
> Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
> Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a><br>
<br>
<br>_______________________________________________<br>
Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org" target="_blank">Pacemaker@oss.clusterlabs.org</a><br>
<a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a><br>
<br></blockquote></div><br></div></div>