[Pacemaker] Being fenced node is killed again and again even the connection is recovered!

Steven Dake sdake at redhat.com
Fri May 14 10:26:03 UTC 2010


ifconfig eth0 down is not a valid test case.  that will likely lead to
bad things happening.

I recommend using iptables to test the software.

Also Corosync 1.2.2 is out which fixes bugs vs corosync 1.2.0.

Regards
-steve

On Fri, 2010-05-14 at 18:02 +0800, Javen Wu wrote:
> I forget mention the version I used. 
> I used SLES11-SP1-HAE Beta5
> Pacemaker 1.0.7
> Corosync 1.2.0
> Cluster Glue 1.0.3
> 
> 
> 2010/5/14 Javen Wu <wu.javen at gmail.com>
>         Hi Folks,
>         
>         I setup a three nodes cluster with SBD STONITH configured.
>         After I manually isolate one node by running "ifconfig eth1
>         down" on the node. The node is fenced as expected.
>         But after reboot, even the network is recovered, the node is
>         killed again once I start openais&pacemaker.
>         I saw the state of the node become from OFFLINE to ONLINE from
>         `crm_mon -n` before being killed. And I saw SBD slot from
>         reset->clear->reset.
>         
>         I attached the syslog and corosync log.
>         And my CIB configuration is very simple.
>         
>         Could you help me check what's the problem? In my mind, it's
>         not expected behaviour.
>         
>         ===%<====CIB information=====================
>         
>         <cib validate-with="pacemaker-1.0" crm_feature_set="3.0.1"
>         have-quorum="1" admin_epoch="0" epoch="349" num_updates="99"
>         cib-last-written="Fri May 14 14:50:21 2010" dc-uuid="vm209">
>           <configuration>
>             <crm_config>
>               <cluster_property_set id="cib-bootstrap-options">
>                 <nvpair id="cib-bootstrap-options-dc-version"
>         name="dc-version"
>         value="1.1.1-530add2a3721a0ecccb24660a97dbfdaa3e68f51"/>
>                 <nvpair
>         id="cib-bootstrap-options-cluster-infrastructure"
>         name="cluster-infrastructure" value="openais"/>
>                 <nvpair
>         id="cib-bootstrap-options-expected-quorum-votes"
>         name="expected-quorum-votes" value="3"/>
>               </cluster_property_set>
>             </crm_config>
>             <nodes>
>               <node id="vm208" uname="vm208" type="normal"/>
>               <node id="vm209" uname="vm209" type="normal"/>
>               <node id="vm210" uname="vm210" type="normal"/>
>             </nodes>
>             <resources>
>               <clone id="Fencing">
>                 <primitive class="stonith" id="sbd-fencing"
>         type="external/sbd">
>                   <instance_attributes
>         id="sbd-fencing-instance_attributes">
>                     <nvpair
>         id="sbd-fencing-instance_attributes-sbd_device"
>         name="sbd_device" value="/dev/sdc"/>
>                   </instance_attributes>
>                   <operations>
>                     <op id="sbd-fencing-monitor-20s" interval="20s"
>         name="monitor"/>
>                   </operations>
>                 </primitive>
>               </clone>
>             </resources>
>             <constraints/>
>             <rsc_defaults/>
>             <op_defaults/>
>           </configuration>
>           <status>
>             <node_state id="vm209" uname="vm209" ha="active"
>         in_ccm="true" crmd="online" join="member" expected="member"
>         crm-debug-origin="post_cache_update" shutdown="0">
>               <transient_attributes id="vm209">
>                 <instance_attributes id="status-vm209">
>                   <nvpair id="status-vm209-probe_complete"
>         name="probe_complete" value="true"/>
>                 </instance_attributes>
>               </transient_attributes>
>               <lrm id="vm209">
>                 <lrm_resources>
>                   <lrm_resource id="sbd-fencing:0" type="external/sbd"
>         class="stonith">
>                     <lrm_rsc_op id="sbd-fencing:0_monitor_0"
>         operation="monitor" crm-debug-origin="build_active_RAs"
>         crm_feature_set="3.0.1"
>         transition-key="4:1:7:f0adcb5c-10d1-4525-b094-b5ab1f776ee0"
>         transition-magic="0:7;4:1:7:f0adcb5c-10d1-4525-b094-b5ab1f776ee0" call-id="2" rc-code="7" op-status="0" interval="0" last-run="1273820137" last-rc-change="1273820137" exec-time="60" queue-time="0" op-digest="4c3fd39434577fbb6540606d808ed050"/>
>                     <lrm_rsc_op id="sbd-fencing:0_start_0"
>         operation="start" crm-debug-origin="build_active_RAs"
>         crm_feature_set="3.0.1"
>         transition-key="5:1:0:f0adcb5c-10d1-4525-b094-b5ab1f776ee0"
>         transition-magic="0:0;5:1:0:f0adcb5c-10d1-4525-b094-b5ab1f776ee0" call-id="3" rc-code="0" op-status="0" interval="0" last-run="1273820137" last-rc-change="1273820137" exec-time="10" queue-time="0" op-digest="4c3fd39434577fbb6540606d808ed050"/>
>                     <lrm_rsc_op id="sbd-fencing:0_monitor_20000"
>         operation="monitor" crm-debug-origin="build_active_RAs"
>         crm_feature_set="3.0.1"
>         transition-key="6:2:0:f0adcb5c-10d1-4525-b094-b5ab1f776ee0"
>         transition-magic="0:0;6:2:0:f0adcb5c-10d1-4525-b094-b5ab1f776ee0" call-id="4" rc-code="0" op-status="0" interval="20000" last-run="1273822956" last-rc-change="1273820137" exec-time="1170" queue-time="0" op-digest="4029bbaef749649e82d602afb46dd872"/>
>                   </lrm_resource>
>                 </lrm_resources>
>               </lrm>
>             </node_state>
>             <node_state id="vm208" uname="vm208" ha="dead"
>         in_ccm="false" crmd="offline"
>         crm-debug-origin="send_stonith_update" join="down"
>         expected="down" shutdown="0"/>
>             <node_state id="vm210" uname="vm210" ha="active"
>         in_ccm="true" crmd="online"
>         crm-debug-origin="post_cache_update" join="member"
>         expected="member" shutdown="0">
>               <transient_attributes id="vm210">
>                 <instance_attributes id="status-vm210">
>                   <nvpair id="status-vm210-probe_complete"
>         name="probe_complete" value="true"/>
>                 </instance_attributes>
>               </transient_attributes>
>               <lrm id="vm210">
>                 <lrm_resources>
>                   <lrm_resource id="sbd-fencing:2" type="external/sbd"
>         class="stonith">
>                     <lrm_rsc_op id="sbd-fencing:2_monitor_0"
>         operation="monitor" crm-debug-origin="build_active_RAs"
>         crm_feature_set="3.0.1"
>         transition-key="8:5:7:f0adcb5c-10d1-4525-b094-b5ab1f776ee0"
>         transition-magic="0:7;8:5:7:f0adcb5c-10d1-4525-b094-b5ab1f776ee0" call-id="2" rc-code="7" op-status="0" interval="0" last-run="1273820388" last-rc-change="1273820388" exec-time="20" queue-time="0" op-digest="4c3fd39434577fbb6540606d808ed050"/>
>                     <lrm_rsc_op id="sbd-fencing:2_start_0"
>         operation="start" crm-debug-origin="build_active_RAs"
>         crm_feature_set="3.0.1"
>         transition-key="13:5:0:f0adcb5c-10d1-4525-b094-b5ab1f776ee0"
>         transition-magic="0:0;13:5:0:f0adcb5c-10d1-4525-b094-b5ab1f776ee0" call-id="3" rc-code="0" op-status="0" interval="0" last-run="1273820388" last-rc-change="1273820388" exec-time="10" queue-time="0" op-digest="4c3fd39434577fbb6540606d808ed050"/>
>                     <lrm_rsc_op id="sbd-fencing:2_monitor_20000"
>         operation="monitor" crm-debug-origin="build_active_RAs"
>         crm_feature_set="3.0.1"
>         transition-key="14:5:0:f0adcb5c-10d1-4525-b094-b5ab1f776ee0"
>         transition-magic="0:0;14:5:0:f0adcb5c-10d1-4525-b094-b5ab1f776ee0" call-id="4" rc-code="0" op-status="0" interval="20000" last-run="1273822976" last-rc-change="1273820389" exec-time="1040" queue-time="0" op-digest="4029bbaef749649e82d602afb46dd872"/>
>                   </lrm_resource>
>                 </lrm_resources>
>               </lrm>
>             </node_state>
>           </status>
>         </cib>
>         
>         
>         
>         
> 
> 
> 
> -- 
> Javen Wu
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf





More information about the Pacemaker mailing list