[Pacemaker] Problems with SBD fencing

Dejan Muhamedagic dejanmm at fastmail.fm
Tue Aug 6 05:15:17 EDT 2013 

Hi,

On Thu, Aug 01, 2013 at 07:58:55PM +0200, Jan Christian Kaldestad wrote:
> Thanks for the explanation. But I'm quite confused about the SBD stonith
> resource configuration, as the SBD fencing wiki clearly states:
> "The sbd agent does not need to and should not be cloned. If all of your
> nodes run SBD, as is most likely, not even a monitor action provides a real
> benefit, since the daemon would suicide the node if there was a problem. "
> 
> and also this thread
> http://oss.clusterlabs.org/pipermail/pacemaker/2012-March/013507.htmlmention
> that there should be only one SBD resource configured.
> 
> Can someone please clarify? Should I configure 2 separate SBD resources,
> one for each cluster node?

No. One sbd resource is sufficient.

Thanks,

Dejan

> 
> -- 
> Best regards
> Jan
> 
> 
> On Thu, Aug 1, 2013 at 6:47 PM, Andreas Mock <andreas.mock at web.de> wrote:
> 
> > Hi Jan,****
> >
> > ** **
> >
> > first of all I don't know the SBD-Fencing-Infrastructure (just read the***
> > *
> >
> > article linked by you). But as far as I understand the "normal" fencing***
> > *
> >
> > (initiated on behalf of pacemaker) is done in the following way.****
> >
> > ** **
> >
> > SBD fencing resoure (agent) is writing a request for self-stonithing into*
> > ***
> >
> > one or more SBD partitions where the SBD-daemon is listening and hopefully
> > ****
> >
> > reacting on.****
> >
> > So, I'm pretty sure (without knowing) that you have to configure the****
> >
> > stonith agent in a way that pacemaker knows howto talk to the stonith agent
> > ****
> >
> > to kill a certain cluster node.****
> >
> > What is the problem in you scenario: The agent which should be contacted**
> > **
> >
> > to stonith the node2 is/was running on node2 and can't be connected
> > anymore.****
> >
> > ** **
> >
> > Because of that stonith agent configuration is most of the times done the*
> > ***
> >
> > following way in a two node cluster:****
> >
> > On every node runs a stonith agent. The stonith agent is configured to****
> >
> > stonith the OTHER node. You have to be sure that this is technically ****
> >
> > always possible.****
> >
> > This can be achieved with resource clones or - which is IMHO simpler - in
> > ****
> >
> > a 2-node-environment with two stonith resources and a negative colocation*
> > ***
> >
> > constraint.****
> >
> > ** **
> >
> > As far as I know there is also a self-stonith-safty-belt implemented****
> >
> > in a way that a stonith agent on a node to be shot is never contacted.****
> >
> > (Do I remember correct?)****
> >
> > ** **
> >
> > I'm sure this may solve your problem.****
> >
> > ** **
> >
> > Best regards****
> >
> > Andreas Mock****
> >
> > ** **
> >
> > ** **
> >
> > *Von:* Jan Christian Kaldestad [mailto:janck76 at gmail.com]
> > *Gesendet:* Donnerstag, 1. August 2013 15:46
> > *An:* pacemaker at oss.clusterlabs.org
> > *Betreff:* [Pacemaker] Problems with SBD fencing****
> >
> > ** **
> >
> > Hi,****
> >
> >
> > I am evaluating the SLES HA Extension 11 SP3 product. The cluster
> > consists of 2-nodes (active/passive), using SBD stonith resource on a
> > shared SAN disk. Configuration according to
> > http://www.linux-ha.org/wiki/SBD_Fencing****
> >
> > The SBD daemon is running on both nodes, and the stontih resource (defined
> > as primitive) is running on one node only.
> > There is also a monitor operation for the stonith resource
> > (interval=36000, timeout=20)****
> >
> > I am having some problems getting failover/fencing to work as expected in
> > the following scenario:
> > - Node 1 is running the resources that I created (except stonith)
> > - Node 2 is running the stonith resource
> > - Disconnect Node 2 from the network by bringing the interface down
> > - Node 2 status changes to UNCLEAN (offline), but the stonith resource
> > does not switch over to Node 1 and Node 2 does not reboot as I would expect.
> > - Checking the logs on Node 1, I notice the following:
> >  Aug  1 12:00:01 slesha1n1i-u pengine[8915]:  warning: pe_fence_node: Node
> > slesha1n2i-u will be fenced because the node is no longer part of the
> > cluster
> >  Aug  1 12:00:01 slesha1n1i-u pengine[8915]:  warning:
> > determine_online_status: Node slesha1n2i-u is unclean
> >  Aug  1 12:00:01 slesha1n1i-u pengine[8915]:  warning: custom_action:
> > Action stonith_sbd_stop_0 on slesha1n2i-u is unrunnable (offline)
> >  Aug  1 12:00:01 slesha1n1i-u pengine[8915]:  warning: stage6: Scheduling
> > Node slesha1n2i-u for STONITH
> >  Aug  1 12:00:01 slesha1n1i-u pengine[8915]:   notice: LogActions: Move
> > stonith_sbd   (Started slesha1n2i-u -> slesha1n1i-u)
> >  ...
> >  Aug  1 12:00:01 slesha1n1i-u crmd[8916]:   notice: te_fence_node:
> > Executing reboot fencing operation (24) on slesha1n2i-u (timeout=60000)
> >  Aug  1 12:00:01 slesha1n1i-u stonith-ng[8912]:   notice: handle_request:
> > Client crmd.8916.3144546f wants to fence (reboot) 'slesha1n2i-u' with
> > device '(any)'
> >  Aug  1 12:00:01 slesha1n1i-u stonith-ng[8912]:   notice:
> > initiate_remote_stonith_op: Initiating remote operation reboot for
> > slesha1n2i-u: 8c00ff7b-2986-4b2a-8b4a-760e8346349b (0)
> >  Aug  1 12:00:01 slesha1n1i-u stonith-ng[8912]:    error: remote_op_done:
> > Operation reboot of slesha1n2i-u by slesha1n1i-u for
> > crmd.8916 at slesha1n1i-u.8c00ff7b: No route to host
> >  Aug  1 12:00:01 slesha1n1i-u crmd[8916]:   notice:
> > tengine_stonith_callback: Stonith operation
> > 3/24:3:0:8a0f32b2-f91c-4cdf-9cee-1ba9b6e187ab: No route to host (-113)
> >  Aug  1 12:00:01 slesha1n1i-u crmd[8916]:   notice:
> > tengine_stonith_callback: Stonith operation 3 for slesha1n2i-u failed (No
> > route to host): aborting transition.
> >  Aug  1 12:00:01 slesha1n1i-u crmd[8916]:   notice:
> > tengine_stonith_notify: Peer slesha1n2i-u was not terminated
> > (st_notify_fence) by slesha1n1i-u for slesha1n1i-u: No route to host
> > (ref=8c00ff7b-2986-4b2a-8b4a-760e8346349b) by client crmd.8916
> >  Aug  1 12:00:01 slesha1n1i-u crmd[8916]:   notice: run_graph: Transition
> > 3 (Complete=1, Pending=0, Fired=0, Skipped=5, Incomplete=0,
> > Source=/var/lib/pacemaker/pengine/pe-warn-15.bz2): Stopped
> >  Aug  1 12:00:01 slesha1n1i-u pengine[8915]:   notice: unpack_config: On
> > loss of CCM Quorum: Ignore
> >  Aug  1 12:00:01 slesha1n1i-u pengine[8915]:  warning: pe_fence_node: Node
> > slesha1n2i-u will be fenced because the node is no longer part of the
> > cluster
> >  Aug  1 12:00:01 slesha1n1i-u pengine[8915]:  warning:
> > determine_online_status: Node slesha1n2i-u is unclean
> >  Aug  1 12:00:01 slesha1n1i-u pengine[8915]:  warning: custom_action:
> > Action stonith_sbd_stop_0 on slesha1n2i-u is unrunnable (offline)
> >  Aug  1 12:00:01 slesha1n1i-u pengine[8915]:  warning: stage6: Scheduling
> > Node slesha1n2i-u for STONITH
> >  Aug  1 12:00:01 slesha1n1i-u pengine[8915]:   notice: LogActions: Move
> > stonith_sbd   (Started slesha1n2i-u -> slesha1n1i-u)
> >  ...
> >  Aug  1 12:00:02 slesha1n1i-u crmd[8916]:   notice: too_many_st_failures:
> > Too many failures to fence slesha1n2i-u (11), giving up
> >  ****
> >
> >  ****
> >
> >
> > - Then I bring Node 1 online again and start the cluster service, checking
> > logs:
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ] CLM CONFIGURATION
> > CHANGE
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ] New Configuration:
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ]  r(0) ip(x.x.x.x)
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ] Members Left:
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ] Members Joined:
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [pcmk  ] notice:
> > pcmk_peer_update: Transitional membership event on ring 376: memb=1, new=0,
> > lost=0
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
> > pcmk_peer_update: memb: slesha1n1i-u 168824371
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ] CLM CONFIGURATION
> > CHANGE
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ] New Configuration:
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ]  r(0) ip(x.x.x.x)
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ]  r(0) ip(y.y.y.y)
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ] Members Left:
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ] Members Joined:
> >  Aug  1 12:31:13 slesha1n1i-u cib[8911]:   notice: ais_dispatch_message:
> > Membership 376: quorum acquired
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ]  r(0) ip(y.y.y.y)
> >  Aug  1 12:31:13 slesha1n1i-u crmd[8916]:   notice: ais_dispatch_message:
> > Membership 376: quorum acquired
> >  Aug  1 12:31:13 slesha1n1i-u cib[8911]:   notice: crm_update_peer_state:
> > crm_update_ais_node: Node slesha1n2i-u[168824372] - state is now member
> > (was lost)
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [pcmk  ] notice:
> > pcmk_peer_update: Stable membership event on ring 376: memb=2, new=1, lost=0
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
> > update_member: Node 168824372/slesha1n2i-u is now: member
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
> > pcmk_peer_update: NEW:  slesha1n2i-u 168824372
> >  Aug  1 12:31:13 slesha1n1i-u crmd[8916]:   notice: crm_update_peer_state:
> > crm_update_ais_node: Node slesha1n2i-u[168824372] - state is now member
> > (was lost)
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
> > pcmk_peer_update: MEMB: slesha1n1i-u 168824371
> >  Aug  1 12:31:13 slesha1n1i-u crmd[8916]:   notice: peer_update_callback:
> > Node return implies stonith of slesha1n2i-u (action 24) completed
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
> > pcmk_peer_update: MEMB: slesha1n2i-u 168824372
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
> > send_member_notification: Sending membership update 376 to 2 children
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [TOTEM ] A processor joined
> > or left the membership and a new membership was formed.
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
> > update_member: 0x69f2f0 Node 168824372 (slesha1n2i-u) born on: 376
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
> > send_member_notification: Sending membership update 376 to 2 children
> >  Aug  1 12:31:13 slesha1n1i-u crmd[8916]:   notice: crm_update_quorum:
> > Updating quorum status to true (call=119)
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CPG   ] chosen downlist:
> > sender r(0) ip(x.x.x.x) ; members(old:1 left:0)
> >  Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [MAIN  ] Completed service
> > synchronization, ready to provide service.
> >  Aug  1 12:31:13 slesha1n1i-u crmd[8916]:   notice: too_many_st_failures:
> > Too many failures to fence slesha1n2i-u (13), giving up
> >  Aug  1 12:31:13 slesha1n1i-u crmd[8916]:   notice: too_many_st_failures:
> > Too many failures to fence slesha1n2i-u (13), giving up
> >  Aug  1 12:31:14 slesha1n1i-u mgmtd: [8917]: info: CIB query: cib
> >  Aug  1 12:31:14 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
> > update_member: Node slesha1n2i-u now has process list:
> > 00000000000000000000000000151302 (1381122)
> >  Aug  1 12:31:14 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
> > send_member_notification: Sending membership update 376 to 2 children
> >  Aug  1 12:31:14 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
> > update_member: Node slesha1n2i-u now has process list:
> > 00000000000000000000000000141302 (1315586)
> >  Aug  1 12:31:14 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
> > send_member_notification: Sending membership update 376 to 2 children
> >  Aug  1 12:31:14 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
> > update_member: Node slesha1n2i-u now has process list:
> > 00000000000000000000000000101302 (1053442)
> >  Aug  1 12:31:14 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
> > send_member_notification: Sending membership update 376 to 2 children
> >  Aug  1 12:31:15 slesha1n1i-u crmd[8916]:   notice: do_state_transition:
> > State transition S_IDLE -> S_INTEGRATION [ input=I_NODE_JOIN
> > cause=C_HA_MESSAGE origin=route_message ]
> >  Aug  1 12:31:15 slesha1n1i-u crmd[8916]:   notice: too_many_st_failures:
> > Too many failures to fence slesha1n2i-u (13), giving up
> >  Aug  1 12:31:15 slesha1n1i-u crmd[8916]:   notice: too_many_st_failures:
> > Too many failures to fence slesha1n2i-u (13), giving up****
> >
> > - Cluster status changes to Online for both nodes, but the stonith
> > resource won't start on any of the nodes.
> > - Trying to start the resource manually, but no success.
> > - Trying to restart the corosync process on Node 1 (rcopenais restart),
> > but it hangs forever. Checking logs:
> >  Aug  1 12:42:08 slesha1n1i-u corosync[8905]:  [SERV  ] Unloading all
> > Corosync service engines.
> >  Aug  1 12:42:08 slesha1n1i-u corosync[8905]:  [pcmk  ] notice:
> > pcmk_shutdown: Shuting down Pacemaker
> >  Aug  1 12:42:08 slesha1n1i-u corosync[8905]:  [pcmk  ] notice:
> > stop_child: Sent -15 to mgmtd: [8917]
> >  Aug  1 12:42:08 slesha1n1i-u mgmtd: [8917]: info: mgmtd is shutting down
> >  Aug  1 12:42:08 slesha1n1i-u mgmtd: [8917]: info: final_crm: client_id=1
> > cib_name=live
> >  Aug  1 12:42:08 slesha1n1i-u mgmtd: [8917]: info: final_crm: client_id=2
> > cib_name=live
> >  Aug  1 12:42:08 slesha1n1i-u mgmtd: [8917]: debug: [mgmtd] stopped
> >  Aug  1 12:42:08 slesha1n1i-u corosync[8905]:  [pcmk  ] notice:
> > pcmk_shutdown: mgmtd confirmed stopped
> >  Aug  1 12:42:08 slesha1n1i-u corosync[8905]:  [pcmk  ] notice:
> > stop_child: Sent -15 to crmd: [8916]
> >  Aug  1 12:42:08 slesha1n1i-u crmd[8916]:   notice: crm_shutdown:
> > Requesting shutdown, upper limit is 1200000ms
> >  Aug  1 12:42:08 slesha1n1i-u attrd[8914]:   notice: attrd_trigger_update:
> > Sending flush op to all hosts for: shutdown (1375353728)
> >  Aug  1 12:42:08 slesha1n1i-u attrd[8914]:   notice: attrd_perform_update:
> > Sent update 22: shutdown=1375353728
> >  Aug  1 12:42:08 slesha1n1i-u crmd[8916]:   notice: too_many_st_failures:
> > Too many failures to fence slesha1n2i-u (13), giving up
> >  Aug  1 12:42:08 slesha1n1i-u crmd[8916]:  warning: do_log: FSA: Input
> > I_TE_SUCCESS from abort_transition_graph() received in state S_POLICY_ENGINE
> >  Aug  1 12:42:38 slesha1n1i-u corosync[8905]:  [pcmk  ] notice:
> > pcmk_shutdown: Still waiting for crmd (pid=8916, seq=6) to terminate...
> >  Aug  1 12:43:08 slesha1n1i-u corosync[8905]:  [pcmk  ] notice:
> > pcmk_shutdown: Still waiting for crmd (pid=8916, seq=6) to terminate...***
> > *
> >
> > ...****
> >
> > - Finally I kill the corosync process on Node 1 (killall -9 corosync),
> > then corsoync restarts.
> > - Checking status. All resources are up and running on Node 1, and the
> > stonith resource is running on Node 2 again.****
> >
> >  ****
> >
> > I have tested the same scenario several times. Sometimes the fencing
> > mechaism works as expected, but other times the stonith resource is not
> > transferred to Node 1 - as described here. So I need some assistance to
> > overcome this problem....****
> >
> >  ****
> >
> > --
> > Best regards
> > Jan ****
> >
> > _______________________________________________
> > Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> > http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> >
> > Project Home: http://www.clusterlabs.org
> > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > Bugs: http://bugs.clusterlabs.org
> >
> >
> 
> 
> -- 
> mvh
> Jan Christian

> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org

More information about the Pacemaker mailing list