[Pacemaker] Problems with SBD fencing

Andreas Mock andreas.mock at web.de
Thu Aug 1 16:47:33 UTC 2013


Hi Jan,

 

first of all I don't know the SBD-Fencing-Infrastructure (just read the

article linked by you). But as far as I understand the "normal" fencing

(initiated on behalf of pacemaker) is done in the following way.

 

SBD fencing resoure (agent) is writing a request for self-stonithing into

one or more SBD partitions where the SBD-daemon is listening and hopefully

reacting on.

So, I'm pretty sure (without knowing) that you have to configure the

stonith agent in a way that pacemaker knows howto talk to the stonith agent

to kill a certain cluster node.

What is the problem in you scenario: The agent which should be contacted

to stonith the node2 is/was running on node2 and can't be connected anymore.

 

Because of that stonith agent configuration is most of the times done the

following way in a two node cluster:

On every node runs a stonith agent. The stonith agent is configured to

stonith the OTHER node. You have to be sure that this is technically 

always possible.

This can be achieved with resource clones or - which is IMHO simpler - in 

a 2-node-environment with two stonith resources and a negative colocation

constraint.

 

As far as I know there is also a self-stonith-safty-belt implemented

in a way that a stonith agent on a node to be shot is never contacted.

(Do I remember correct?)

 

I'm sure this may solve your problem.

 

Best regards

Andreas Mock

 

 

Von: Jan Christian Kaldestad [mailto:janck76 at gmail.com] 
Gesendet: Donnerstag, 1. August 2013 15:46
An: pacemaker at oss.clusterlabs.org
Betreff: [Pacemaker] Problems with SBD fencing

 

Hi,


I am evaluating the SLES HA Extension 11 SP3 product. The cluster  consists
of 2-nodes (active/passive), using SBD stonith resource on a shared SAN
disk. Configuration according to http://www.linux-ha.org/wiki/SBD_Fencing

The SBD daemon is running on both nodes, and the stontih resource (defined
as primitive) is running on one node only.
There is also a monitor operation for the stonith resource (interval=36000,
timeout=20)

I am having some problems getting failover/fencing to work as expected in
the following scenario:
- Node 1 is running the resources that I created (except stonith)
- Node 2 is running the stonith resource
- Disconnect Node 2 from the network by bringing the interface down
- Node 2 status changes to UNCLEAN (offline), but the stonith resource does
not switch over to Node 1 and Node 2 does not reboot as I would expect.
- Checking the logs on Node 1, I notice the following:
 Aug  1 12:00:01 slesha1n1i-u pengine[8915]:  warning: pe_fence_node: Node
slesha1n2i-u will be fenced because the node is no longer part of the
cluster
 Aug  1 12:00:01 slesha1n1i-u pengine[8915]:  warning:
determine_online_status: Node slesha1n2i-u is unclean
 Aug  1 12:00:01 slesha1n1i-u pengine[8915]:  warning: custom_action: Action
stonith_sbd_stop_0 on slesha1n2i-u is unrunnable (offline)
 Aug  1 12:00:01 slesha1n1i-u pengine[8915]:  warning: stage6: Scheduling
Node slesha1n2i-u for STONITH
 Aug  1 12:00:01 slesha1n1i-u pengine[8915]:   notice: LogActions: Move
stonith_sbd   (Started slesha1n2i-u -> slesha1n1i-u)
 ...
 Aug  1 12:00:01 slesha1n1i-u crmd[8916]:   notice: te_fence_node: Executing
reboot fencing operation (24) on slesha1n2i-u (timeout=60000)
 Aug  1 12:00:01 slesha1n1i-u stonith-ng[8912]:   notice: handle_request:
Client crmd.8916.3144546f wants to fence (reboot) 'slesha1n2i-u' with device
'(any)'
 Aug  1 12:00:01 slesha1n1i-u stonith-ng[8912]:   notice:
initiate_remote_stonith_op: Initiating remote operation reboot for
slesha1n2i-u: 8c00ff7b-2986-4b2a-8b4a-760e8346349b (0)
 Aug  1 12:00:01 slesha1n1i-u stonith-ng[8912]:    error: remote_op_done:
Operation reboot of slesha1n2i-u by slesha1n1i-u for
crmd.8916 at slesha1n1i-u.8c00ff7b: No route to host
 Aug  1 12:00:01 slesha1n1i-u crmd[8916]:   notice:
tengine_stonith_callback: Stonith operation
3/24:3:0:8a0f32b2-f91c-4cdf-9cee-1ba9b6e187ab: No route to host (-113)
 Aug  1 12:00:01 slesha1n1i-u crmd[8916]:   notice:
tengine_stonith_callback: Stonith operation 3 for slesha1n2i-u failed (No
route to host): aborting transition.
 Aug  1 12:00:01 slesha1n1i-u crmd[8916]:   notice: tengine_stonith_notify:
Peer slesha1n2i-u was not terminated (st_notify_fence) by slesha1n1i-u for
slesha1n1i-u: No route to host (ref=8c00ff7b-2986-4b2a-8b4a-760e8346349b) by
client crmd.8916
 Aug  1 12:00:01 slesha1n1i-u crmd[8916]:   notice: run_graph: Transition 3
(Complete=1, Pending=0, Fired=0, Skipped=5, Incomplete=0,
Source=/var/lib/pacemaker/pengine/pe-warn-15.bz2): Stopped
 Aug  1 12:00:01 slesha1n1i-u pengine[8915]:   notice: unpack_config: On
loss of CCM Quorum: Ignore
 Aug  1 12:00:01 slesha1n1i-u pengine[8915]:  warning: pe_fence_node: Node
slesha1n2i-u will be fenced because the node is no longer part of the
cluster
 Aug  1 12:00:01 slesha1n1i-u pengine[8915]:  warning:
determine_online_status: Node slesha1n2i-u is unclean
 Aug  1 12:00:01 slesha1n1i-u pengine[8915]:  warning: custom_action: Action
stonith_sbd_stop_0 on slesha1n2i-u is unrunnable (offline)
 Aug  1 12:00:01 slesha1n1i-u pengine[8915]:  warning: stage6: Scheduling
Node slesha1n2i-u for STONITH
 Aug  1 12:00:01 slesha1n1i-u pengine[8915]:   notice: LogActions: Move
stonith_sbd   (Started slesha1n2i-u -> slesha1n1i-u)
 ...
 Aug  1 12:00:02 slesha1n1i-u crmd[8916]:   notice: too_many_st_failures:
Too many failures to fence slesha1n2i-u (11), giving up
 

 


- Then I bring Node 1 online again and start the cluster service, checking
logs:
 Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ] CLM CONFIGURATION
CHANGE
 Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ] New Configuration:
 Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ]  r(0) ip(x.x.x.x)
 Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ] Members Left:
 Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ] Members Joined:
 Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [pcmk  ] notice:
pcmk_peer_update: Transitional membership event on ring 376: memb=1, new=0,
lost=0
 Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
pcmk_peer_update: memb: slesha1n1i-u 168824371
 Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ] CLM CONFIGURATION
CHANGE
 Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ] New Configuration:
 Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ]  r(0) ip(x.x.x.x)
 Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ]  r(0) ip(y.y.y.y)
 Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ] Members Left:
 Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ] Members Joined:
 Aug  1 12:31:13 slesha1n1i-u cib[8911]:   notice: ais_dispatch_message:
Membership 376: quorum acquired
 Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CLM   ]  r(0) ip(y.y.y.y)
 Aug  1 12:31:13 slesha1n1i-u crmd[8916]:   notice: ais_dispatch_message:
Membership 376: quorum acquired
 Aug  1 12:31:13 slesha1n1i-u cib[8911]:   notice: crm_update_peer_state:
crm_update_ais_node: Node slesha1n2i-u[168824372] - state is now member (was
lost)
 Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [pcmk  ] notice:
pcmk_peer_update: Stable membership event on ring 376: memb=2, new=1, lost=0
 Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [pcmk  ] info: update_member:
Node 168824372/slesha1n2i-u is now: member
 Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
pcmk_peer_update: NEW:  slesha1n2i-u 168824372
 Aug  1 12:31:13 slesha1n1i-u crmd[8916]:   notice: crm_update_peer_state:
crm_update_ais_node: Node slesha1n2i-u[168824372] - state is now member (was
lost)
 Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
pcmk_peer_update: MEMB: slesha1n1i-u 168824371
 Aug  1 12:31:13 slesha1n1i-u crmd[8916]:   notice: peer_update_callback:
Node return implies stonith of slesha1n2i-u (action 24) completed
 Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
pcmk_peer_update: MEMB: slesha1n2i-u 168824372
 Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
send_member_notification: Sending membership update 376 to 2 children
 Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [TOTEM ] A processor joined
or left the membership and a new membership was formed.
 Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [pcmk  ] info: update_member:
0x69f2f0 Node 168824372 (slesha1n2i-u) born on: 376
 Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
send_member_notification: Sending membership update 376 to 2 children
 Aug  1 12:31:13 slesha1n1i-u crmd[8916]:   notice: crm_update_quorum:
Updating quorum status to true (call=119)
 Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [CPG   ] chosen downlist:
sender r(0) ip(x.x.x.x) ; members(old:1 left:0)
 Aug  1 12:31:13 slesha1n1i-u corosync[8905]:  [MAIN  ] Completed service
synchronization, ready to provide service.
 Aug  1 12:31:13 slesha1n1i-u crmd[8916]:   notice: too_many_st_failures:
Too many failures to fence slesha1n2i-u (13), giving up
 Aug  1 12:31:13 slesha1n1i-u crmd[8916]:   notice: too_many_st_failures:
Too many failures to fence slesha1n2i-u (13), giving up
 Aug  1 12:31:14 slesha1n1i-u mgmtd: [8917]: info: CIB query: cib
 Aug  1 12:31:14 slesha1n1i-u corosync[8905]:  [pcmk  ] info: update_member:
Node slesha1n2i-u now has process list: 00000000000000000000000000151302
(1381122)
 Aug  1 12:31:14 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
send_member_notification: Sending membership update 376 to 2 children
 Aug  1 12:31:14 slesha1n1i-u corosync[8905]:  [pcmk  ] info: update_member:
Node slesha1n2i-u now has process list: 00000000000000000000000000141302
(1315586)
 Aug  1 12:31:14 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
send_member_notification: Sending membership update 376 to 2 children
 Aug  1 12:31:14 slesha1n1i-u corosync[8905]:  [pcmk  ] info: update_member:
Node slesha1n2i-u now has process list: 00000000000000000000000000101302
(1053442)
 Aug  1 12:31:14 slesha1n1i-u corosync[8905]:  [pcmk  ] info:
send_member_notification: Sending membership update 376 to 2 children
 Aug  1 12:31:15 slesha1n1i-u crmd[8916]:   notice: do_state_transition:
State transition S_IDLE -> S_INTEGRATION [ input=I_NODE_JOIN
cause=C_HA_MESSAGE origin=route_message ]
 Aug  1 12:31:15 slesha1n1i-u crmd[8916]:   notice: too_many_st_failures:
Too many failures to fence slesha1n2i-u (13), giving up
 Aug  1 12:31:15 slesha1n1i-u crmd[8916]:   notice: too_many_st_failures:
Too many failures to fence slesha1n2i-u (13), giving up

- Cluster status changes to Online for both nodes, but the stonith resource
won't start on any of the nodes.
- Trying to start the resource manually, but no success.
- Trying to restart the corosync process on Node 1 (rcopenais restart), but
it hangs forever. Checking logs:
 Aug  1 12:42:08 slesha1n1i-u corosync[8905]:  [SERV  ] Unloading all
Corosync service engines.
 Aug  1 12:42:08 slesha1n1i-u corosync[8905]:  [pcmk  ] notice:
pcmk_shutdown: Shuting down Pacemaker
 Aug  1 12:42:08 slesha1n1i-u corosync[8905]:  [pcmk  ] notice: stop_child:
Sent -15 to mgmtd: [8917]
 Aug  1 12:42:08 slesha1n1i-u mgmtd: [8917]: info: mgmtd is shutting down
 Aug  1 12:42:08 slesha1n1i-u mgmtd: [8917]: info: final_crm: client_id=1
cib_name=live
 Aug  1 12:42:08 slesha1n1i-u mgmtd: [8917]: info: final_crm: client_id=2
cib_name=live
 Aug  1 12:42:08 slesha1n1i-u mgmtd: [8917]: debug: [mgmtd] stopped
 Aug  1 12:42:08 slesha1n1i-u corosync[8905]:  [pcmk  ] notice:
pcmk_shutdown: mgmtd confirmed stopped
 Aug  1 12:42:08 slesha1n1i-u corosync[8905]:  [pcmk  ] notice: stop_child:
Sent -15 to crmd: [8916]
 Aug  1 12:42:08 slesha1n1i-u crmd[8916]:   notice: crm_shutdown: Requesting
shutdown, upper limit is 1200000ms
 Aug  1 12:42:08 slesha1n1i-u attrd[8914]:   notice: attrd_trigger_update:
Sending flush op to all hosts for: shutdown (1375353728)
 Aug  1 12:42:08 slesha1n1i-u attrd[8914]:   notice: attrd_perform_update:
Sent update 22: shutdown=1375353728
 Aug  1 12:42:08 slesha1n1i-u crmd[8916]:   notice: too_many_st_failures:
Too many failures to fence slesha1n2i-u (13), giving up
 Aug  1 12:42:08 slesha1n1i-u crmd[8916]:  warning: do_log: FSA: Input
I_TE_SUCCESS from abort_transition_graph() received in state S_POLICY_ENGINE
 Aug  1 12:42:38 slesha1n1i-u corosync[8905]:  [pcmk  ] notice:
pcmk_shutdown: Still waiting for crmd (pid=8916, seq=6) to terminate...
 Aug  1 12:43:08 slesha1n1i-u corosync[8905]:  [pcmk  ] notice:
pcmk_shutdown: Still waiting for crmd (pid=8916, seq=6) to terminate...

...

- Finally I kill the corosync process on Node 1 (killall -9 corosync), then
corsoync restarts.
- Checking status. All resources are up and running on Node 1, and the
stonith resource is running on Node 2 again.

 

I have tested the same scenario several times. Sometimes the fencing
mechaism works as expected, but other times the stonith resource is not
transferred to Node 1 - as described here. So I need some assistance to
overcome this problem....

 

--
Best regards
Jan 

-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/pacemaker/attachments/20130801/740280f4/attachment.htm>


More information about the Pacemaker mailing list