[ClusterLabs] How to set up fencing/stonith

Casey & Gina caseyandgina at icloud.com
Fri May 18 16:41:26 EDT 2018


> Having it started on one node is normal. Fence devices default to
> requires=quorum, meaning they can start on a new node even before the
> original node is fenced. It looks like that's what happened here, but
> something went wrong with the fencing, so the cluster assumes it's
> still active on the old node as well.

Okay, I was thinking it was normal to not show as running on all nodes per previous E-mails...  Could it be because I've already powered the VM off, that the fence command trying to power it off received a failure response?

> I'm not sure what went wrong with the fencing. Once fencing succeeds,
> the node should show up as offline without also being unclean. Anything
> interesting in the logs around the time of the fencing?

To test this out, I've tried a simpler setup with just the fencing resource and a VIP.

------
Online: [ d-gp2-dbpg0-1 d-gp2-dbpg0-2 d-gp2-dbpg0-3 ]

Full list of resources:

 vfencing       (stonith:external/vcenter):     Started d-gp2-dbpg0-2
 postgresql-master-vip  (ocf::heartbeat:IPaddr2):       Started d-gp2-dbpg0-1

PCSD Status:
  d-gp2-dbpg0-1: Online
  d-gp2-dbpg0-2: Online
  d-gp2-dbpg0-3: Online
------

Looks good with the VIP up on node 1.  After this, I log into vSphere and power off the primary node, which results in this state:

------
Node d-gp2-dbpg0-1: UNCLEAN (offline)
Online: [ d-gp2-dbpg0-2 d-gp2-dbpg0-3 ]

Full list of resources:

 vfencing       (stonith:external/vcenter):     Started d-gp2-dbpg0-2
 postgresql-master-vip  (ocf::heartbeat:IPaddr2):       Started d-gp2-dbpg0-1 (UNCLEAN)
------

Here's the log from node 2 during that time:

------
[4246] d-gp2-dbpg0-2 corosyncnotice  [TOTEM ] A processor failed, forming new configuration.
[4246] d-gp2-dbpg0-2 corosyncnotice  [TOTEM ] A new membership (10.124.164.63:96) was formed. Members left: 1
[4246] d-gp2-dbpg0-2 corosyncnotice  [TOTEM ] Failed to receive the leave message. failed: 1
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: pcmk_cpg_membership: Node 1 left group cib (peer=d-gp2-dbpg0-1, counter=6.0)
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: crm_update_peer_proc:        pcmk_cpg_membership: Node d-gp2-dbpg0-1[1] - corosync-cpg is now offline
May 18 20:36:16 [4284] d-gp2-dbpg0-2      attrd:     info: pcmk_cpg_membership: Node 1 left group attrd (peer=d-gp2-dbpg0-1, counter=6.0)
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:   notice: crm_update_peer_state_iter:  crm_update_peer_proc: Node d-gp2-dbpg0-1[1] - state is now lost (was member)
May 18 20:36:16 [4284] d-gp2-dbpg0-2      attrd:     info: crm_update_peer_proc:        pcmk_cpg_membership: Node d-gp2-dbpg0-1[1] - corosync-cpg is now offline
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:   notice: crm_reap_dead_member:        Removing d-gp2-dbpg0-1/1 from the membership list
May 18 20:36:16 [4284] d-gp2-dbpg0-2      attrd:   notice: crm_update_peer_state_iter:  crm_update_peer_proc: Node d-gp2-dbpg0-1[1] - state is now lost (was member)
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:   notice: reap_crm_member:     Purged 1 peers with id=1 and/or uname=d-gp2-dbpg0-1 from the membership cache
May 18 20:36:16 [4284] d-gp2-dbpg0-2      attrd:   notice: crm_reap_dead_member:        Removing d-gp2-dbpg0-1/1 from the membership list
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: pcmk_cpg_membership: Node 2 still member of group cib (peer=d-gp2-dbpg0-2, counter=6.0)
May 18 20:36:16 [4284] d-gp2-dbpg0-2      attrd:   notice: reap_crm_member:     Purged 1 peers with id=1 and/or uname=d-gp2-dbpg0-1 from the membership cache
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: pcmk_cpg_membership: Node 3 still member of group cib (peer=d-gp2-dbpg0-3, counter=6.1)
May 18 20:36:16 [4282] d-gp2-dbpg0-2 stonith-ng:     info: pcmk_cpg_membership: Node 1 left group stonith-ng (peer=d-gp2-dbpg0-1, counter=6.0)
May 18 20:36:16 [4284] d-gp2-dbpg0-2      attrd:     info: pcmk_cpg_membership: Node 2 still member of group attrd (peer=d-gp2-dbpg0-2, counter=6.0)
May 18 20:36:16 [4284] d-gp2-dbpg0-2      attrd:     info: pcmk_cpg_membership: Node 3 still member of group attrd (peer=d-gp2-dbpg0-3, counter=6.1)
May 18 20:36:16 [4282] d-gp2-dbpg0-2 stonith-ng:     info: crm_update_peer_proc:        pcmk_cpg_membership: Node d-gp2-dbpg0-1[1] - corosync-cpg is now offline
May 18 20:36:16 [4282] d-gp2-dbpg0-2 stonith-ng:   notice: crm_update_peer_state_iter:  crm_update_peer_proc: Node d-gp2-dbpg0-1[1] - state is now lost (was member)
May 18 20:36:16 [4278] d-gp2-dbpg0-2 pacemakerd:     info: pcmk_quorum_notification:    Membership 96: quorum retained (2)
May 18 20:36:16 [4278] d-gp2-dbpg0-2 pacemakerd:   notice: crm_update_peer_state_iter:  crm_reap_unseen_nodes: Node d-gp2-dbpg0-1[1] - state is now lost (was member)
May 18 20:36:16 [4282] d-gp2-dbpg0-2 stonith-ng:   notice: crm_reap_dead_member:        Removing d-gp2-dbpg0-1/1 from the membership list
May 18 20:36:16 [4278] d-gp2-dbpg0-2 pacemakerd:     info: pcmk_cpg_membership: Node 1 left group pacemakerd (peer=d-gp2-dbpg0-1, counter=5.0)
May 18 20:36:16 [4282] d-gp2-dbpg0-2 stonith-ng:   notice: reap_crm_member:     Purged 1 peers with id=1 and/or uname=d-gp2-dbpg0-1 from the membership cache
May 18 20:36:16 [4282] d-gp2-dbpg0-2 stonith-ng:     info: pcmk_cpg_membership: Node 2 still member of group stonith-ng (peer=d-gp2-dbpg0-2, counter=6.0)
May 18 20:36:16 [4278] d-gp2-dbpg0-2 pacemakerd:     info: crm_update_peer_proc:        pcmk_cpg_membership: Node d-gp2-dbpg0-1[1] - corosync-cpg is now offline
May 18 20:36:16 [4282] d-gp2-dbpg0-2 stonith-ng:     info: pcmk_cpg_membership: Node 3 still member of group stonith-ng (peer=d-gp2-dbpg0-3, counter=6.1)
May 18 20:36:16 [4278] d-gp2-dbpg0-2 pacemakerd:     info: pcmk_cpg_membership: Node 2 still member of group pacemakerd (peer=d-gp2-dbpg0-2, counter=5.0)
May 18 20:36:16 [4278] d-gp2-dbpg0-2 pacemakerd:     info: pcmk_cpg_membership: Node 3 still member of group pacemakerd (peer=d-gp2-dbpg0-3, counter=5.1)
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_perform_op:      Diff: --- 0.35.5 2
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_perform_op:      Diff: +++ 0.35.6 (null)
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_perform_op:      +  /cib:  @num_updates=6
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_perform_op:      +  /cib/status/node_state[@id='1']:  @crm-debug-origin=peer_update_callback
May 18 20:36:16 [4286] d-gp2-dbpg0-2       crmd:     info: pcmk_quorum_notification:    Membership 96: quorum retained (2)
May 18 20:36:16 [4286] d-gp2-dbpg0-2       crmd:   notice: crm_update_peer_state_iter:  crm_reap_unseen_nodes: Node d-gp2-dbpg0-1[1] - state is now lost (was member)
May 18 20:36:16 [4286] d-gp2-dbpg0-2       crmd:     info: pcmk_cpg_membership: Node 1 left group crmd (peer=d-gp2-dbpg0-1, counter=6.0)
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=d-gp2-dbpg0-3/crmd/164, version=0.35.6)
May 18 20:36:16 [4286] d-gp2-dbpg0-2       crmd:     info: crm_update_peer_proc:        pcmk_cpg_membership: Node d-gp2-dbpg0-1[1] - corosync-cpg is now offline
May 18 20:36:16 [4286] d-gp2-dbpg0-2       crmd:     info: pcmk_cpg_membership: Node 2 still member of group crmd (peer=d-gp2-dbpg0-2, counter=6.0)
May 18 20:36:16 [4286] d-gp2-dbpg0-2       crmd:     info: pcmk_cpg_membership: Node 3 still member of group crmd (peer=d-gp2-dbpg0-3, counter=6.1)
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=d-gp2-dbpg0-3/crmd/168, version=0.35.6)
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_perform_op:      Diff: --- 0.35.6 2
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_perform_op:      Diff: +++ 0.35.7 (null)
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_perform_op:      +  /cib:  @num_updates=7
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_perform_op:      +  /cib/status/node_state[@id='1']:  @in_ccm=false, @crm-debug-origin=post_cache_update
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_perform_op:      +  /cib/status/node_state[@id='2']:  @crm-debug-origin=post_cache_update
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_perform_op:      +  /cib/status/node_state[@id='3']:  @crm-debug-origin=post_cache_update
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=d-gp2-dbpg0-3/crmd/169, version=0.35.7)
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_perform_op:      Diff: --- 0.35.7 2
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_perform_op:      Diff: +++ 0.35.8 (null)
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_perform_op:      -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_perform_op:      +  /cib:  @num_updates=8
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_process_request: Completed cib_delete operation for section //node_state[@uname='d-gp2-dbpg0-1']/transient_attributes: OK (rc=0, origin=d-gp2-dbpg0-3/crmd/171, version=0.35.8)
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_perform_op:      Diff: --- 0.35.8 2
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_perform_op:      Diff: +++ 0.35.9 (null)
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_perform_op:      +  /cib:  @num_updates=9
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_perform_op:      +  /cib/status/node_state[@id='1']:  @crmd=offline, @crm-debug-origin=peer_update_callback
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=d-gp2-dbpg0-3/crmd/172, version=0.35.9)
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=d-gp2-dbpg0-3/crmd/176, version=0.35.9)
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_perform_op:      Diff: --- 0.35.9 2
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_perform_op:      Diff: +++ 0.35.10 (null)
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_perform_op:      +  /cib:  @num_updates=10
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_perform_op:      +  /cib/status/node_state[@id='1']:  @crm-debug-origin=do_state_transition, @join=down
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_perform_op:      +  /cib/status/node_state[@id='2']:  @crm-debug-origin=do_state_transition
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_perform_op:      +  /cib/status/node_state[@id='3']:  @crm-debug-origin=do_state_transition
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=d-gp2-dbpg0-3/crmd/177, version=0.35.10)
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_process_request: Completed cib_modify operation for section cib: OK (rc=0, origin=d-gp2-dbpg0-3/crmd/178, version=0.35.10)
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_perform_op:      Diff: --- 0.35.10 2
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_perform_op:      Diff: +++ 0.35.11 (null)
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_perform_op:      +  /cib:  @num_updates=11
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=d-gp2-dbpg0-3/attrd/10, version=0.35.11)
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_perform_op:      Diff: --- 0.35.11 2
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_perform_op:      Diff: +++ 0.35.12 (null)
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_perform_op:      +  /cib:  @num_updates=12
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=d-gp2-dbpg0-3/attrd/11, version=0.35.12)
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=d-gp2-dbpg0-3/attrd/12, version=0.35.12)
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_perform_op:      Diff: --- 0.35.12 2
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_perform_op:      Diff: +++ 0.35.13 (null)
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_perform_op:      +  /cib:  @num_updates=13
May 18 20:36:16 [4281] d-gp2-dbpg0-2        cib:     info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=d-gp2-dbpg0-3/attrd/13, version=0.35.13)
May 18 20:36:23 d-gp2-dbpg0-2 stonith: [16244]: CRIT: external_reset_req: 'vcenter reset' for host d-gp2-dbpg0-1 failed with rc 255
May 18 20:36:23 [4282] d-gp2-dbpg0-2 stonith-ng:  warning: log_action:  fence_legacy[16243] stderr: [ Died at /usr/lib/stonith/plugins/external/vcenter line 22. ]
May 18 20:36:23 [4282] d-gp2-dbpg0-2 stonith-ng:  warning: log_action:  fence_legacy[16243] stderr: [   ...propagated at /usr/lib/stonith/plugins/external/vcenter line 22. ]
May 18 20:36:23 [4282] d-gp2-dbpg0-2 stonith-ng:     info: internal_stonith_action_execute:     Attempt 2 to execute fence_legacy (reboot). remaining timeout is 57
May 18 20:36:27 d-gp2-dbpg0-2 stonith: [16265]: CRIT: external_reset_req: 'vcenter reset' for host d-gp2-dbpg0-1 failed with rc 255
May 18 20:36:27 [4282] d-gp2-dbpg0-2 stonith-ng:  warning: log_action:  fence_legacy[16264] stderr: [ Died at /usr/lib/stonith/plugins/external/vcenter line 22. ]
May 18 20:36:27 [4282] d-gp2-dbpg0-2 stonith-ng:  warning: log_action:  fence_legacy[16264] stderr: [   ...propagated at /usr/lib/stonith/plugins/external/vcenter line 22. ]
May 18 20:36:27 [4282] d-gp2-dbpg0-2 stonith-ng:     info: update_remaining_timeout:    Attempted to execute agent fence_legacy (reboot) the maximum number of times (2) allowed
May 18 20:36:27 [4282] d-gp2-dbpg0-2 stonith-ng:    error: log_operation:       Operation 'reboot' [16264] (call 4 from crmd.4267) for host 'd-gp2-dbpg0-1' with device 'vfencing' returned: -201 (Generic Pacemaker error)
May 18 20:36:27 [4282] d-gp2-dbpg0-2 stonith-ng:  warning: log_operation:       vfencing:16264 [ Performing: stonith -t external/vcenter -T reset d-gp2-dbpg0-1 ]
May 18 20:36:27 [4282] d-gp2-dbpg0-2 stonith-ng:  warning: log_operation:       vfencing:16264 [ failed: d-gp2-dbpg0-1 5 ]
May 18 20:36:35 [4282] d-gp2-dbpg0-2 stonith-ng:   notice: remote_op_done:      Operation reboot of d-gp2-dbpg0-1 by <no-one> for crmd.4267 at d-gp2-dbpg0-3.7d21135c: No route to host
May 18 20:36:23 [4282] d-gp2-dbpg0-2 stonith-ng:  warning: log_action:  fence_legacy[16243] stderr: [ Died at /usr/lib/stonith/plugins/external/vcenter line 22. ]
May 18 20:36:23 [4282] d-gp2-dbpg0-2 stonith-ng:  warning: log_action:  fence_legacy[16243] stderr: [   ...propagated at /usr/lib/stonith/plugins/external/vcenter line 22. ]
May 18 20:36:23 [4282] d-gp2-dbpg0-2 stonith-ng:     info: internal_stonith_action_execute:     Attempt 2 to execute fence_legacy (reboot). remaining timeout is 57
May 18 20:36:27 d-gp2-dbpg0-2 stonith: [16265]: CRIT: external_reset_req: 'vcenter reset' for host d-gp2-dbpg0-1 failed with rc 255
May 18 20:36:27 [4282] d-gp2-dbpg0-2 stonith-ng:  warning: log_action:  fence_legacy[16264] stderr: [ Died at /usr/lib/stonith/plugins/external/vcenter line 22. ]
May 18 20:36:27 [4282] d-gp2-dbpg0-2 stonith-ng:  warning: log_action:  fence_legacy[16264] stderr: [   ...propagated at /usr/lib/stonith/plugins/external/vcenter line 22. ]
May 18 20:36:27 [4282] d-gp2-dbpg0-2 stonith-ng:     info: update_remaining_timeout:    Attempted to execute agent fence_legacy (reboot) the maximum number of times (2) allowed
May 18 20:36:27 [4282] d-gp2-dbpg0-2 stonith-ng:    error: log_operation:       Operation 'reboot' [16264] (call 4 from crmd.4267) for host 'd-gp2-dbpg0-1' with device 'vfencing' returned: -201 (Generic Pacemaker error)
May 18 20:36:27 [4282] d-gp2-dbpg0-2 stonith-ng:  warning: log_operation:       vfencing:16264 [ Performing: stonith -t external/vcenter -T reset d-gp2-dbpg0-1 ]
May 18 20:36:27 [4282] d-gp2-dbpg0-2 stonith-ng:  warning: log_operation:       vfencing:16264 [ failed: d-gp2-dbpg0-1 5 ]
May 18 20:36:35 [4282] d-gp2-dbpg0-2 stonith-ng:   notice: remote_op_done:      Operation reboot of d-gp2-dbpg0-1 by <no-one> for crmd.4267 at d-gp2-dbpg0-3.7d21135c: No route to host
May 18 20:36:41 d-gp2-dbpg0-2 stonith: [16323]: CRIT: external_reset_req: 'vcenter reset' for host d-gp2-dbpg0-1 failed with rc 255
May 18 20:36:41 [4282] d-gp2-dbpg0-2 stonith-ng:  warning: log_action:  fence_legacy[16322] stderr: [ Died at /usr/lib/stonith/plugins/external/vcenter line 22. ]
May 18 20:36:41 [4282] d-gp2-dbpg0-2 stonith-ng:  warning: log_action:  fence_legacy[16322] stderr: [   ...propagated at /usr/lib/stonith/plugins/external/vcenter line 22. ]
May 18 20:36:41 [4282] d-gp2-dbpg0-2 stonith-ng:     info: internal_stonith_action_execute:     Attempt 2 to execute fence_legacy (reboot). remaining timeout is 57
May 18 20:36:45 d-gp2-dbpg0-2 stonith: [16344]: CRIT: external_reset_req: 'vcenter reset' for host d-gp2-dbpg0-1 failed with rc 255
May 18 20:36:45 [4282] d-gp2-dbpg0-2 stonith-ng:  warning: log_action:  fence_legacy[16343] stderr: [ Died at /usr/lib/stonith/plugins/external/vcenter line 22. ]
May 18 20:36:45 [4282] d-gp2-dbpg0-2 stonith-ng:  warning: log_action:  fence_legacy[16343] stderr: [   ...propagated at /usr/lib/stonith/plugins/external/vcenter line 22. ]
May 18 20:36:45 [4282] d-gp2-dbpg0-2 stonith-ng:     info: update_remaining_timeout:    Attempted to execute agent fence_legacy (reboot) the maximum number of times (2) allowed
May 18 20:36:45 [4282] d-gp2-dbpg0-2 stonith-ng:    error: log_operation:       Operation 'reboot' [16343] (call 5 from crmd.4267) for host 'd-gp2-dbpg0-1' with device 'vfencing' returned: -201 (Generic Pacemaker error)
May 18 20:36:45 [4282] d-gp2-dbpg0-2 stonith-ng:  warning: log_operation:       vfencing:16343 [ Performing: stonith -t external/vcenter -T reset d-gp2-dbpg0-1 ]
May 18 20:36:45 [4282] d-gp2-dbpg0-2 stonith-ng:  warning: log_operation:       vfencing:16343 [ failed: d-gp2-dbpg0-1 5 ]
May 18 20:36:53 [4282] d-gp2-dbpg0-2 stonith-ng:   notice: remote_op_done:      Operation reboot of d-gp2-dbpg0-1 by <no-one> for crmd.4267 at d-gp2-dbpg0-3.e29edf29: No route to host
May 18 20:36:59 d-gp2-dbpg0-2 stonith: [16383]: CRIT: external_reset_req: 'vcenter reset' for host d-gp2-dbpg0-1 failed with rc 255
May 18 20:36:59 [4282] d-gp2-dbpg0-2 stonith-ng:  warning: log_action:  fence_legacy[16382] stderr: [ Died at /usr/lib/stonith/plugins/external/vcenter line 22. ]
May 18 20:36:59 [4282] d-gp2-dbpg0-2 stonith-ng:  warning: log_action:  fence_legacy[16382] stderr: [   ...propagated at /usr/lib/stonith/plugins/external/vcenter line 22. ]
May 18 20:36:59 [4282] d-gp2-dbpg0-2 stonith-ng:     info: internal_stonith_action_execute:     Attempt 2 to execute fence_legacy (reboot). remaining timeout is 57
May 18 20:37:03 d-gp2-dbpg0-2 stonith: [16404]: CRIT: external_reset_req: 'vcenter reset' for host d-gp2-dbpg0-1 failed with rc 255
May 18 20:37:03 [4282] d-gp2-dbpg0-2 stonith-ng:  warning: log_action:  fence_legacy[16403] stderr: [ Died at /usr/lib/stonith/plugins/external/vcenter line 22. ]
May 18 20:37:03 [4282] d-gp2-dbpg0-2 stonith-ng:  warning: log_action:  fence_legacy[16403] stderr: [   ...propagated at /usr/lib/stonith/plugins/external/vcenter line 22. ]
May 18 20:37:03 [4282] d-gp2-dbpg0-2 stonith-ng:     info: update_remaining_timeout:    Attempted to execute agent fence_legacy (reboot) the maximum number of times (2) allowed
May 18 20:37:03 [4282] d-gp2-dbpg0-2 stonith-ng:    error: log_operation:       Operation 'reboot' [16403] (call 6 from crmd.4267) for host 'd-gp2-dbpg0-1' with device 'vfencing' returned: -201 (Generic Pacemaker error)
May 18 20:37:03 [4282] d-gp2-dbpg0-2 stonith-ng:  warning: log_operation:       vfencing:16403 [ Performing: stonith -t external/vcenter -T reset d-gp2-dbpg0-1 ]
May 18 20:37:03 [4282] d-gp2-dbpg0-2 stonith-ng:  warning: log_operation:       vfencing:16403 [ failed: d-gp2-dbpg0-1 5 ]
May 18 20:37:11 [4282] d-gp2-dbpg0-2 stonith-ng:   notice: remote_op_done:      Operation reboot of d-gp2-dbpg0-1 by <no-one> for crmd.4267 at d-gp2-dbpg0-3.25d3b994: No route to host
[...continues indefinitely...]
------

Thanks,
-- 
Casey


More information about the Users mailing list