Dec 16 15:08:54 [642] xstorage2 corosync notice  [TOTEM ] A processor failed, forming new configuration.
Dec 16 15:08:56 [642] xstorage2 corosync notice  [TOTEM ] A new membership (10.100.100.2:408) was formed. Members left: 1
Dec 16 15:08:56 [642] xstorage2 corosync notice  [TOTEM ] Failed to receive the leave message. failed: 1
Dec 16 15:08:56 [666]      attrd:     info: pcmk_cpg_membership:        Group attrd event 2: xstha1 (node 1 pid 710) left via cluster exit
Dec 16 15:08:56 [663]        cib:     info: pcmk_cpg_membership:        Group cib event 2: xstha1 (node 1 pid 707) left via cluster exit
Dec 16 15:08:56 [662] pacemakerd:     info: pcmk_cpg_membership:        Group pacemakerd event 2: xstha1 (node 1 pid 687) left via cluster exit
Dec 16 15:08:56 [642] xstorage2 corosync notice  [QUORUM] Members[1]: 2
Dec 16 15:08:56 [662] pacemakerd:     info: crm_update_peer_proc:       pcmk_cpg_membership: Node xstha1[1] - corosync-cpg is now offline
Dec 16 15:08:56 [666]      attrd:     info: crm_update_peer_proc:       pcmk_cpg_membership: Node xstha1[1] - corosync-cpg is now offline
Dec 16 15:08:56 [662] pacemakerd:     info: pcmk_cpg_membership:        Group pacemakerd event 2: xstha2 (node 2 pid 662) is member
Dec 16 15:08:56 [642] xstorage2 corosync notice  [MAIN  ] Completed service synchronization, ready to provide service.
Dec 16 15:08:56 [668]       crmd:     info: pcmk_cpg_membership:        Group crmd event 2: xstha1 (node 1 pid 712) left via cluster exit
Dec 16 15:08:56 [664] stonith-ng:     info: pcmk_cpg_membership:        Group stonith-ng event 2: xstha1 (node 1 pid 708) left via cluster exit
Dec 16 15:08:56 [663]        cib:     info: crm_update_peer_proc:       pcmk_cpg_membership: Node xstha1[1] - corosync-cpg is now offline
Dec 16 15:08:56 [668]       crmd:     info: crm_update_peer_proc:       pcmk_cpg_membership: Node xstha1[1] - corosync-cpg is now offline
Dec 16 15:08:56 [666]      attrd:   notice: attrd_remove_voter: Lost attribute writer xstha1
Dec 16 15:08:56 [664] stonith-ng:     info: crm_update_peer_proc:       pcmk_cpg_membership: Node xstha1[1] - corosync-cpg is now offline
Dec 16 15:08:56 [662] pacemakerd:     info: pcmk_quorum_notification:   Quorum retained | membership=408 members=1
Dec 16 15:08:56 [663]        cib:   notice: crm_update_peer_state_iter: Node xstha1 state is now lost | nodeid=1 previous=member source=crm_update_peer_proc
Dec 16 15:08:56 [664] stonith-ng:   notice: crm_update_peer_state_iter: Node xstha1 state is now lost | nodeid=1 previous=member source=crm_update_peer_proc
Dec 16 15:08:56 [662] pacemakerd:   notice: crm_update_peer_state_iter: Node xstha1 state is now lost | nodeid=1 previous=member source=crm_reap_unseen_nodes
Dec 16 15:08:56 [668]       crmd:     info: peer_update_callback:       Client xstha1/peer now has status [offline] (DC=xstha1, changed=4000000)
Dec 16 15:08:56 [663]        cib:     info: crm_reap_dead_member:       Removing node with name xstha1 and id 1 from membership cache
Dec 16 15:08:56 [666]      attrd:     info: attrd_start_election_if_needed:     Starting an election to determine the writer
Dec 16 15:08:56 [663]        cib:   notice: reap_crm_member:    Purged 1 peer with id=1 and/or uname=xstha1 from the membership cache
Dec 16 15:08:56 [668]       crmd:   notice: peer_update_callback:       Our peer on the DC (xstha1) is dead
Dec 16 15:08:56 [663]        cib:     info: pcmk_cpg_membership:        Group cib event 2: xstha2 (node 2 pid 663) is member
Dec 16 15:08:56 [664] stonith-ng:     info: crm_reap_dead_member:       Removing node with name xstha1 and id 1 from membership cache
Dec 16 15:08:56 [662] pacemakerd:     info: mcp_cpg_deliver:    Ignoring process list sent by peer for local node
Dec 16 15:08:56 [666]      attrd:   notice: crm_update_peer_state_iter: Node xstha1 state is now lost | nodeid=1 previous=member source=crm_update_peer_proc
Dec 16 15:08:56 [664] stonith-ng:   notice: reap_crm_member:    Purged 1 peer with id=1 and/or uname=xstha1 from the membership cache
Dec 16 15:08:56 [668]       crmd:     info: controld_delete_node_state: Deleting transient attributes for node xstha1 (via CIB call 18) | xpath=//node_state[@uname='xstha1']/transient_attributes
Dec 16 15:08:56 [664] stonith-ng:     info: pcmk_cpg_membership:        Group stonith-ng event 2: xstha2 (node 2 pid 664) is member
Dec 16 15:08:56 [666]      attrd:   notice: attrd_peer_remove:  Removing all xstha1 attributes for peer loss
Dec 16 15:08:56 [668]       crmd:     info: pcmk_cpg_membership:        Group crmd event 2: xstha2 (node 2 pid 668) is member
Dec 16 15:08:56 [663]        cib:     info: cib_process_request:        Forwarding cib_delete operation for section //node_state[@uname='xstha1']/transient_attributes to all (origin=local/crmd/18)
Dec 16 15:08:56 [666]      attrd:     info: crm_reap_dead_member:       Removing node with name xstha1 and id 1 from membership cache
Dec 16 15:08:56 [668]       crmd:   notice: do_state_transition:        State transition S_NOT_DC -> S_ELECTION | input=I_ELECTION cause=C_CRMD_STATUS_CALLBACK origin=peer_update_callback
Dec 16 15:08:56 [666]      attrd:   notice: reap_crm_member:    Purged 1 peer with id=1 and/or uname=xstha1 from the membership cache
Dec 16 15:08:56 [668]       crmd:     info: update_dc:  Unset DC. Was xstha1
Dec 16 15:08:56 [666]      attrd:     info: pcmk_cpg_membership:        Group attrd event 2: xstha2 (node 2 pid 666) is member
Dec 16 15:08:56 [666]      attrd:     info: election_check:     election-attrd won by local node
Dec 16 15:08:56 [668]       crmd:     info: pcmk_quorum_notification:   Quorum retained | membership=408 members=1
Dec 16 15:08:56 [666]      attrd:   notice: attrd_declare_winner:       Recorded local node as attribute writer (was unset)
Dec 16 15:08:56 [668]       crmd:   notice: crm_update_peer_state_iter: Node xstha1 state is now lost | nodeid=1 previous=member source=crm_reap_unseen_nodes
Dec 16 15:08:56 [668]       crmd:     info: peer_update_callback:       Cluster node xstha1 is now lost (was member)
Dec 16 15:08:56 [666]      attrd:     info: write_attribute:    Processed 1 private change for #attrd-protocol, id=n/a, set=n/a
Dec 16 15:08:56 [668]       crmd:     info: election_check:     election-DC won by local node
Dec 16 15:08:56 [668]       crmd:     info: do_log:     Input I_ELECTION_DC received in state S_ELECTION from election_win_cb
Dec 16 15:08:56 [668]       crmd:   notice: do_state_transition:        State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=election_win_cb
Dec 16 15:08:56 [668]       crmd:     info: do_te_control:      Registering TE UUID: f340fcfc-17fa-ebf0-c5bf-8299546d41b6
Dec 16 15:08:56 [663]        cib:     info: cib_process_request:        Completed cib_delete operation for section //node_state[@uname='xstha1']/transient_attributes: OK (rc=0, origin=xstha2/crmd/18, version=0.46.19)
Dec 16 15:08:56 [668]       crmd:     info: set_graph_functions:        Setting custom graph functions
Dec 16 15:08:56 [668]       crmd:     info: do_dc_takeover:     Taking over DC status for this partition
Dec 16 15:08:56 [663]        cib:     info: cib_process_readwrite:      We are now in R/W mode
Dec 16 15:08:56 [663]        cib:     info: cib_process_request:        Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/19, version=0.46.19)
Dec 16 15:08:56 [663]        cib:     info: cib_process_request:        Forwarding cib_modify operation for section cib to all (origin=local/crmd/20)
Dec 16 15:08:56 [663]        cib:     info: cib_process_request:        Completed cib_modify operation for section cib: OK (rc=0, origin=xstha2/crmd/20, version=0.46.19)
Dec 16 15:08:56 [663]        cib:     info: cib_process_request:        Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/22)
Dec 16 15:08:56 [663]        cib:     info: cib_process_request:        Completed cib_modify operation for section crm_config: OK (rc=0, origin=xstha2/crmd/22, version=0.46.19)
Dec 16 15:08:56 [663]        cib:     info: cib_process_request:        Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/24)
Dec 16 15:08:56 [663]        cib:     info: cib_process_request:        Completed cib_modify operation for section crm_config: OK (rc=0, origin=xstha2/crmd/24, version=0.46.19)
Dec 16 15:08:56 [663]        cib:     info: cib_process_request:        Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/26)
Dec 16 15:08:56 [668]       crmd:     info: corosync_cluster_name:      Cannot get totem.cluster_name: CS_ERR_NOT_EXIST  (12)
Dec 16 15:08:56 [668]       crmd:     info: join_make_offer:    Making join-1 offers based on membership event 408
Dec 16 15:08:56 [668]       crmd:     info: join_make_offer:    Sending join-1 offer to xstha2
Dec 16 15:08:56 [668]       crmd:     info: join_make_offer:    Not making join-1 offer to inactive node xstha1
Dec 16 15:08:56 [668]       crmd:     info: do_dc_join_offer_all:       Waiting on join-1 requests from 1 outstanding node
Dec 16 15:08:56 [663]        cib:     info: cib_process_request:        Completed cib_modify operation for section crm_config: OK (rc=0, origin=xstha2/crmd/26, version=0.46.19)
Dec 16 15:08:56 [668]       crmd:     info: update_dc:  Set DC to xstha2 (3.0.14)
Dec 16 15:08:56 [668]       crmd:     info: crm_update_peer_expected:   update_dc: Node xstha2[2] - expected state is now member (was (null))
Dec 16 15:08:56 [668]       crmd:     info: do_state_transition:        State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state
Dec 16 15:08:56 [663]        cib:     info: cib_process_replace:        Digest matched on replace from xstha2: 4835352cb7b4920917d8beee219bc962
Dec 16 15:08:56 [663]        cib:     info: cib_process_replace:        Replaced 0.46.19 with 0.46.19 from xstha2
Dec 16 15:08:56 [668]       crmd:     info: controld_delete_node_state: Deleting resource history for node xstha2 (via CIB call 31) | xpath=//node_state[@uname='xstha2']/lrm
Dec 16 15:08:56 [663]        cib:     info: cib_process_request:        Completed cib_replace operation for section 'all': OK (rc=0, origin=xstha2/crmd/29, version=0.46.19)
Dec 16 15:08:56 [663]        cib:     info: cib_process_request:        Forwarding cib_modify operation for section nodes to all (origin=local/crmd/30)
Dec 16 15:08:56 [663]        cib:     info: cib_process_request:        Forwarding cib_delete operation for section //node_state[@uname='xstha2']/lrm to all (origin=local/crmd/31)
Dec 16 15:08:56 [663]        cib:     info: cib_process_request:        Forwarding cib_modify operation for section status to all (origin=local/crmd/32)
Dec 16 15:08:56 [663]        cib:     info: cib_process_request:        Completed cib_modify operation for section nodes: OK (rc=0, origin=xstha2/crmd/30, version=0.46.19)
Dec 16 15:08:56 [663]        cib:     info: cib_perform_op:     Diff: --- 0.46.19 2
Dec 16 15:08:56 [663]        cib:     info: cib_perform_op:     Diff: +++ 0.46.20 (null)
Dec 16 15:08:56 [663]        cib:     info: cib_perform_op:     -- /cib/status/node_state[@id='2']/lrm[@id='2']
Dec 16 15:08:56 [663]        cib:     info: cib_perform_op:     +  /cib:  @num_updates=20
Dec 16 15:08:56 [663]        cib:     info: cib_process_request:        Completed cib_delete operation for section //node_state[@uname='xstha2']/lrm: OK (rc=0, origin=xstha2/crmd/31, version=0.46.20)
Dec 16 15:08:56 [663]        cib:     info: cib_perform_op:     Diff: --- 0.46.20 2
Dec 16 15:08:56 [663]        cib:     info: cib_perform_op:     Diff: +++ 0.46.21 (null)
Dec 16 15:08:56 [663]        cib:     info: cib_perform_op:     +  /cib:  @num_updates=21
Dec 16 15:08:56 [663]        cib:     info: cib_perform_op:     +  /cib/status/node_state[@id='2']:  @crm-debug-origin=do_lrm_query_internal
Dec 16 15:08:56 [663]        cib:     info: cib_perform_op:     ++ /cib/status/node_state[@id='2']:  <lrm id="2"/>
Dec 16 15:08:56 [663]        cib:     info: cib_perform_op:     ++                                     <lrm_resources>
Dec 16 15:08:56 [663]        cib:     info: cib_perform_op:     ++                                       <lrm_resource id="zpool_data" type="ZFS" class="ocf" provider="heartbeat">
Dec 16 15:08:56 [663]        cib:     info: cib_perform_op:     ++                                         <lrm_rsc_op id="zpool_data_last_0" operation_key="zpool_data_monitor_0" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="3:3:7:cc8faf12-ac24-cc9c-c212-effe6840ca76" transition-magic="0:7;3:3:7:cc8faf12-ac24-cc9c-c212-effe6840ca76" exit-reason="" on_node="xstha2" call-id="13" rc-code="7" op-status="0" interval="0" last-run="1608127496" last-rc-change="1608127
Dec 16 15:08:56 [663]        cib:     info: cib_perform_op:     ++                                       </lrm_resource>
Dec 16 15:08:56 [663]        cib:     info: cib_perform_op:     ++                                       <lrm_resource id="xstha1-stonith" type="external/ipmi" class="stonith">
Dec 16 15:08:56 [663]        cib:     info: cib_perform_op:     ++                                         <lrm_rsc_op id="xstha1-stonith_last_0" operation_key="xstha1-stonith_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="9:3:0:cc8faf12-ac24-cc9c-c212-effe6840ca76" transition-magic="0:0;9:3:0:cc8faf12-ac24-cc9c-c212-effe6840ca76" exit-reason="" on_node="xstha2" call-id="22" rc-code="0" op-status="0" interval="0" last-run="1608127496" last-rc-change="160
Dec 16 15:08:56 [663]        cib:     info: cib_perform_op:     ++                                         <lrm_rsc_op id="xstha1-stonith_monitor_25000" operation_key="xstha1-stonith_monitor_25000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="10:3:0:cc8faf12-ac24-cc9c-c212-effe6840ca76" transition-magic="0:0;10:3:0:cc8faf12-ac24-cc9c-c212-effe6840ca76" exit-reason="" on_node="xstha2" call-id="24" rc-code="0" op-status="0" interval="25000" last-rc-change="1608
Dec 16 15:08:56 [663]        cib:     info: cib_perform_op:     ++                                       </lrm_resource>
Dec 16 15:08:56 [663]        cib:     info: cib_perform_op:     ++                                       <lrm_resource id="xstha2-stonith" type="external/ipmi" class="stonith">
Dec 16 15:08:56 [663]        cib:     info: cib_perform_op:     ++                                         <lrm_rsc_op id="xstha2-stonith_last_0" operation_key="xstha2-stonith_monitor_0" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="5:3:7:cc8faf12-ac24-cc9c-c212-effe6840ca76" transition-magic="0:7;5:3:7:cc8faf12-ac24-cc9c-c212-effe6840ca76" exit-reason="" on_node="xstha2" call-id="21" rc-code="7" op-status="0" interval="0" last-run="1608127496" last-rc-change=
Dec 16 15:08:56 [663]        cib:     info: cib_perform_op:     ++                                       </lrm_resource>
Dec 16 15:08:56 [663]        cib:     info: cib_perform_op:     ++                                       <lrm_resource id="xstha1_san0_IP" type="IPaddr" class="ocf" provider="heartbeat">
Dec 16 15:08:56 [663]        cib:     info: cib_perform_op:     ++                                         <lrm_rsc_op id="xstha1_san0_IP_last_0" operation_key="xstha1_san0_IP_monitor_0" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="1:3:7:cc8faf12-ac24-cc9c-c212-effe6840ca76" transition-magic="0:7;1:3:7:cc8faf12-ac24-cc9c-c212-effe6840ca76" exit-reason="" on_node="xstha2" call-id="5" rc-code="7" op-status="0" interval="0" last-run="1608127496" last-rc-change="
Dec 16 15:08:56 [663]        cib:     info: cib_perform_op:     ++                                       </lrm_resource>
Dec 16 15:08:56 [663]        cib:     info: cib_perform_op:     ++                                       <lrm_resource id="xstha2_san0_IP" type="IPaddr" class="ocf" provider="heartbeat">
Dec 16 15:08:56 [663]        cib:     info: cib_perform_op:     ++                                         <lrm_rsc_op id="xstha2_san0_IP_last_0" operation_key="xstha2_san0_IP_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="7:3:0:cc8faf12-ac24-cc9c-c212-effe6840ca76" transition-magic="0:0;7:3:0:cc8faf12-ac24-cc9c-c212-effe6840ca76" exit-reason="" on_node="xstha2" call-id="23" rc-code="0" op-status="0" interval="0" last-run="1608127497" last-rc-change="160
Dec 16 15:08:56 [663]        cib:     info: cib_perform_op:     ++                                       </lrm_resource>
Dec 16 15:08:56 [663]        cib:     info: cib_perform_op:     ++                                     </lrm_resources>
Dec 16 15:08:56 [663]        cib:     info: cib_perform_op:     ++                                   </lrm>
Dec 16 15:08:56 [663]        cib:     info: cib_process_request:        Completed cib_modify operation for section status: OK (rc=0, origin=xstha2/crmd/32, version=0.46.21)
Dec 16 15:08:56 [668]       crmd:     info: do_state_transition:        State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state
Dec 16 15:08:56 [668]       crmd:     info: abort_transition_graph:     Transition aborted: Peer Cancelled | source=do_te_invoke:143 complete=true
Dec 16 15:08:56 [663]        cib:     info: cib_process_request:        Forwarding cib_modify operation for section nodes to all (origin=local/crmd/35)
Dec 16 15:08:56 [663]        cib:     info: cib_process_request:        Forwarding cib_modify operation for section status to all (origin=local/crmd/36)
Dec 16 15:08:56 [663]        cib:     info: cib_process_request:        Forwarding cib_modify operation for section cib to all (origin=local/crmd/37)
Dec 16 15:08:56 [663]        cib:     info: cib_process_request:        Completed cib_modify operation for section nodes: OK (rc=0, origin=xstha2/crmd/35, version=0.46.21)
Dec 16 15:08:56 [663]        cib:     info: cib_perform_op:     Diff: --- 0.46.21 2
Dec 16 15:08:56 [663]        cib:     info: cib_perform_op:     Diff: +++ 0.46.22 (null)
Dec 16 15:08:56 [663]        cib:     info: cib_perform_op:     +  /cib:  @num_updates=22
Dec 16 15:08:56 [663]        cib:     info: cib_perform_op:     +  /cib/status/node_state[@id='1']:  @in_ccm=false, @crmd=offline, @crm-debug-origin=do_state_transition, @join=down
Dec 16 15:08:56 [663]        cib:     info: cib_perform_op:     +  /cib/status/node_state[@id='2']:  @crm-debug-origin=do_state_transition
Dec 16 15:08:56 [663]        cib:     info: cib_process_request:        Completed cib_modify operation for section status: OK (rc=0, origin=xstha2/crmd/36, version=0.46.22)
Dec 16 15:08:56 [663]        cib:     info: cib_perform_op:     Diff: --- 0.46.22 2
Dec 16 15:08:56 [663]        cib:     info: cib_perform_op:     Diff: +++ 0.46.23 (null)
Dec 16 15:08:56 [663]        cib:     info: cib_perform_op:     +  /cib:  @num_updates=23, @dc-uuid=2
Dec 16 15:08:56 [663]        cib:     info: cib_file_backup:    Archived previous version as /sonicle/var/cluster/lib/pacemaker/cib/cib-8.raw
Dec 16 15:08:56 [663]        cib:     info: cib_process_request:        Completed cib_modify operation for section cib: OK (rc=0, origin=xstha2/crmd/37, version=0.46.23)
Dec 16 15:08:56 [663]        cib:     info: cib_file_write_with_digest: Wrote version 0.46.0 of the CIB to disk (digest: 1ea3e3ee6c388f74623494869acf32d0)
Dec 16 15:08:56 [663]        cib:     info: cib_file_write_with_digest: Reading cluster configuration file /sonicle/var/cluster/lib/pacemaker/cib/cib.4LaWbc (digest: /sonicle/var/cluster/lib/pacemaker/cib/cib.5LaWbc)
Dec 16 15:08:56 [667]    pengine:  warning: unpack_config:      Support for stonith-action of 'poweroff' is deprecated and will be removed in a future release (use 'off' instead)
Dec 16 15:08:56 [667]    pengine:  warning: pe_fence_node:      Cluster node xstha1 will be fenced: peer is no longer part of the cluster
Dec 16 15:08:56 [667]    pengine:  warning: determine_online_status:    Node xstha1 is unclean
Dec 16 15:08:56 [667]    pengine:     info: determine_online_status_fencing:    Node xstha2 is active
Dec 16 15:08:56 [667]    pengine:     info: determine_online_status:    Node xstha2 is online
Dec 16 15:08:56 [667]    pengine:     info: unpack_node_loop:   Node 1 is already processed
Dec 16 15:08:56 [667]    pengine:     info: unpack_node_loop:   Node 2 is already processed
Dec 16 15:08:56 [667]    pengine:     info: unpack_node_loop:   Node 1 is already processed
Dec 16 15:08:56 [667]    pengine:     info: unpack_node_loop:   Node 2 is already processed
Dec 16 15:08:56 [667]    pengine:     info: common_print:       xstha1_san0_IP  (ocf::heartbeat:IPaddr):        Started xstha1 (UNCLEAN)
Dec 16 15:08:56 [667]    pengine:     info: common_print:       xstha2_san0_IP  (ocf::heartbeat:IPaddr):        Started xstha2
Dec 16 15:08:56 [667]    pengine:     info: common_print:       zpool_data      (ocf::heartbeat:ZFS):   Started xstha1 (UNCLEAN)
Dec 16 15:08:56 [667]    pengine:     info: common_print:       xstha1-stonith  (stonith:external/ipmi):        Started xstha2
Dec 16 15:08:56 [667]    pengine:     info: common_print:       xstha2-stonith  (stonith:external/ipmi):        Started xstha1 (UNCLEAN)
Dec 16 15:08:56 [667]    pengine:     info: pcmk__native_allocate:      Resource xstha2-stonith cannot run anywhere
Dec 16 15:08:56 [667]    pengine:  warning: custom_action:      Action xstha1_san0_IP_stop_0 on xstha1 is unrunnable (offline)
Dec 16 15:08:56 [667]    pengine:  warning: custom_action:      Action zpool_data_stop_0 on xstha1 is unrunnable (offline)
Dec 16 15:08:56 [667]    pengine:  warning: custom_action:      Action xstha2-stonith_stop_0 on xstha1 is unrunnable (offline)
Dec 16 15:08:56 [667]    pengine:  warning: custom_action:      Action xstha2-stonith_stop_0 on xstha1 is unrunnable (offline)
Dec 16 15:08:56 [667]    pengine:  warning: stage6:     Scheduling Node xstha1 for STONITH
Dec 16 15:08:56 [667]    pengine:     info: native_stop_constraints:    xstha1_san0_IP_stop_0 is implicit after xstha1 is fenced
Dec 16 15:08:56 [667]    pengine:     info: native_stop_constraints:    zpool_data_stop_0 is implicit after xstha1 is fenced
Dec 16 15:08:56 [667]    pengine:     info: native_stop_constraints:    xstha2-stonith_stop_0 is implicit after xstha1 is fenced
Dec 16 15:08:56 [667]    pengine:   notice: LogNodeActions:      * Fence (off) xstha1 'peer is no longer part of the cluster'
Dec 16 15:08:56 [667]    pengine:   notice: LogAction:   * Move       xstha1_san0_IP     ( xstha1 -> xstha2 )
Dec 16 15:08:56 [667]    pengine:     info: LogActions: Leave   xstha2_san0_IP  (Started xstha2)
Dec 16 15:08:56 [667]    pengine:   notice: LogAction:   * Move       zpool_data         ( xstha1 -> xstha2 )
Dec 16 15:08:56 [667]    pengine:     info: LogActions: Leave   xstha1-stonith  (Started xstha2)
Dec 16 15:08:56 [667]    pengine:   notice: LogAction:   * Stop       xstha2-stonith     (           xstha1 )   due to node availability
Dec 16 15:08:56 [667]    pengine:  warning: process_pe_message: Calculated transition 0 (with warnings), saving inputs in /sonicle/var/cluster/lib/pacemaker/pengine/pe-warn-6.bz2
Dec 16 15:08:56 [668]       crmd:     info: do_state_transition:        State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
Dec 16 15:08:56 [668]       crmd:     info: do_te_invoke:       Processing graph 0 (ref=pe_calc-dc-1608127736-14) derived from /sonicle/var/cluster/lib/pacemaker/pengine/pe-warn-6.bz2
Dec 16 15:08:56 [668]       crmd:   notice: te_fence_node:      Requesting fencing (off) of node xstha1 | action=1 timeout=60000
Dec 16 15:08:56 [664] stonith-ng:   notice: handle_request:     Client crmd.668.c46cefe4 wants to fence (off) 'xstha1' with device '(any)'
Dec 16 15:08:56 [664] stonith-ng:   notice: initiate_remote_stonith_op: Requesting peer fencing (off) targeting xstha1 | id=3cdbf44e-e860-c100-95e0-db72cc63ae16 state=0
Dec 16 15:08:56 [664] stonith-ng:     info: dynamic_list_search_cb:     Refreshing port list for xstha1-stonith
Dec 16 15:08:56 [664] stonith-ng:     info: process_remote_stonith_query:       Query result 1 of 1 from xstha2 for xstha1/off (1 devices) 3cdbf44e-e860-c100-95e0-db72cc63ae16
Dec 16 15:08:56 [664] stonith-ng:     info: call_remote_stonith:        Total timeout set to 60 for peer's fencing targeting xstha1 for crmd.668|id=3cdbf44e-e860-c100-95e0-db72cc63ae16
Dec 16 15:08:56 [664] stonith-ng:   notice: call_remote_stonith:        Requesting that xstha2 perform 'off' action targeting xstha1 | for client crmd.668 (72s, 0s)
Dec 16 15:08:56 [664] stonith-ng:   notice: can_fence_host_with_device: xstha1-stonith can fence (off) xstha1: dynamic-list
Dec 16 15:08:56 [664] stonith-ng:     info: stonith_fence_get_devices_cb:       Found 1 matching devices for 'xstha1'
Dec 16 15:08:56 [664] stonith-ng:   notice: schedule_stonith_command:   Delaying 'off' action targeting xstha1 on xstha1-stonith for 10s (timeout=60s, requested_delay=0s, base=10s, max=10s)