[Pacemaker] Node fails to rejoin cluster

[Tal Yalon]

Thanks for replying Andrew.

Here's the other node's log (the one that fenced the non-responsive node) -
please let me know if there's any other information that may help. It's a
bit long, but it captures the moment node-1 finds out that node-2 is
non-responsive, then fences it and then gets stuck in an endless election
loop.

Feb  6 01:32:39 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:32:49 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:32:59 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:33:08 node-1 corosync[22668]:   [TOTEM ] A processor failed,
forming new configuration.
Feb  6 01:33:10 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:33:18 node-1 corosync[22668]:   [TOTEM ] A processor joined or
left the membership and a new membership was formed.
Feb  6 01:33:18 node-1 cib[22710]:   notice: ais_dispatch_message:
Membership 900: quorum lost
Feb  6 01:33:18 node-1 cib[22710]:     info: crm_update_peer: Node node-2:
id=2 state=lost (new) addr=r(0) ip(9.151.142.20)  votes=1 born=888 seen=896
proc=00000000000000000000000000111312
Feb  6 01:33:18 node-1 crmd[22715]:   notice: ais_dispatch_message:
Membership 900: quorum lost
Feb  6 01:33:18 node-1 crmd[22715]:     info: ais_status_callback: status:
node-2 is now lost (was member)
Feb  6 01:33:18 node-1 crmd[22715]:     info: crm_update_peer: Node node-2:
id=2 state=lost (new) addr=r(0) ip(9.151.142.20)  votes=1 born=888 seen=896
proc=00000000000000000000000000111312
Feb  6 01:33:18 node-1 crmd[22715]:  warning: check_dead_member: Our DC
node (node-2) left the cluster
Feb  6 01:33:18 node-1 corosync[22668]:   [CPG   ] chosen downlist: sender
r(0) ip(9.151.142.10) ; members(old:2 left:1)
Feb  6 01:33:18 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_NOT_DC -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL
origin=check_dead_member ]
Feb  6 01:33:18 node-1 corosync[22668]:   [MAIN  ] Completed service
synchronization, ready to provide service.
Feb  6 01:33:18 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC
cause=C_FSA_INTERNAL origin=do_election_check ]
Feb  6 01:33:18 node-1 crmd[22715]:     info: do_te_control: Registering TE
UUID: b5af419e-6ebc-4277-a17a-ba30b3b9fb67
Feb  6 01:33:18 node-1 crmd[22715]:     info: set_graph_functions: Setting
custom graph functions
Feb  6 01:33:18 node-1 crmd[22715]:     info: do_dc_takeover: Taking over
DC status for this partition
Feb  6 01:33:18 node-1 cib[22710]:     info: cib_process_readwrite: We are
now in R/W mode
Feb  6 01:33:18 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_master for section 'all' (origin=local/crmd/12,
version=0.108.162): ok (rc=0)
Feb  6 01:33:18 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section cib (origin=local/crmd/13,
version=0.108.163): ok (rc=0)
Feb  6 01:33:18 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section crm_config (origin=local/crmd/15,
version=0.108.164): ok (rc=0)
Feb  6 01:33:18 node-1 crmd[22715]:     info: join_make_offer: Making join
offers based on membership 900
Feb  6 01:33:18 node-1 crmd[22715]:     info: do_dc_join_offer_all: join-1:
Waiting on 1 outstanding join acks
Feb  6 01:33:18 node-1 crmd[22715]:     info: ais_dispatch_message:
Membership 900: quorum still lost
Feb  6 01:33:18 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section crm_config (origin=local/crmd/17,
version=0.108.165): ok (rc=0)
Feb  6 01:33:18 node-1 crmd[22715]:     info: crmd_ais_dispatch: Setting
expected votes to 2
Feb  6 01:33:18 node-1 crmd[22715]:     info: update_dc: Set DC to node-1
(3.0.6)
Feb  6 01:33:18 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section crm_config (origin=local/crmd/20,
version=0.108.166): ok (rc=0)
Feb  6 01:33:18 node-1 crmd[22715]:     info: ais_dispatch_message:
Membership 900: quorum still lost
Feb  6 01:33:18 node-1 crmd[22715]:     info: crmd_ais_dispatch: Setting
expected votes to 2
Feb  6 01:33:18 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section crm_config (origin=local/crmd/23,
version=0.108.167): ok (rc=0)
Feb  6 01:33:18 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED
cause=C_FSA_INTERNAL origin=check_join_state ]
Feb  6 01:33:18 node-1 crmd[22715]:     info: do_dc_join_finalize: join-1:
Syncing the CIB from node-1 to the rest of the cluster
Feb  6 01:33:18 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_sync for section 'all' (origin=local/crmd/24,
version=0.108.167): ok (rc=0)
Feb  6 01:33:18 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section nodes (origin=local/crmd/25,
version=0.108.168): ok (rc=0)
Feb  6 01:33:18 node-1 crmd[22715]:     info: do_dc_join_ack: join-1:
Updating node state to member for node-1
Feb  6 01:33:18 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_delete for section //node_state[@uname='node-1']/lrm
(origin=local/crmd/26, version=0.108.169): ok (rc=0)
Feb  6 01:33:18 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED
cause=C_FSA_INTERNAL origin=check_join_state ]
Feb  6 01:33:18 node-1 crmd[22715]:     info: abort_transition_graph:
do_te_invoke:162 - Triggered transition abort (complete=1) : Peer Cancelled
Feb  6 01:33:18 node-1 attrd[22713]:   notice: attrd_local_callback:
Sending full refresh (origin=crmd)
Feb  6 01:33:18 node-1 attrd[22713]:   notice: attrd_trigger_update:
Sending flush op to all hosts for: probe_complete (true)
Feb  6 01:33:18 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section nodes (origin=local/crmd/28,
version=0.108.171): ok (rc=0)
Feb  6 01:33:18 node-1 crmd[22715]:  warning: match_down_event: No match
for shutdown action on node-2
Feb  6 01:33:18 node-1 crmd[22715]:     info: te_update_diff:
Stonith/shutdown of node-2 not matched
Feb  6 01:33:18 node-1 crmd[22715]:     info: abort_transition_graph:
te_update_diff:234 - Triggered transition abort (complete=1,
tag=node_state, id=node-2, magic=NA, cib=0.108.172) : Node failure
Feb  6 01:33:18 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section cib (origin=local/crmd/30,
version=0.108.173): ok (rc=0)
Feb  6 01:33:18 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:33:18 node-1 pengine[22714]:  warning: pe_fence_node: Node node-2
will be fenced because it is un-expectedly down
Feb  6 01:33:18 node-1 pengine[22714]:  warning: determine_online_status:
Node node-2 is unclean
Feb  6 01:33:18 node-1 pengine[22714]:  warning: unpack_rsc_op: Processing
failed op st_ipmi:0_last_failure_0 on node-2: unknown error (1)
Feb  6 01:33:18 node-1 pengine[22714]:   notice: common_apply_stickiness:
fencing can fail 999999 more times on node-2 before being forced off
Feb  6 01:33:18 node-1 pengine[22714]:   notice: common_apply_stickiness:
fencing can fail 999999 more times on node-2 before being forced off
Feb  6 01:33:18 node-1 pengine[22714]:  warning: custom_action: Action
cluster_ip_stop_0 on node-2 is unrunnable (offline)
Feb  6 01:33:18 node-1 pengine[22714]:  warning: custom_action: Marking
node node-2 unclean
Feb  6 01:33:18 node-1 pengine[22714]:  warning: custom_action: Action
myresource_stop_0 on node-2 is unrunnable (offline)
Feb  6 01:33:18 node-1 pengine[22714]:  warning: custom_action: Marking
node node-2 unclean
Feb  6 01:33:18 node-1 pengine[22714]:  warning: custom_action: Action
st_ipmi:0_stop_0 on node-2 is unrunnable (offline)
Feb  6 01:33:18 node-1 pengine[22714]:  warning: custom_action: Marking
node node-2 unclean
Feb  6 01:33:18 node-1 pengine[22714]:  warning: stage6: Scheduling Node
node-2 for STONITH
Feb  6 01:33:18 node-1 pengine[22714]:   notice: LogActions: Move
 cluster_ip#011(Started node-2 -> node-1)
Feb  6 01:33:18 node-1 pengine[22714]:   notice: LogActions: Move
 myresource#011(Started node-2 -> node-1)
Feb  6 01:33:18 node-1 pengine[22714]:   notice: LogActions: Stop
 st_ipmi:0#011(node-2)
Feb  6 01:33:18 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:33:18 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 0 (ref=pe_calc-dc-1360114398-12) derived from
/var/lib/pengine/pe-warn-9.bz2
Feb  6 01:33:18 node-1 crmd[22715]:   notice: te_fence_node: Executing
reboot fencing operation (20) on node-2 (timeout=60000)
Feb  6 01:33:18 node-1 stonith-ng[22711]:     info:
initiate_remote_stonith_op: Initiating remote operation reboot for node-2:
1b4ce854-f298-4d58-bcc8-641294c7b716
Feb  6 01:33:18 node-1 pengine[22714]:  warning: process_pe_message:
Transition 0: WARNINGs found during PE processing. PEngine Input stored in:
/var/lib/pengine/pe-warn-9.bz2
Feb  6 01:33:18 node-1 pengine[22714]:   notice: process_pe_message:
Configuration WARNINGs found during PE processing.  Please run "crm_verify
-L" to identify issues.
Feb  6 01:33:19 node-1 stonith-ng[22711]:     info:
can_fence_host_with_device: Refreshing port list for st_ipmi:1
Feb  6 01:33:19 node-1 stonith-ng[22711]:     info:
can_fence_host_with_device: st_ipmi:1 can fence node-2: dynamic-list
Feb  6 01:33:19 node-1 stonith-ng[22711]:     info: call_remote_stonith:
Requesting that node-1 perform op reboot node-2
Feb  6 01:33:19 node-1 stonith-ng[22711]:     info:
can_fence_host_with_device: st_ipmi:1 can fence node-2: dynamic-list
Feb  6 01:33:19 node-1 stonith-ng[22711]:     info: stonith_fence: Found 1
matching devices for 'node-2'
Feb  6 01:33:19 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_fence from node-1: rc=-1
Feb  6 01:33:19 node-1 stonith-ng[22711]:   notice: log_operation:
Operation 'reboot' [32224] (call 0 from
e237586f-a38e-4e6d-8b19-519592f198ac) for host 'node-2' with device
'st_ipmi:1' returned: 0
Feb  6 01:33:19 node-1 stonith-ng[22711]:   notice: remote_op_done:
Operation reboot of node-2 by node-1 for
node-1[e237586f-a38e-4e6d-8b19-519592f198ac]: OK
Feb  6 01:33:19 node-1 crmd[22715]:     info: tengine_stonith_callback:
StonithOp <st-reply st_origin="stonith_construct_async_reply"
t="stonith-ng" st_op="reboot"
st_remote_op="1b4ce854-f298-4d58-bcc8-641294c7b716"
st_clientid="e237586f-a38e-4e6d-8b19-519592f198ac" st_target="node-2"
st_device_action="st_fence" st_callid="0" st_callopt="0" st_rc="0"
src="node-1" seq="4" state="2" />
Feb  6 01:33:19 node-1 crmd[22715]:   notice: crmd_peer_update: Status
update: Client node-2/crmd now has status [offline] (DC=true)
Feb  6 01:33:19 node-1 crmd[22715]:   notice: tengine_stonith_notify: Peer
node-2 was terminated (reboot) by node-1 for node-1: OK
(ref=1b4ce854-f298-4d58-bcc8-641294c7b716)
Feb  6 01:33:19 node-1 crmd[22715]:     info: te_rsc_command: Initiating
action 6: start cluster_ip_start_0 on node-1 (local)
Feb  6 01:33:19 node-1 lrmd: [22712]: info: rsc:cluster_ip:7: start
Feb  6 01:33:19 node-1 crmd[22715]:     info: cib_fencing_updated: Fencing
update 33 for node-2: complete
Feb  6 01:33:19 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_delete for section //node_state[@uname='node-2']/lrm
(origin=local/crmd/34, version=0.108.176): ok (rc=0)
Feb  6 01:33:19 node-1 crmd[22715]:     info: abort_transition_graph:
te_update_diff:320 - Triggered transition abort (complete=0,
tag=lrm_rsc_op, id=cluster_ip_last_0,
magic=0:0;6:0:0:9204064b-65aa-46f0-9f0f-4213cbbf70bc, cib=0.108.176) :
Resource op removal
Feb  6 01:33:19 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_delete for section
//node_state[@uname='node-2']/transient_attributes (origin=local/crmd/35,
version=0.108.177): ok (rc=0)
Feb  6 01:33:19 node-1 crmd[22715]:     info: abort_transition_graph:
te_update_diff:194 - Triggered transition abort (complete=0,
tag=transient_attributes, id=node-2, magic=NA, cib=0.108.177) : Transient
attribute: removal
Feb  6 01:33:19 node-1 IPaddr2(cluster_ip)[32226]: INFO: ip -f inet addr
add 192.168.65.93/20 brd 192.168.79.255 dev bond0
Feb  6 01:33:19 node-1 IPaddr2(cluster_ip)[32226]: INFO: ip link set bond0
up
Feb  6 01:33:19 node-1 IPaddr2(cluster_ip)[32226]: INFO:
/usr/lib64/heartbeat/send_arp -i 200 -r 5 -p
/var/run/heartbeat/rsctmp/send_arp-192.168.65.93 bond0 192.168.65.93 auto
not_used not_used
Feb  6 01:33:19 node-1 crmd[22715]:     info: process_lrm_event: LRM
operation cluster_ip_start_0 (call=7, rc=0, cib-update=37, confirmed=true)
ok
Feb  6 01:33:19 node-1 crmd[22715]:   notice: run_graph: ==== Transition 0
(Complete=10, Pending=0, Fired=0, Skipped=3, Incomplete=0,
Source=/var/lib/pengine/pe-warn-9.bz2): Stopped
Feb  6 01:33:19 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE [ input=I_PE_CALC
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:33:19 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:33:19 node-1 pengine[22714]:   notice: LogActions: Start
myresource#011(node-1)
Feb  6 01:33:19 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:33:19 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 1 (ref=pe_calc-dc-1360114399-14) derived from
/var/lib/pengine/pe-input-157.bz2
Feb  6 01:33:19 node-1 crmd[22715]:     info: te_rsc_command: Initiating
action 7: monitor cluster_ip_monitor_30000 on node-1 (local)
Feb  6 01:33:19 node-1 lrmd: [22712]: info: rsc:cluster_ip:8: monitor
Feb  6 01:33:19 node-1 crmd[22715]:     info: te_rsc_command: Initiating
action 8: start myresource_start_0 on node-1 (local)
Feb  6 01:33:19 node-1 lrmd: [22712]: info: rsc:myresource:9: start
Feb  6 01:33:19 node-1 crmd[22715]:     info: process_lrm_event: LRM
operation cluster_ip_monitor_30000 (call=8, rc=0, cib-update=39,
confirmed=false) ok
Feb  6 01:33:19 node-1 pengine[22714]:   notice: process_pe_message:
Transition 1: PEngine Input stored in: /var/lib/pengine/pe-input-157.bz2
Feb  6 01:33:20 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:33:22 node-1 crmd[22715]:     info: process_lrm_event: LRM
operation myresource_start_0 (call=9, rc=0, cib-update=40, confirmed=true)
ok
Feb  6 01:33:22 node-1 crmd[22715]:     info: te_rsc_command: Initiating
action 9: monitor myresource_monitor_2000 on node-1 (local)
Feb  6 01:33:22 node-1 lrmd: [22712]: info: rsc:myresource:10: monitor
Feb  6 01:33:22 node-1 crmd[22715]:     info: process_lrm_event: LRM
operation myresource_monitor_2000 (call=10, rc=0, cib-update=41,
confirmed=false) ok
Feb  6 01:33:22 node-1 crmd[22715]:   notice: run_graph: ==== Transition 1
(Complete=3, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-157.bz2): Complete
Feb  6 01:33:22 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:33:23 node-1 lrmd: [22712]: info: RA output:
(cluster_ip:start:stderr) ARPING 192.168.65.93 from 192.168.65.93
bond0#012Sent 5 probes (5 broadcast(s))#012Received 0 response(s)
Feb  6 01:33:30 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:33:32 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:33:32 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:33:32 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:33:32 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:33:32 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:33:32 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 2 (ref=pe_calc-dc-1360114412-18) derived from
/var/lib/pengine/pe-input-158.bz2
Feb  6 01:33:32 node-1 crmd[22715]:   notice: run_graph: ==== Transition 2
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-158.bz2): Complete
Feb  6 01:33:32 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:33:32 node-1 pengine[22714]:   notice: process_pe_message:
Transition 2: PEngine Input stored in: /var/lib/pengine/pe-input-158.bz2
Feb  6 01:33:41 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:33:42 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:33:42 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:33:42 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:33:42 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:33:42 node-1 pengine[22714]:   notice: process_pe_message:
Transition 3: PEngine Input stored in: /var/lib/pengine/pe-input-158.bz2
Feb  6 01:33:42 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:33:42 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 3 (ref=pe_calc-dc-1360114422-19) derived from
/var/lib/pengine/pe-input-158.bz2
Feb  6 01:33:42 node-1 crmd[22715]:   notice: run_graph: ==== Transition 3
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-158.bz2): Complete
Feb  6 01:33:42 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:33:51 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:33:52 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:33:52 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:33:52 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:33:52 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:33:52 node-1 pengine[22714]:   notice: process_pe_message:
Transition 4: PEngine Input stored in: /var/lib/pengine/pe-input-158.bz2
Feb  6 01:33:52 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:33:52 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 4 (ref=pe_calc-dc-1360114432-20) derived from
/var/lib/pengine/pe-input-158.bz2
Feb  6 01:33:52 node-1 crmd[22715]:   notice: run_graph: ==== Transition 4
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-158.bz2): Complete
Feb  6 01:33:52 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:34:01 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:34:02 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:34:02 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:34:02 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:34:02 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:34:02 node-1 pengine[22714]:   notice: process_pe_message:
Transition 5: PEngine Input stored in: /var/lib/pengine/pe-input-158.bz2
Feb  6 01:34:02 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:34:02 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 5 (ref=pe_calc-dc-1360114442-21) derived from
/var/lib/pengine/pe-input-158.bz2
Feb  6 01:34:02 node-1 crmd[22715]:   notice: run_graph: ==== Transition 5
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-158.bz2): Complete
Feb  6 01:34:02 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:34:12 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:34:12 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:34:12 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:34:12 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:34:12 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:34:12 node-1 pengine[22714]:   notice: process_pe_message:
Transition 6: PEngine Input stored in: /var/lib/pengine/pe-input-158.bz2
Feb  6 01:34:12 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:34:12 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 6 (ref=pe_calc-dc-1360114452-22) derived from
/var/lib/pengine/pe-input-158.bz2
Feb  6 01:34:12 node-1 crmd[22715]:   notice: run_graph: ==== Transition 6
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-158.bz2): Complete
Feb  6 01:34:12 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:34:22 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:34:22 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:34:22 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:34:22 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:34:22 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:34:22 node-1 pengine[22714]:   notice: process_pe_message:
Transition 7: PEngine Input stored in: /var/lib/pengine/pe-input-158.bz2
Feb  6 01:34:22 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:34:22 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 7 (ref=pe_calc-dc-1360114462-23) derived from
/var/lib/pengine/pe-input-158.bz2
Feb  6 01:34:22 node-1 crmd[22715]:   notice: run_graph: ==== Transition 7
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-158.bz2): Complete
Feb  6 01:34:22 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:34:32 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:34:32 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:34:32 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:34:32 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:34:32 node-1 pengine[22714]:   notice: process_pe_message:
Transition 8: PEngine Input stored in: /var/lib/pengine/pe-input-158.bz2
Feb  6 01:34:32 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:34:32 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 8 (ref=pe_calc-dc-1360114472-24) derived from
/var/lib/pengine/pe-input-158.bz2
Feb  6 01:34:32 node-1 crmd[22715]:   notice: run_graph: ==== Transition 8
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-158.bz2): Complete
Feb  6 01:34:32 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:34:33 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:34:42 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:34:42 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:34:42 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:34:42 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:34:42 node-1 pengine[22714]:   notice: process_pe_message:
Transition 9: PEngine Input stored in: /var/lib/pengine/pe-input-158.bz2
Feb  6 01:34:42 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:34:42 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 9 (ref=pe_calc-dc-1360114482-25) derived from
/var/lib/pengine/pe-input-158.bz2
Feb  6 01:34:42 node-1 crmd[22715]:   notice: run_graph: ==== Transition 9
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-158.bz2): Complete
Feb  6 01:34:42 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:34:43 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:34:52 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:34:52 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:34:52 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:34:52 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:34:52 node-1 pengine[22714]:   notice: process_pe_message:
Transition 10: PEngine Input stored in: /var/lib/pengine/pe-input-158.bz2
Feb  6 01:34:52 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:34:52 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 10 (ref=pe_calc-dc-1360114492-26) derived from
/var/lib/pengine/pe-input-158.bz2
Feb  6 01:34:52 node-1 crmd[22715]:   notice: run_graph: ==== Transition 10
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-158.bz2): Complete
Feb  6 01:34:52 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:34:53 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:35:02 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:35:02 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:35:02 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:35:02 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:35:02 node-1 pengine[22714]:   notice: process_pe_message:
Transition 11: PEngine Input stored in: /var/lib/pengine/pe-input-158.bz2
Feb  6 01:35:02 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:35:02 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 11 (ref=pe_calc-dc-1360114502-27) derived from
/var/lib/pengine/pe-input-158.bz2
Feb  6 01:35:02 node-1 crmd[22715]:   notice: run_graph: ==== Transition 11
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-158.bz2): Complete
Feb  6 01:35:02 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:35:04 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:35:12 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:35:12 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:35:12 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:35:12 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:35:12 node-1 pengine[22714]:   notice: process_pe_message:
Transition 12: PEngine Input stored in: /var/lib/pengine/pe-input-158.bz2
Feb  6 01:35:12 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:35:12 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 12 (ref=pe_calc-dc-1360114512-28) derived from
/var/lib/pengine/pe-input-158.bz2
Feb  6 01:35:12 node-1 crmd[22715]:   notice: run_graph: ==== Transition 12
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-158.bz2): Complete
Feb  6 01:35:12 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:35:14 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:35:22 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:35:22 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:35:22 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:35:22 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:35:22 node-1 pengine[22714]:   notice: process_pe_message:
Transition 13: PEngine Input stored in: /var/lib/pengine/pe-input-158.bz2
Feb  6 01:35:22 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:35:22 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 13 (ref=pe_calc-dc-1360114522-29) derived from
/var/lib/pengine/pe-input-158.bz2
Feb  6 01:35:22 node-1 crmd[22715]:   notice: run_graph: ==== Transition 13
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-158.bz2): Complete
Feb  6 01:35:22 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:35:24 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:35:32 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:35:32 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:35:32 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:35:32 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:35:32 node-1 pengine[22714]:   notice: process_pe_message:
Transition 14: PEngine Input stored in: /var/lib/pengine/pe-input-158.bz2
Feb  6 01:35:32 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:35:32 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 14 (ref=pe_calc-dc-1360114532-30) derived from
/var/lib/pengine/pe-input-158.bz2
Feb  6 01:35:32 node-1 crmd[22715]:   notice: run_graph: ==== Transition 14
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-158.bz2): Complete
Feb  6 01:35:32 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:35:35 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:35:42 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:35:42 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:35:42 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:35:42 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:35:42 node-1 pengine[22714]:   notice: process_pe_message:
Transition 15: PEngine Input stored in: /var/lib/pengine/pe-input-158.bz2
Feb  6 01:35:42 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:35:42 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 15 (ref=pe_calc-dc-1360114542-31) derived from
/var/lib/pengine/pe-input-158.bz2
Feb  6 01:35:42 node-1 crmd[22715]:   notice: run_graph: ==== Transition 15
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-158.bz2): Complete
Feb  6 01:35:42 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:35:45 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:35:52 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:35:52 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:35:52 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:35:52 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:35:52 node-1 pengine[22714]:   notice: process_pe_message:
Transition 16: PEngine Input stored in: /var/lib/pengine/pe-input-158.bz2
Feb  6 01:35:52 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:35:52 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 16 (ref=pe_calc-dc-1360114552-32) derived from
/var/lib/pengine/pe-input-158.bz2
Feb  6 01:35:52 node-1 crmd[22715]:   notice: run_graph: ==== Transition 16
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-158.bz2): Complete
Feb  6 01:35:52 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:35:56 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:36:02 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:36:02 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:36:02 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:36:02 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:36:02 node-1 pengine[22714]:   notice: process_pe_message:
Transition 17: PEngine Input stored in: /var/lib/pengine/pe-input-158.bz2
Feb  6 01:36:02 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:36:02 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 17 (ref=pe_calc-dc-1360114562-33) derived from
/var/lib/pengine/pe-input-158.bz2
Feb  6 01:36:02 node-1 crmd[22715]:   notice: run_graph: ==== Transition 17
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-158.bz2): Complete
Feb  6 01:36:02 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:36:06 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:36:12 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:36:12 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:36:12 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:36:12 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:36:12 node-1 pengine[22714]:   notice: process_pe_message:
Transition 18: PEngine Input stored in: /var/lib/pengine/pe-input-158.bz2
Feb  6 01:36:12 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:36:12 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 18 (ref=pe_calc-dc-1360114572-34) derived from
/var/lib/pengine/pe-input-158.bz2
Feb  6 01:36:12 node-1 crmd[22715]:   notice: run_graph: ==== Transition 18
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-158.bz2): Complete
Feb  6 01:36:12 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:36:17 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:36:22 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:36:22 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:36:22 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:36:22 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:36:22 node-1 pengine[22714]:   notice: process_pe_message:
Transition 19: PEngine Input stored in: /var/lib/pengine/pe-input-158.bz2
Feb  6 01:36:22 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:36:22 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 19 (ref=pe_calc-dc-1360114582-35) derived from
/var/lib/pengine/pe-input-158.bz2
Feb  6 01:36:22 node-1 crmd[22715]:   notice: run_graph: ==== Transition 19
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-158.bz2): Complete
Feb  6 01:36:22 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:36:28 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:36:32 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:36:32 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:36:32 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:36:32 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:36:32 node-1 pengine[22714]:   notice: process_pe_message:
Transition 20: PEngine Input stored in: /var/lib/pengine/pe-input-158.bz2
Feb  6 01:36:32 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:36:32 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 20 (ref=pe_calc-dc-1360114592-36) derived from
/var/lib/pengine/pe-input-158.bz2
Feb  6 01:36:32 node-1 crmd[22715]:   notice: run_graph: ==== Transition 20
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-158.bz2): Complete
Feb  6 01:36:32 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:36:39 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:36:42 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:36:42 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:36:42 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:36:42 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:36:42 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:36:42 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 21 (ref=pe_calc-dc-1360114602-37) derived from
/var/lib/pengine/pe-input-158.bz2
Feb  6 01:36:42 node-1 crmd[22715]:   notice: run_graph: ==== Transition 21
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-158.bz2): Complete
Feb  6 01:36:42 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:36:42 node-1 pengine[22714]:   notice: process_pe_message:
Transition 21: PEngine Input stored in: /var/lib/pengine/pe-input-158.bz2
Feb  6 01:36:49 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:36:52 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:36:52 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:36:52 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:36:52 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:36:52 node-1 pengine[22714]:   notice: process_pe_message:
Transition 22: PEngine Input stored in: /var/lib/pengine/pe-input-158.bz2
Feb  6 01:36:52 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:36:52 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 22 (ref=pe_calc-dc-1360114612-38) derived from
/var/lib/pengine/pe-input-158.bz2
Feb  6 01:36:52 node-1 crmd[22715]:   notice: run_graph: ==== Transition 22
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-158.bz2): Complete
Feb  6 01:36:52 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:37:00 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:37:02 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:37:02 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:37:02 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:37:02 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:37:02 node-1 pengine[22714]:   notice: process_pe_message:
Transition 23: PEngine Input stored in: /var/lib/pengine/pe-input-158.bz2
Feb  6 01:37:02 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:37:02 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 23 (ref=pe_calc-dc-1360114622-39) derived from
/var/lib/pengine/pe-input-158.bz2
Feb  6 01:37:02 node-1 crmd[22715]:   notice: run_graph: ==== Transition 23
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-158.bz2): Complete
Feb  6 01:37:02 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:37:11 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:37:12 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:37:12 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:37:12 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:37:12 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:37:12 node-1 pengine[22714]:   notice: process_pe_message:
Transition 24: PEngine Input stored in: /var/lib/pengine/pe-input-158.bz2
Feb  6 01:37:12 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:37:12 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 24 (ref=pe_calc-dc-1360114632-40) derived from
/var/lib/pengine/pe-input-158.bz2
Feb  6 01:37:12 node-1 crmd[22715]:   notice: run_graph: ==== Transition 24
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-158.bz2): Complete
Feb  6 01:37:12 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:37:22 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:37:22 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:37:22 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:37:22 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:37:22 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:37:22 node-1 pengine[22714]:   notice: process_pe_message:
Transition 25: PEngine Input stored in: /var/lib/pengine/pe-input-158.bz2
Feb  6 01:37:22 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:37:22 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 25 (ref=pe_calc-dc-1360114642-41) derived from
/var/lib/pengine/pe-input-158.bz2
Feb  6 01:37:22 node-1 crmd[22715]:   notice: run_graph: ==== Transition 25
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-158.bz2): Complete
Feb  6 01:37:22 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:37:32 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:37:32 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:37:32 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:37:32 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:37:32 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:37:32 node-1 pengine[22714]:   notice: process_pe_message:
Transition 26: PEngine Input stored in: /var/lib/pengine/pe-input-158.bz2
Feb  6 01:37:32 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:37:32 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 26 (ref=pe_calc-dc-1360114652-42) derived from
/var/lib/pengine/pe-input-158.bz2
Feb  6 01:37:32 node-1 crmd[22715]:   notice: run_graph: ==== Transition 26
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-158.bz2): Complete
Feb  6 01:37:32 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:37:42 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:37:42 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:37:42 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:37:42 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:37:42 node-1 pengine[22714]:   notice: process_pe_message:
Transition 27: PEngine Input stored in: /var/lib/pengine/pe-input-158.bz2
Feb  6 01:37:42 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:37:42 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 27 (ref=pe_calc-dc-1360114662-43) derived from
/var/lib/pengine/pe-input-158.bz2
Feb  6 01:37:42 node-1 crmd[22715]:   notice: run_graph: ==== Transition 27
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-158.bz2): Complete
Feb  6 01:37:42 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:37:43 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:37:52 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:37:52 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:37:52 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:37:52 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:37:52 node-1 pengine[22714]:   notice: process_pe_message:
Transition 28: PEngine Input stored in: /var/lib/pengine/pe-input-158.bz2
Feb  6 01:37:53 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:37:53 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 28 (ref=pe_calc-dc-1360114672-44) derived from
/var/lib/pengine/pe-input-158.bz2
Feb  6 01:37:53 node-1 crmd[22715]:   notice: run_graph: ==== Transition 28
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-158.bz2): Complete
Feb  6 01:37:53 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:37:54 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:38:03 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:38:03 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:38:03 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:38:03 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:38:03 node-1 pengine[22714]:   notice: process_pe_message:
Transition 29: PEngine Input stored in: /var/lib/pengine/pe-input-158.bz2
Feb  6 01:38:03 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:38:03 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 29 (ref=pe_calc-dc-1360114683-45) derived from
/var/lib/pengine/pe-input-158.bz2
Feb  6 01:38:03 node-1 crmd[22715]:   notice: run_graph: ==== Transition 29
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-158.bz2): Complete
Feb  6 01:38:03 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:38:04 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:38:13 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:38:13 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:38:13 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:38:13 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:38:13 node-1 pengine[22714]:   notice: process_pe_message:
Transition 30: PEngine Input stored in: /var/lib/pengine/pe-input-158.bz2
Feb  6 01:38:13 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:38:13 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 30 (ref=pe_calc-dc-1360114693-46) derived from
/var/lib/pengine/pe-input-158.bz2
Feb  6 01:38:13 node-1 crmd[22715]:   notice: run_graph: ==== Transition 30
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-158.bz2): Complete
Feb  6 01:38:13 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:38:15 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:38:20 node-1 cib[22710]:     info: cib_stats: Processed 64
operations (625.00us average, 0% utilization) in the last 10min
Feb  6 01:38:23 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:38:23 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:38:23 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:38:23 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:38:23 node-1 pengine[22714]:   notice: process_pe_message:
Transition 31: PEngine Input stored in: /var/lib/pengine/pe-input-158.bz2
Feb  6 01:38:23 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:38:23 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 31 (ref=pe_calc-dc-1360114703-47) derived from
/var/lib/pengine/pe-input-158.bz2
Feb  6 01:38:23 node-1 crmd[22715]:   notice: run_graph: ==== Transition 31
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-158.bz2): Complete
Feb  6 01:38:23 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:38:26 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:38:33 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:38:33 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:38:33 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:38:33 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:38:33 node-1 pengine[22714]:   notice: process_pe_message:
Transition 32: PEngine Input stored in: /var/lib/pengine/pe-input-158.bz2
Feb  6 01:38:33 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:38:33 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 32 (ref=pe_calc-dc-1360114713-48) derived from
/var/lib/pengine/pe-input-158.bz2
Feb  6 01:38:33 node-1 crmd[22715]:   notice: run_graph: ==== Transition 32
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-158.bz2): Complete
Feb  6 01:38:33 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:38:36 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:38:43 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:38:43 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:38:43 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:38:43 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:38:43 node-1 pengine[22714]:   notice: process_pe_message:
Transition 33: PEngine Input stored in: /var/lib/pengine/pe-input-158.bz2
Feb  6 01:38:43 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:38:43 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 33 (ref=pe_calc-dc-1360114723-49) derived from
/var/lib/pengine/pe-input-158.bz2
Feb  6 01:38:43 node-1 crmd[22715]:   notice: run_graph: ==== Transition 33
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-158.bz2): Complete
Feb  6 01:38:43 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:38:47 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:38:53 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:38:53 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:38:53 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:38:53 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:38:53 node-1 pengine[22714]:   notice: process_pe_message:
Transition 34: PEngine Input stored in: /var/lib/pengine/pe-input-158.bz2
Feb  6 01:38:53 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:38:53 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 34 (ref=pe_calc-dc-1360114733-50) derived from
/var/lib/pengine/pe-input-158.bz2
Feb  6 01:38:53 node-1 crmd[22715]:   notice: run_graph: ==== Transition 34
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-158.bz2): Complete
Feb  6 01:38:53 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:38:58 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:39:03 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:39:03 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:39:03 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:39:03 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:39:03 node-1 pengine[22714]:   notice: process_pe_message:
Transition 35: PEngine Input stored in: /var/lib/pengine/pe-input-158.bz2
Feb  6 01:39:03 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:39:03 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 35 (ref=pe_calc-dc-1360114743-51) derived from
/var/lib/pengine/pe-input-158.bz2
Feb  6 01:39:03 node-1 crmd[22715]:   notice: run_graph: ==== Transition 35
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-158.bz2): Complete
Feb  6 01:39:03 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:39:08 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:39:13 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:39:13 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:39:13 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:39:13 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:39:13 node-1 pengine[22714]:   notice: process_pe_message:
Transition 36: PEngine Input stored in: /var/lib/pengine/pe-input-158.bz2
Feb  6 01:39:13 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:39:13 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 36 (ref=pe_calc-dc-1360114753-52) derived from
/var/lib/pengine/pe-input-158.bz2
Feb  6 01:39:13 node-1 crmd[22715]:   notice: run_graph: ==== Transition 36
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-158.bz2): Complete
Feb  6 01:39:13 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:39:19 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:39:23 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:39:23 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:39:23 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:39:23 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:39:23 node-1 pengine[22714]:   notice: process_pe_message:
Transition 37: PEngine Input stored in: /var/lib/pengine/pe-input-158.bz2
Feb  6 01:39:23 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:39:23 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 37 (ref=pe_calc-dc-1360114763-53) derived from
/var/lib/pengine/pe-input-158.bz2
Feb  6 01:39:23 node-1 crmd[22715]:   notice: run_graph: ==== Transition 37
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-158.bz2): Complete
Feb  6 01:39:23 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:39:30 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:39:33 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:39:33 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:39:33 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:39:33 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:39:33 node-1 pengine[22714]:   notice: process_pe_message:
Transition 38: PEngine Input stored in: /var/lib/pengine/pe-input-158.bz2
Feb  6 01:39:33 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:39:33 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 38 (ref=pe_calc-dc-1360114773-54) derived from
/var/lib/pengine/pe-input-158.bz2
Feb  6 01:39:33 node-1 crmd[22715]:   notice: run_graph: ==== Transition 38
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-158.bz2): Complete
Feb  6 01:39:33 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:39:38 node-1 corosync[22668]:   [TOTEM ] A processor joined or
left the membership and a new membership was formed.
Feb  6 01:39:38 node-1 crmd[22715]:   notice: ais_dispatch_message:
Membership 904: quorum acquired
Feb  6 01:39:38 node-1 cib[22710]:   notice: ais_dispatch_message:
Membership 904: quorum acquired
Feb  6 01:39:38 node-1 cib[22710]:     info: crm_update_peer: Node node-2:
id=2 state=member (new) addr=r(0) ip(9.151.142.20)  votes=1 born=888
seen=904 proc=00000000000000000000000000111312
Feb  6 01:39:38 node-1 crmd[22715]:     info: ais_status_callback: status:
node-2 is now member (was lost)
Feb  6 01:39:38 node-1 crmd[22715]:  warning: match_down_event: No match
for shutdown action on node-2
Feb  6 01:39:38 node-1 cib[22710]:     info: ais_dispatch_message:
Membership 904: quorum retained
Feb  6 01:39:38 node-1 crmd[22715]:     info: crm_update_peer: Node node-2:
id=2 state=member (new) addr=r(0) ip(9.151.142.20)  votes=1 born=888
seen=904 proc=00000000000000000000000000000001
Feb  6 01:39:38 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_delete for section //node_state[@uname='node-2']/lrm
(origin=local/crmd/79, version=0.108.183): ok (rc=0)
Feb  6 01:39:38 node-1 corosync[22668]:   [CPG   ] chosen downlist: sender
r(0) ip(9.151.142.10) ; members(old:1 left:0)
Feb  6 01:39:38 node-1 corosync[22668]:   [MAIN  ] Completed service
synchronization, ready to provide service.
Feb  6 01:39:38 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_delete for section
//node_state[@uname='node-2']/transient_attributes (origin=local/crmd/80,
version=0.108.184): ok (rc=0)
Feb  6 01:39:38 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section nodes (origin=local/crmd/81,
version=0.108.185): ok (rc=0)
Feb  6 01:39:38 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section cib (origin=local/crmd/83,
version=0.108.187): ok (rc=0)
Feb  6 01:39:38 node-1 crmd[22715]:     info: crmd_ais_dispatch: Setting
expected votes to 2
Feb  6 01:39:38 node-1 crmd[22715]:     info: ais_dispatch_message:
Membership 904: quorum retained
Feb  6 01:39:38 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section crm_config (origin=local/crmd/85,
version=0.108.188): ok (rc=0)
Feb  6 01:39:38 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section nodes (origin=local/crmd/86,
version=0.108.189): ok (rc=0)
Feb  6 01:39:38 node-1 crmd[22715]:     info: crmd_ais_dispatch: Setting
expected votes to 2
Feb  6 01:39:38 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section crm_config (origin=local/crmd/89,
version=0.108.191): ok (rc=0)
Feb  6 01:39:38 node-1 crmd[22715]:  warning: match_down_event: No match
for shutdown action on node-2
Feb  6 01:39:38 node-1 crmd[22715]:     info: te_update_diff:
Stonith/shutdown of node-2 not matched
Feb  6 01:39:38 node-1 crmd[22715]:     info: abort_transition_graph:
te_update_diff:234 - Triggered transition abort (complete=1,
tag=node_state, id=node-2, magic=NA, cib=0.108.186) : Node failure
Feb  6 01:39:38 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL
origin=abort_transition_graph ]
Feb  6 01:39:38 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:39:38 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:39:38 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 39 (ref=pe_calc-dc-1360114778-57) derived from
/var/lib/pengine/pe-input-159.bz2
Feb  6 01:39:38 node-1 crmd[22715]:   notice: run_graph: ==== Transition 39
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-159.bz2): Complete
Feb  6 01:39:38 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:39:38 node-1 pengine[22714]:   notice: process_pe_message:
Transition 39: PEngine Input stored in: /var/lib/pengine/pe-input-159.bz2
Feb  6 01:39:39 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_INTEGRATION [ input=I_NODE_JOIN cause=C_HA_MESSAGE
origin=route_message ]
Feb  6 01:39:39 node-1 crmd[22715]:     info: abort_transition_graph:
do_te_invoke:169 - Triggered transition abort (complete=1) : Peer Halt
Feb  6 01:39:39 node-1 crmd[22715]:     info: do_dc_join_offer_all: A new
node joined the cluster
Feb  6 01:39:39 node-1 crmd[22715]:     info: join_make_offer: Making join
offers based on membership 904
Feb  6 01:39:39 node-1 crmd[22715]:     info: join_make_offer: Peer process
on node-2 is not active (yet?): 00000001 2
Feb  6 01:39:39 node-1 crmd[22715]:     info: do_dc_join_offer_all: join-2:
Waiting on 1 outstanding join acks
Feb  6 01:39:39 node-1 crmd[22715]:     info: update_dc: Set DC to node-1
(3.0.6)
Feb  6 01:39:39 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED
cause=C_FSA_INTERNAL origin=check_join_state ]
Feb  6 01:39:39 node-1 crmd[22715]:     info: do_dc_join_finalize: join-2:
Syncing the CIB from node-1 to the rest of the cluster
Feb  6 01:39:39 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_sync for section 'all' (origin=local/crmd/92,
version=0.108.191): ok (rc=0)
Feb  6 01:39:39 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section nodes (origin=local/crmd/93,
version=0.108.192): ok (rc=0)
Feb  6 01:39:40 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:39:40 node-1 crmd[22715]:     info: do_dc_join_ack: join-2:
Updating node state to member for node-1
Feb  6 01:39:40 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_delete for section //node_state[@uname='node-1']/lrm
(origin=local/crmd/94, version=0.108.193): ok (rc=0)
Feb  6 01:39:40 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED
cause=C_FSA_INTERNAL origin=check_join_state ]
Feb  6 01:39:40 node-1 crmd[22715]:     info: abort_transition_graph:
do_te_invoke:162 - Triggered transition abort (complete=1) : Peer Cancelled
Feb  6 01:39:40 node-1 attrd[22713]:   notice: attrd_local_callback:
Sending full refresh (origin=crmd)
Feb  6 01:39:40 node-1 attrd[22713]:   notice: attrd_trigger_update:
Sending flush op to all hosts for: probe_complete (true)
Feb  6 01:39:40 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section nodes (origin=local/crmd/96,
version=0.108.195): ok (rc=0)
Feb  6 01:39:40 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section cib (origin=local/crmd/98,
version=0.108.197): ok (rc=0)
Feb  6 01:39:40 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:39:40 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:39:40 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 40 (ref=pe_calc-dc-1360114780-62) derived from
/var/lib/pengine/pe-input-160.bz2
Feb  6 01:39:40 node-1 crmd[22715]:   notice: run_graph: ==== Transition 40
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-160.bz2): Complete
Feb  6 01:39:40 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:39:40 node-1 pengine[22714]:   notice: process_pe_message:
Transition 40: PEngine Input stored in: /var/lib/pengine/pe-input-160.bz2
Feb  6 01:39:50 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:39:50 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:39:50 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:39:50 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:39:50 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:39:50 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 41 (ref=pe_calc-dc-1360114790-63) derived from
/var/lib/pengine/pe-input-161.bz2
Feb  6 01:39:50 node-1 crmd[22715]:   notice: run_graph: ==== Transition 41
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-161.bz2): Complete
Feb  6 01:39:50 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:39:50 node-1 pengine[22714]:   notice: process_pe_message:
Transition 41: PEngine Input stored in: /var/lib/pengine/pe-input-161.bz2
Feb  6 01:39:51 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:39:59 node-1 crmd[22715]:     info: do_election_count_vote:
Election 2 (owner: node-2) pass: vote from node-2 (Uptime)
Feb  6 01:39:59 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL
origin=do_election_count_vote ]
Feb  6 01:39:59 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC
cause=C_FSA_INTERNAL origin=do_election_check ]
Feb  6 01:39:59 node-1 crmd[22715]:     info: do_dc_takeover: Taking over
DC status for this partition
Feb  6 01:39:59 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_master for section 'all' (origin=local/crmd/101,
version=0.108.199): ok (rc=0)
Feb  6 01:39:59 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section cib (origin=local/crmd/102,
version=0.108.200): ok (rc=0)
Feb  6 01:39:59 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section crm_config (origin=local/crmd/104,
version=0.108.201): ok (rc=0)
Feb  6 01:39:59 node-1 crmd[22715]:     info: join_make_offer: Peer process
on node-2 is not active (yet?): 00000001 2
Feb  6 01:39:59 node-1 crmd[22715]:     info: do_dc_join_offer_all: join-3:
Waiting on 1 outstanding join acks
Feb  6 01:39:59 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section crm_config (origin=local/crmd/106,
version=0.108.202): ok (rc=0)
Feb  6 01:39:59 node-1 crmd[22715]:     info: ais_dispatch_message:
Membership 904: quorum retained
Feb  6 01:39:59 node-1 crmd[22715]:     info: crmd_ais_dispatch: Setting
expected votes to 2
Feb  6 01:39:59 node-1 crmd[22715]:     info: update_dc: Set DC to node-1
(3.0.6)
Feb  6 01:39:59 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section crm_config (origin=local/crmd/109,
version=0.108.203): ok (rc=0)
Feb  6 01:39:59 node-1 crmd[22715]:     info: ais_dispatch_message:
Membership 904: quorum retained
Feb  6 01:39:59 node-1 crmd[22715]:     info: crmd_ais_dispatch: Setting
expected votes to 2
Feb  6 01:39:59 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section crm_config (origin=local/crmd/112,
version=0.108.204): ok (rc=0)
Feb  6 01:39:59 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED
cause=C_FSA_INTERNAL origin=check_join_state ]
Feb  6 01:39:59 node-1 crmd[22715]:     info: do_dc_join_finalize: join-3:
Syncing the CIB from node-1 to the rest of the cluster
Feb  6 01:39:59 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_sync for section 'all' (origin=local/crmd/113,
version=0.108.204): ok (rc=0)
Feb  6 01:39:59 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section nodes (origin=local/crmd/114,
version=0.108.205): ok (rc=0)
Feb  6 01:40:00 node-1 lrmd: [22712]: WARN: G_SIG_dispatch: Dispatch
function for SIGCHLD was delayed 510 ms (> 100 ms) before being called
(GSource: 0x18bfad0)
Feb  6 01:40:00 node-1 lrmd: [22712]: info: G_SIG_dispatch: started at
440356723 should have started at 440356672
Feb  6 01:40:00 node-1 crmd[22715]:     info: do_dc_join_ack: join-3:
Updating node state to member for node-1
Feb  6 01:40:00 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_delete for section //node_state[@uname='node-1']/lrm
(origin=local/crmd/115, version=0.108.206): ok (rc=0)
Feb  6 01:40:00 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED
cause=C_FSA_INTERNAL origin=check_join_state ]
Feb  6 01:40:00 node-1 crmd[22715]:     info: abort_transition_graph:
do_te_invoke:162 - Triggered transition abort (complete=1) : Peer Cancelled
Feb  6 01:40:00 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section nodes (origin=local/crmd/117,
version=0.108.208): ok (rc=0)
Feb  6 01:40:00 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section cib (origin=local/crmd/119,
version=0.108.210): ok (rc=0)
Feb  6 01:40:00 node-1 attrd[22713]:   notice: attrd_local_callback:
Sending full refresh (origin=crmd)
Feb  6 01:40:00 node-1 attrd[22713]:   notice: attrd_trigger_update:
Sending flush op to all hosts for: probe_complete (true)
Feb  6 01:40:00 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:40:00 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:40:00 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 42 (ref=pe_calc-dc-1360114800-69) derived from
/var/lib/pengine/pe-input-162.bz2
Feb  6 01:40:00 node-1 crmd[22715]:   notice: run_graph: ==== Transition 42
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-162.bz2): Complete
Feb  6 01:40:00 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:40:00 node-1 pengine[22714]:   notice: process_pe_message:
Transition 42: PEngine Input stored in: /var/lib/pengine/pe-input-162.bz2
Feb  6 01:40:02 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:40:10 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:40:10 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:40:10 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:40:10 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:40:10 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:40:10 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 43 (ref=pe_calc-dc-1360114810-70) derived from
/var/lib/pengine/pe-input-163.bz2
Feb  6 01:40:10 node-1 crmd[22715]:   notice: run_graph: ==== Transition 43
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-163.bz2): Complete
Feb  6 01:40:10 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:40:10 node-1 pengine[22714]:   notice: process_pe_message:
Transition 43: PEngine Input stored in: /var/lib/pengine/pe-input-163.bz2
Feb  6 01:40:12 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:40:19 node-1 crmd[22715]:     info: do_election_count_vote:
Election 3 (owner: node-2) pass: vote from node-2 (Uptime)
Feb  6 01:40:19 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL
origin=do_election_count_vote ]
Feb  6 01:40:19 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC
cause=C_FSA_INTERNAL origin=do_election_check ]
Feb  6 01:40:19 node-1 crmd[22715]:     info: do_dc_takeover: Taking over
DC status for this partition
Feb  6 01:40:19 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_master for section 'all' (origin=local/crmd/122,
version=0.108.212): ok (rc=0)
Feb  6 01:40:19 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section cib (origin=local/crmd/123,
version=0.108.213): ok (rc=0)
Feb  6 01:40:19 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section crm_config (origin=local/crmd/125,
version=0.108.214): ok (rc=0)
Feb  6 01:40:19 node-1 crmd[22715]:     info: join_make_offer: Peer process
on node-2 is not active (yet?): 00000001 2
Feb  6 01:40:19 node-1 crmd[22715]:     info: do_dc_join_offer_all: join-4:
Waiting on 1 outstanding join acks
Feb  6 01:40:19 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section crm_config (origin=local/crmd/127,
version=0.108.215): ok (rc=0)
Feb  6 01:40:19 node-1 crmd[22715]:     info: ais_dispatch_message:
Membership 904: quorum retained
Feb  6 01:40:19 node-1 crmd[22715]:     info: crmd_ais_dispatch: Setting
expected votes to 2
Feb  6 01:40:19 node-1 crmd[22715]:     info: update_dc: Set DC to node-1
(3.0.6)
Feb  6 01:40:19 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section crm_config (origin=local/crmd/130,
version=0.108.216): ok (rc=0)
Feb  6 01:40:19 node-1 crmd[22715]:     info: ais_dispatch_message:
Membership 904: quorum retained
Feb  6 01:40:19 node-1 crmd[22715]:     info: crmd_ais_dispatch: Setting
expected votes to 2
Feb  6 01:40:19 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section crm_config (origin=local/crmd/133,
version=0.108.217): ok (rc=0)
Feb  6 01:40:19 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED
cause=C_FSA_INTERNAL origin=check_join_state ]
Feb  6 01:40:19 node-1 crmd[22715]:     info: do_dc_join_finalize: join-4:
Syncing the CIB from node-1 to the rest of the cluster
Feb  6 01:40:19 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_sync for section 'all' (origin=local/crmd/134,
version=0.108.217): ok (rc=0)
Feb  6 01:40:19 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section nodes (origin=local/crmd/135,
version=0.108.218): ok (rc=0)
Feb  6 01:40:20 node-1 crmd[22715]:     info: do_dc_join_ack: join-4:
Updating node state to member for node-1
Feb  6 01:40:20 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_delete for section //node_state[@uname='node-1']/lrm
(origin=local/crmd/136, version=0.108.219): ok (rc=0)
Feb  6 01:40:20 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED
cause=C_FSA_INTERNAL origin=check_join_state ]
Feb  6 01:40:20 node-1 crmd[22715]:     info: abort_transition_graph:
do_te_invoke:162 - Triggered transition abort (complete=1) : Peer Cancelled
Feb  6 01:40:20 node-1 attrd[22713]:   notice: attrd_local_callback:
Sending full refresh (origin=crmd)
Feb  6 01:40:20 node-1 attrd[22713]:   notice: attrd_trigger_update:
Sending flush op to all hosts for: probe_complete (true)
Feb  6 01:40:20 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section nodes (origin=local/crmd/138,
version=0.108.221): ok (rc=0)
Feb  6 01:40:20 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section cib (origin=local/crmd/140,
version=0.108.223): ok (rc=0)
Feb  6 01:40:20 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:40:20 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:40:20 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 44 (ref=pe_calc-dc-1360114820-76) derived from
/var/lib/pengine/pe-input-164.bz2
Feb  6 01:40:20 node-1 crmd[22715]:   notice: run_graph: ==== Transition 44
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-164.bz2): Complete
Feb  6 01:40:20 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:40:20 node-1 pengine[22714]:   notice: process_pe_message:
Transition 44: PEngine Input stored in: /var/lib/pengine/pe-input-164.bz2
Feb  6 01:40:23 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:40:30 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:40:30 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:40:30 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:40:30 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:40:30 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:40:30 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 45 (ref=pe_calc-dc-1360114830-77) derived from
/var/lib/pengine/pe-input-165.bz2
Feb  6 01:40:30 node-1 crmd[22715]:   notice: run_graph: ==== Transition 45
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-165.bz2): Complete
Feb  6 01:40:30 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:40:30 node-1 pengine[22714]:   notice: process_pe_message:
Transition 45: PEngine Input stored in: /var/lib/pengine/pe-input-165.bz2
Feb  6 01:40:34 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:40:39 node-1 crmd[22715]:     info: do_election_count_vote:
Election 4 (owner: node-2) pass: vote from node-2 (Uptime)
Feb  6 01:40:39 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL
origin=do_election_count_vote ]
Feb  6 01:40:39 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC
cause=C_FSA_INTERNAL origin=do_election_check ]
Feb  6 01:40:39 node-1 crmd[22715]:     info: do_dc_takeover: Taking over
DC status for this partition
Feb  6 01:40:39 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_master for section 'all' (origin=local/crmd/143,
version=0.108.225): ok (rc=0)
Feb  6 01:40:39 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section cib (origin=local/crmd/144,
version=0.108.226): ok (rc=0)
Feb  6 01:40:39 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section crm_config (origin=local/crmd/146,
version=0.108.227): ok (rc=0)
Feb  6 01:40:39 node-1 crmd[22715]:     info: join_make_offer: Peer process
on node-2 is not active (yet?): 00000001 2
Feb  6 01:40:39 node-1 crmd[22715]:     info: do_dc_join_offer_all: join-5:
Waiting on 1 outstanding join acks
Feb  6 01:40:39 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section crm_config (origin=local/crmd/148,
version=0.108.228): ok (rc=0)
Feb  6 01:40:39 node-1 crmd[22715]:     info: ais_dispatch_message:
Membership 904: quorum retained
Feb  6 01:40:39 node-1 crmd[22715]:     info: crmd_ais_dispatch: Setting
expected votes to 2
Feb  6 01:40:39 node-1 crmd[22715]:     info: update_dc: Set DC to node-1
(3.0.6)
Feb  6 01:40:39 node-1 crmd[22715]:     info: ais_dispatch_message:
Membership 904: quorum retained
Feb  6 01:40:39 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section crm_config (origin=local/crmd/151,
version=0.108.229): ok (rc=0)
Feb  6 01:40:39 node-1 crmd[22715]:     info: crmd_ais_dispatch: Setting
expected votes to 2
Feb  6 01:40:39 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section crm_config (origin=local/crmd/154,
version=0.108.230): ok (rc=0)
Feb  6 01:40:39 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED
cause=C_FSA_INTERNAL origin=check_join_state ]
Feb  6 01:40:39 node-1 crmd[22715]:     info: do_dc_join_finalize: join-5:
Syncing the CIB from node-1 to the rest of the cluster
Feb  6 01:40:39 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_sync for section 'all' (origin=local/crmd/155,
version=0.108.230): ok (rc=0)
Feb  6 01:40:39 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section nodes (origin=local/crmd/156,
version=0.108.231): ok (rc=0)
Feb  6 01:40:40 node-1 crmd[22715]:     info: do_dc_join_ack: join-5:
Updating node state to member for node-1
Feb  6 01:40:40 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_delete for section //node_state[@uname='node-1']/lrm
(origin=local/crmd/157, version=0.108.232): ok (rc=0)
Feb  6 01:40:40 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED
cause=C_FSA_INTERNAL origin=check_join_state ]
Feb  6 01:40:40 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section nodes (origin=local/crmd/159,
version=0.108.234): ok (rc=0)
Feb  6 01:40:40 node-1 attrd[22713]:   notice: attrd_local_callback:
Sending full refresh (origin=crmd)
Feb  6 01:40:40 node-1 attrd[22713]:   notice: attrd_trigger_update:
Sending flush op to all hosts for: probe_complete (true)
Feb  6 01:40:40 node-1 crmd[22715]:     info: abort_transition_graph:
do_te_invoke:162 - Triggered transition abort (complete=1) : Peer Cancelled
Feb  6 01:40:40 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section cib (origin=local/crmd/161,
version=0.108.236): ok (rc=0)
Feb  6 01:40:40 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:40:40 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:40:40 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 46 (ref=pe_calc-dc-1360114840-83) derived from
/var/lib/pengine/pe-input-166.bz2
Feb  6 01:40:40 node-1 crmd[22715]:   notice: run_graph: ==== Transition 46
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-166.bz2): Complete
Feb  6 01:40:40 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:40:40 node-1 pengine[22714]:   notice: process_pe_message:
Transition 46: PEngine Input stored in: /var/lib/pengine/pe-input-166.bz2
Feb  6 01:40:44 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:40:50 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:40:50 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:40:50 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:40:50 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:40:50 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:40:50 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 47 (ref=pe_calc-dc-1360114850-84) derived from
/var/lib/pengine/pe-input-167.bz2
Feb  6 01:40:50 node-1 crmd[22715]:   notice: run_graph: ==== Transition 47
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-167.bz2): Complete
Feb  6 01:40:50 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:40:50 node-1 pengine[22714]:   notice: process_pe_message:
Transition 47: PEngine Input stored in: /var/lib/pengine/pe-input-167.bz2
Feb  6 01:40:55 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:40:59 node-1 crmd[22715]:     info: do_election_count_vote:
Election 5 (owner: node-2) pass: vote from node-2 (Uptime)
Feb  6 01:40:59 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL
origin=do_election_count_vote ]
Feb  6 01:40:59 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC
cause=C_FSA_INTERNAL origin=do_election_check ]
Feb  6 01:40:59 node-1 crmd[22715]:     info: do_dc_takeover: Taking over
DC status for this partition
Feb  6 01:40:59 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_master for section 'all' (origin=local/crmd/164,
version=0.108.238): ok (rc=0)
Feb  6 01:40:59 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section cib (origin=local/crmd/165,
version=0.108.239): ok (rc=0)
Feb  6 01:40:59 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section crm_config (origin=local/crmd/167,
version=0.108.240): ok (rc=0)
Feb  6 01:40:59 node-1 crmd[22715]:     info: join_make_offer: Peer process
on node-2 is not active (yet?): 00000001 2
Feb  6 01:40:59 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section crm_config (origin=local/crmd/169,
version=0.108.241): ok (rc=0)
Feb  6 01:40:59 node-1 crmd[22715]:     info: do_dc_join_offer_all: join-6:
Waiting on 1 outstanding join acks
Feb  6 01:40:59 node-1 crmd[22715]:     info: ais_dispatch_message:
Membership 904: quorum retained
Feb  6 01:40:59 node-1 crmd[22715]:     info: crmd_ais_dispatch: Setting
expected votes to 2
Feb  6 01:40:59 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section crm_config (origin=local/crmd/172,
version=0.108.242): ok (rc=0)
Feb  6 01:40:59 node-1 crmd[22715]:     info: ais_dispatch_message:
Membership 904: quorum retained
Feb  6 01:40:59 node-1 crmd[22715]:     info: crmd_ais_dispatch: Setting
expected votes to 2
Feb  6 01:40:59 node-1 crmd[22715]:     info: update_dc: Set DC to node-1
(3.0.6)
Feb  6 01:40:59 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section crm_config (origin=local/crmd/174,
version=0.108.243): ok (rc=0)
Feb  6 01:40:59 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED
cause=C_FSA_INTERNAL origin=check_join_state ]
Feb  6 01:40:59 node-1 crmd[22715]:     info: do_dc_join_finalize: join-6:
Syncing the CIB from node-1 to the rest of the cluster
Feb  6 01:40:59 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_sync for section 'all' (origin=local/crmd/176,
version=0.108.243): ok (rc=0)
Feb  6 01:40:59 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section nodes (origin=local/crmd/177,
version=0.108.244): ok (rc=0)
Feb  6 01:41:00 node-1 crmd[22715]:     info: do_dc_join_ack: join-6:
Updating node state to member for node-1
Feb  6 01:41:00 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_delete for section //node_state[@uname='node-1']/lrm
(origin=local/crmd/178, version=0.108.245): ok (rc=0)
Feb  6 01:41:00 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED
cause=C_FSA_INTERNAL origin=check_join_state ]
Feb  6 01:41:00 node-1 crmd[22715]:     info: abort_transition_graph:
do_te_invoke:162 - Triggered transition abort (complete=1) : Peer Cancelled
Feb  6 01:41:00 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section nodes (origin=local/crmd/180,
version=0.108.247): ok (rc=0)
Feb  6 01:41:00 node-1 attrd[22713]:   notice: attrd_local_callback:
Sending full refresh (origin=crmd)
Feb  6 01:41:00 node-1 attrd[22713]:   notice: attrd_trigger_update:
Sending flush op to all hosts for: probe_complete (true)
Feb  6 01:41:00 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section cib (origin=local/crmd/182,
version=0.108.249): ok (rc=0)
Feb  6 01:41:00 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:41:00 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:41:00 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 48 (ref=pe_calc-dc-1360114860-90) derived from
/var/lib/pengine/pe-input-168.bz2
Feb  6 01:41:00 node-1 crmd[22715]:   notice: run_graph: ==== Transition 48
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-168.bz2): Complete
Feb  6 01:41:00 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:41:00 node-1 pengine[22714]:   notice: process_pe_message:
Transition 48: PEngine Input stored in: /var/lib/pengine/pe-input-168.bz2
Feb  6 01:41:05 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:41:10 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:41:10 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:41:10 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:41:10 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:41:10 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:41:10 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 49 (ref=pe_calc-dc-1360114870-91) derived from
/var/lib/pengine/pe-input-1
bz2
Feb  6 01:41:10 node-1 crmd[22715]:   notice: run_graph: ==== Transition 49
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-169.bz2): Complete
Feb  6 01:41:10 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:41:10 node-1 pengine[22714]:   notice: process_pe_message:
Transition 49: PEngine Input stored in: /var/lib/pengine/pe-input-169.bz2
Feb  6 01:41:16 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:41:19 node-1 crmd[22715]:     info: do_election_count_vote:
Election 6 (owner: node-2) pass: vote from node-2 (Uptime)
Feb  6 01:41:19 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL
origin=do_election_count_vote ]
Feb  6 01:41:19 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC
cause=C_FSA_INTERNAL origin=do_election_check ]
Feb  6 01:41:19 node-1 crmd[22715]:     info: do_dc_takeover: Taking over
DC status for this partition
Feb  6 01:41:19 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_master for section 'all' (origin=local/crmd/185,
version=0.108.251): ok (rc=0)
Feb  6 01:41:19 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section cib (origin=local/crmd/186,
version=0.108.252): ok (rc=0)
Feb  6 01:41:19 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section crm_config (origin=local/crmd/188,
version=0.108.253): ok (rc=0)
Feb  6 01:41:19 node-1 crmd[22715]:     info: join_make_offer: Peer process
on node-2 is not active (yet?): 00000001 2
Feb  6 01:41:19 node-1 crmd[22715]:     info: do_dc_join_offer_all: join-7:
Waiting on 1 outstanding join acks
Feb  6 01:41:19 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section crm_config (origin=local/crmd/190,
version=0.108.254): ok (rc=0)
Feb  6 01:41:19 node-1 crmd[22715]:     info: ais_dispatch_message:
Membership 904: quorum retained
Feb  6 01:41:19 node-1 crmd[22715]:     info: crmd_ais_dispatch: Setting
expected votes to 2
Feb  6 01:41:19 node-1 crmd[22715]:     info: update_dc: Set DC to node-1
(3.0.6)
Feb  6 01:41:19 node-1 crmd[22715]:     info: ais_dispatch_message:
Membership 904: quorum retained
Feb  6 01:41:19 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section crm_config (origin=local/crmd/193,
version=0.108.255): ok (rc=0)
Feb  6 01:41:19 node-1 crmd[22715]:     info: crmd_ais_dispatch: Setting
expected votes to 2
Feb  6 01:41:19 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section crm_config (origin=local/crmd/196,
version=0.108.256): ok (rc=0)
Feb  6 01:41:19 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED
cause=C_FSA_INTERNAL origin=check_join_state ]
Feb  6 01:41:19 node-1 crmd[22715]:     info: do_dc_join_finalize: join-7:
Syncing the CIB from node-1 to the rest of the cluster
Feb  6 01:41:19 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_sync for section 'all' (origin=local/crmd/197,
version=0.108.256): ok (rc=0)
Feb  6 01:41:19 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section nodes (origin=local/crmd/198,
version=0.108.257): ok (rc=0)
Feb  6 01:41:20 node-1 lrmd: [22712]: WARN: G_SIG_dispatch: Dispatch
function for SIGCHLD was delayed 260 ms (> 100 ms) before being called
(GSource: 0x18bfad0)
Feb  6 01:41:20 node-1 lrmd: [22712]: info: G_SIG_dispatch: started at
440364754 should have started at 440364728
Feb  6 01:41:20 node-1 crmd[22715]:     info: do_dc_join_ack: join-7:
Updating node state to member for node-1
Feb  6 01:41:20 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_delete for section //node_state[@uname='node-1']/lrm
(origin=local/crmd/199, version=0.108.258): ok (rc=0)
Feb  6 01:41:20 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED
cause=C_FSA_INTERNAL origin=check_join_state ]
Feb  6 01:41:20 node-1 crmd[22715]:     info: abort_transition_graph:
do_te_invoke:162 - Triggered transition abort (complete=1) : Peer Cancelled
Feb  6 01:41:20 node-1 attrd[22713]:   notice: attrd_local_callback:
Sending full refresh (origin=crmd)
Feb  6 01:41:20 node-1 attrd[22713]:   notice: attrd_trigger_update:
Sending flush op to all hosts for: probe_complete (true)
Feb  6 01:41:20 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section nodes (origin=local/crmd/201,
version=0.108.260): ok (rc=0)
Feb  6 01:41:20 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section cib (origin=local/crmd/203,
version=0.108.262): ok (rc=0)
Feb  6 01:41:20 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:41:20 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:41:20 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 50 (ref=pe_calc-dc-1360114880-97) derived from
/var/lib/pengine/pe-input-170.bz2
Feb  6 01:41:20 node-1 crmd[22715]:   notice: run_graph: ==== Transition 50
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-170.bz2): Complete
Feb  6 01:41:20 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:41:20 node-1 pengine[22714]:   notice: process_pe_message:
Transition 50: PEngine Input stored in: /var/lib/pengine/pe-input-170.bz2
Feb  6 01:41:27 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:41:30 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:41:30 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:41:30 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:41:30 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:41:30 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:41:30 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 51 (ref=pe_calc-dc-1360114890-98) derived from
/var/lib/pengine/pe-input-171.bz2
Feb  6 01:41:30 node-1 crmd[22715]:   notice: run_graph: ==== Transition 51
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-171.bz2): Complete
Feb  6 01:41:30 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:41:30 node-1 pengine[22714]:   notice: process_pe_message:
Transition 51: PEngine Input stored in: /var/lib/pengine/pe-input-171.bz2
Feb  6 01:41:37 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:41:39 node-1 crmd[22715]:     info: do_election_count_vote:
Election 7 (owner: node-2) pass: vote from node-2 (Uptime)
Feb  6 01:41:39 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL
origin=do_election_count_vote ]
Feb  6 01:41:39 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC
cause=C_FSA_INTERNAL origin=do_election_check ]
Feb  6 01:41:39 node-1 crmd[22715]:     info: do_dc_takeover: Taking over
DC status for this partition
Feb  6 01:41:39 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_master for section 'all' (origin=local/crmd/206,
version=0.108.264): ok (rc=0)
Feb  6 01:41:39 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section cib (origin=local/crmd/207,
version=0.108.265): ok (rc=0)
Feb  6 01:41:39 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section crm_config (origin=local/crmd/209,
version=0.108.266): ok (rc=0)
Feb  6 01:41:39 node-1 crmd[22715]:     info: join_make_offer: Peer process
on node-2 is not active (yet?): 00000001 2
Feb  6 01:41:39 node-1 crmd[22715]:     info: do_dc_join_offer_all: join-8:
Waiting on 1 outstanding join acks
Feb  6 01:41:39 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section crm_config (origin=local/crmd/211,
version=0.108.267): ok (rc=0)
Feb  6 01:41:39 node-1 crmd[22715]:     info: ais_dispatch_message:
Membership 904: quorum retained
Feb  6 01:41:39 node-1 crmd[22715]:     info: crmd_ais_dispatch: Setting
expected votes to 2
Feb  6 01:41:39 node-1 crmd[22715]:     info: update_dc: Set DC to node-1
(3.0.6)
Feb  6 01:41:39 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section crm_config (origin=local/crmd/214,
version=0.108.268): ok (rc=0)
Feb  6 01:41:39 node-1 crmd[22715]:     info: ais_dispatch_message:
Membership 904: quorum retained
Feb  6 01:41:39 node-1 crmd[22715]:     info: crmd_ais_dispatch: Setting
expected votes to 2
Feb  6 01:41:39 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section crm_config (origin=local/crmd/217,
version=0.108.269): ok (rc=0)
Feb  6 01:41:39 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED
cause=C_FSA_INTERNAL origin=check_join_state ]
Feb  6 01:41:39 node-1 crmd[22715]:     info: do_dc_join_finalize: join-8:
Syncing the CIB from node-1 to the rest of the cluster
Feb  6 01:41:39 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_sync for section 'all' (origin=local/crmd/218,
version=0.108.269): ok (rc=0)
Feb  6 01:41:39 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section nodes (origin=local/crmd/219,
version=0.108.270): ok (rc=0)
Feb  6 01:41:40 node-1 crmd[22715]:     info: do_dc_join_ack: join-8:
Updating node state to member for node-1
Feb  6 01:41:40 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_delete for section //node_state[@uname='node-1']/lrm
(origin=local/crmd/220, version=0.108.271): ok (rc=0)
Feb  6 01:41:40 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED
cause=C_FSA_INTERNAL origin=check_join_state ]
Feb  6 01:41:40 node-1 crmd[22715]:     info: abort_transition_graph:
do_te_invoke:162 - Triggered transition abort (complete=1) : Peer Cancelled
Feb  6 01:41:40 node-1 attrd[22713]:   notice: attrd_local_callback:
Sending full refresh (origin=crmd)
Feb  6 01:41:40 node-1 attrd[22713]:   notice: attrd_trigger_update:
Sending flush op to all hosts for: probe_complete (true)
Feb  6 01:41:40 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section nodes (origin=local/crmd/222,
version=0.108.273): ok (rc=0)
Feb  6 01:41:40 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section cib (origin=local/crmd/224,
version=0.108.275): ok (rc=0)
Feb  6 01:41:40 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:41:40 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:41:40 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 52 (ref=pe_calc-dc-1360114900-104) derived from
/var/lib/pengine/pe-input-172.bz2
Feb  6 01:41:40 node-1 crmd[22715]:   notice: run_graph: ==== Transition 52
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-172.bz2): Complete
Feb  6 01:41:40 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:41:40 node-1 pengine[22714]:   notice: process_pe_message:
Transition 52: PEngine Input stored in: /var/lib/pengine/pe-input-172.bz2
Feb  6 01:41:48 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:41:50 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:41:50 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:41:50 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:41:50 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:41:50 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:41:50 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 53 (ref=pe_calc-dc-1360114910-105) derived from
/var/lib/pengine/pe-input-173.bz2
Feb  6 01:41:50 node-1 crmd[22715]:   notice: run_graph: ==== Transition 53
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-173.bz2): Complete
Feb  6 01:41:50 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:41:50 node-1 pengine[22714]:   notice: process_pe_message:
Transition 53: PEngine Input stored in: /var/lib/pengine/pe-input-173.bz2
Feb  6 01:41:59 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:41:59 node-1 crmd[22715]:     info: do_election_count_vote:
Election 8 (owner: node-2) pass: vote from node-2 (Uptime)
Feb  6 01:41:59 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL
origin=do_election_count_vote ]
Feb  6 01:41:59 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC
cause=C_FSA_INTERNAL origin=do_election_check ]
Feb  6 01:41:59 node-1 crmd[22715]:     info: do_dc_takeover: Taking over
DC status for this partition
Feb  6 01:41:59 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_master for section 'all' (origin=local/crmd/227,
version=0.108.277): ok (rc=0)
Feb  6 01:41:59 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section cib (origin=local/crmd/228,
version=0.108.278): ok (rc=0)
Feb  6 01:41:59 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section crm_config (origin=local/crmd/230,
version=0.108.279): ok (rc=0)
Feb  6 01:41:59 node-1 crmd[22715]:     info: join_make_offer: Peer process
on node-2 is not active (yet?): 00000001 2
Feb  6 01:41:59 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section crm_config (origin=local/crmd/232,
version=0.108.280): ok (rc=0)
Feb  6 01:41:59 node-1 crmd[22715]:     info: do_dc_join_offer_all: join-9:
Waiting on 1 outstanding join acks
Feb  6 01:41:59 node-1 crmd[22715]:     info: ais_dispatch_message:
Membership 904: quorum retained
Feb  6 01:41:59 node-1 crmd[22715]:     info: crmd_ais_dispatch: Setting
expected votes to 2
Feb  6 01:41:59 node-1 crmd[22715]:     info: update_dc: Set DC to node-1
(3.0.6)
Feb  6 01:41:59 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section crm_config (origin=local/crmd/235,
version=0.108.281): ok (rc=0)
Feb  6 01:41:59 node-1 crmd[22715]:     info: ais_dispatch_message:
Membership 904: quorum retained
Feb  6 01:41:59 node-1 crmd[22715]:     info: crmd_ais_dispatch: Setting
expected votes to 2
Feb  6 01:41:59 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section crm_config (origin=local/crmd/238,
version=0.108.282): ok (rc=0)
Feb  6 01:41:59 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED
cause=C_FSA_INTERNAL origin=check_join_state ]
Feb  6 01:41:59 node-1 crmd[22715]:     info: do_dc_join_finalize: join-9:
Syncing the CIB from node-1 to the rest of the cluster
Feb  6 01:41:59 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_sync for section 'all' (origin=local/crmd/239,
version=0.108.282): ok (rc=0)
Feb  6 01:41:59 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section nodes (origin=local/crmd/240,
version=0.108.283): ok (rc=0)
Feb  6 01:42:00 node-1 lrmd: [22712]: WARN: G_SIG_dispatch: Dispatch
function for SIGCHLD was delayed 530 ms (> 100 ms) before being called
(GSource: 0x18bfad0)
Feb  6 01:42:00 node-1 lrmd: [22712]: info: G_SIG_dispatch: started at
440368741 should have started at 440368688
Feb  6 01:42:00 node-1 crmd[22715]:     info: do_dc_join_ack: join-9:
Updating node state to member for node-1
Feb  6 01:42:00 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_delete for section //node_state[@uname='node-1']/lrm
(origin=local/crmd/241, version=0.108.284): ok (rc=0)
Feb  6 01:42:00 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED
cause=C_FSA_INTERNAL origin=check_join_state ]
Feb  6 01:42:00 node-1 crmd[22715]:     info: abort_transition_graph:
do_te_invoke:162 - Triggered transition abort (complete=1) : Peer Cancelled
Feb  6 01:42:00 node-1 attrd[22713]:   notice: attrd_local_callback:
Sending full refresh (origin=crmd)
Feb  6 01:42:00 node-1 attrd[22713]:   notice: attrd_trigger_update:
Sending flush op to all hosts for: probe_complete (true)
Feb  6 01:42:00 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section nodes (origin=local/crmd/243,
version=0.108.286): ok (rc=0)
Feb  6 01:42:00 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section cib (origin=local/crmd/245,
version=0.108.288): ok (rc=0)
Feb  6 01:42:00 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:42:00 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:42:00 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 54 (ref=pe_calc-dc-1360114920-111) derived from
/var/lib/pengine/pe-input-174.bz2
Feb  6 01:42:00 node-1 crmd[22715]:   notice: run_graph: ==== Transition 54
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-174.bz2): Complete
Feb  6 01:42:00 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:42:00 node-1 pengine[22714]:   notice: process_pe_message:
Transition 54: PEngine Input stored in: /var/lib/pengine/pe-input-174.bz2
Feb  6 01:42:10 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:42:10 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:42:10 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:42:10 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:42:10 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:42:10 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:42:10 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 55 (ref=pe_calc-dc-1360114930-112) derived from
/var/lib/pengine/pe-input-175.bz2
Feb  6 01:42:10 node-1 crmd[22715]:   notice: run_graph: ==== Transition 55
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-175.bz2): Complete
Feb  6 01:42:10 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:42:10 node-1 pengine[22714]:   notice: process_pe_message:
Transition 55: PEngine Input stored in: /var/lib/pengine/pe-input-175.bz2
Feb  6 01:42:19 node-1 crmd[22715]:     info: do_election_count_vote:
Election 9 (owner: node-2) pass: vote from node-2 (Uptime)
Feb  6 01:42:19 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL
origin=do_election_count_vote ]
Feb  6 01:42:19 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC
cause=C_FSA_INTERNAL origin=do_election_check ]
Feb  6 01:42:19 node-1 crmd[22715]:     info: do_dc_takeover: Taking over
DC status for this partition
Feb  6 01:42:19 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_master for section 'all' (origin=local/crmd/248,
version=0.108.290): ok (rc=0)
Feb  6 01:42:19 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section cib (origin=local/crmd/249,
version=0.108.291): ok (rc=0)
Feb  6 01:42:19 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section crm_config (origin=local/crmd/251,
version=0.108.292): ok (rc=0)
Feb  6 01:42:19 node-1 crmd[22715]:     info: join_make_offer: Peer process
on node-2 is not active (yet?): 00000001 2
Feb  6 01:42:19 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section crm_config (origin=local/crmd/253,
version=0.108.293): ok (rc=0)
Feb  6 01:42:19 node-1 crmd[22715]:     info: do_dc_join_offer_all:
join-10: Waiting on 1 outstanding join acks
Feb  6 01:42:19 node-1 crmd[22715]:     info: ais_dispatch_message:
Membership 904: quorum retained
Feb  6 01:42:19 node-1 crmd[22715]:     info: crmd_ais_dispatch: Setting
expected votes to 2
Feb  6 01:42:19 node-1 crmd[22715]:     info: update_dc: Set DC to node-1
(3.0.6)
Feb  6 01:42:19 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section crm_config (origin=local/crmd/256,
version=0.108.294): ok (rc=0)
Feb  6 01:42:19 node-1 crmd[22715]:     info: ais_dispatch_message:
Membership 904: quorum retained
Feb  6 01:42:19 node-1 crmd[22715]:     info: crmd_ais_dispatch: Setting
expected votes to 2
Feb  6 01:42:19 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section crm_config (origin=local/crmd/259,
version=0.108.295): ok (rc=0)
Feb  6 01:42:19 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED
cause=C_FSA_INTERNAL origin=check_join_state ]
Feb  6 01:42:19 node-1 crmd[22715]:     info: do_dc_join_finalize: join-10:
Syncing the CIB from node-1 to the rest of the cluster
Feb  6 01:42:19 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_sync for section 'all' (origin=local/crmd/260,
version=0.108.295): ok (rc=0)
Feb  6 01:42:19 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section nodes (origin=local/crmd/261,
version=0.108.296): ok (rc=0)
Feb  6 01:42:20 node-1 crmd[22715]:     info: do_dc_join_ack: join-10:
Updating node state to member for node-1
Feb  6 01:42:20 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_delete for section //node_state[@uname='node-1']/lrm
(origin=local/crmd/262, version=0.108.297): ok (rc=0)
Feb  6 01:42:20 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED
cause=C_FSA_INTERNAL origin=check_join_state ]
Feb  6 01:42:20 node-1 crmd[22715]:     info: abort_transition_graph:
do_te_invoke:162 - Triggered transition abort (complete=1) : Peer Cancelled
Feb  6 01:42:20 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section nodes (origin=local/crmd/264,
version=0.108.299): ok (rc=0)
Feb  6 01:42:20 node-1 cib[22710]:     info: cib_process_request: Operation
complete: op cib_modify for section cib (origin=local/crmd/266,
version=0.108.301): ok (rc=0)
Feb  6 01:42:20 node-1 attrd[22713]:   notice: attrd_local_callback:
Sending full refresh (origin=crmd)
Feb  6 01:42:20 node-1 attrd[22713]:   notice: attrd_trigger_update:
Sending flush op to all hosts for: probe_complete (true)
Feb  6 01:42:20 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:42:20 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:42:20 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 56 (ref=pe_calc-dc-1360114940-118) derived from
/var/lib/pengine/pe-input-176.bz2
Feb  6 01:42:20 node-1 crmd[22715]:   notice: run_graph: ==== Transition 56
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-176.bz2): Complete
Feb  6 01:42:20 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:42:20 node-1 pengine[22714]:   notice: process_pe_message:
Transition 56: PEngine Input stored in: /var/lib/pengine/pe-input-176.bz2
Feb  6 01:42:21 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1
Feb  6 01:42:30 node-1 crmd[22715]:     info: crm_timer_popped: PEngine
Recheck Timer (I_PE_CALC) just popped (10000ms)
Feb  6 01:42:30 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED
origin=crm_timer_popped ]
Feb  6 01:42:30 node-1 crmd[22715]:     info: do_state_transition:
Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED
Feb  6 01:42:30 node-1 pengine[22714]:   notice: unpack_config: On loss of
CCM Quorum: Ignore
Feb  6 01:42:30 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response ]
Feb  6 01:42:30 node-1 crmd[22715]:     info: do_te_invoke: Processing
graph 57 (ref=pe_calc-dc-1360114950-119) derived from
/var/lib/pengine/pe-input-177.bz2
Feb  6 01:42:30 node-1 crmd[22715]:   notice: run_graph: ==== Transition 57
(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,
Source=/var/lib/pengine/pe-input-177.bz2): Complete
Feb  6 01:42:30 node-1 crmd[22715]:   notice: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS
cause=C_FSA_INTERNAL origin=notify_crmd ]
Feb  6 01:42:30 node-1 pengine[22714]:   notice: process_pe_message:
Transition 57: PEngine Input stored in: /var/lib/pengine/pe-input-177.bz2
Feb  6 01:42:32 node-1 stonith-ng[22711]:     info: stonith_command:
Processed st_execute from lrmd: rc=-1

.. another election and another .. forever






On Thu, Feb 7, 2013 at 7:58 AM, Andrew Beekhof <andrew at beekhof.net> wrote:

> On Wed, Feb 6, 2013 at 9:11 PM, Tal Yalon <yalon at infinidat.com> wrote:
> > Hi all,
> >
> > I have a 2-node cluster, where node-2 got fenced and now after reboot
> tries
> > to rejoin the cluster but fails and gets stuck in a loop for hours and
> never
> > joins back.
> >
> > After another reboot it managed to join, and there was no time difference
> > between the nodes.
> >
> > Below is corosync/pacemaker log of node-2 (the one that was stuck in the
> > loop).
>
> Unfortunately we need the other one.
>
> >Any help would be appreciated, since I have no clue as to what
> > happened.
> >
> > Thanks,
> > Tal
> >
> >
> > Feb  6 01:39:32 node-2 corosync[27428]:   [MAIN  ] Corosync Cluster
> Engine
> > ('1.4.1'): started and ready to provide service.
> > Feb  6 01:39:32 node-2 corosync[27428]:   [MAIN  ] Corosync built-in
> > features: nss dbus rdma snmp
> > Feb  6 01:39:32 node-2 corosync[27428]:   [MAIN  ] Successfully read main
> > configuration file '/etc/corosync/corosync.conf'.
> > Feb  6 01:39:32 node-2 corosync[27428]:   [TOTEM ] Initializing transport
> > (UDP/IP Unicast).
> > Feb  6 01:39:32 node-2 corosync[27428]:   [TOTEM ] Initializing
> > transmit/receive security: libtomcrypt SOBER128/SHA1HMAC (mode 0).
> > Feb  6 01:39:32 node-2 corosync[27428]:   [TOTEM ] The network interface
> > [9.151.142.20] is now up.
> > Feb  6 01:39:32 node-2 corosync[27428]:   [pcmk  ] Logging: Initialized
> > pcmk_startup
> > Feb  6 01:39:32 node-2 corosync[27428]:   [SERV  ] Service engine loaded:
> > Pacemaker Cluster Manager 1.1.6
> > Feb  6 01:39:32 node-2 corosync[27428]:   [SERV  ] Service engine loaded:
> > corosync extended virtual synchrony service
> > Feb  6 01:39:32 node-2 corosync[27428]:   [SERV  ] Service engine loaded:
> > corosync configuration service
> > Feb  6 01:39:32 node-2 corosync[27428]:   [SERV  ] Service engine loaded:
> > corosync cluster closed process group service v1.01
> > Feb  6 01:39:32 node-2 corosync[27428]:   [SERV  ] Service engine loaded:
> > corosync cluster config database access v1.01
> > Feb  6 01:39:32 node-2 corosync[27428]:   [SERV  ] Service engine loaded:
> > corosync profile loading service
> > Feb  6 01:39:32 node-2 corosync[27428]:   [SERV  ] Service engine loaded:
> > corosync cluster quorum service v0.1
> > Feb  6 01:39:32 node-2 corosync[27428]:   [MAIN  ] Compatibility mode
> set to
> > whitetank.  Using V1 and V2 of the synchronization engine.
> > Feb  6 01:39:32 node-2 corosync[27428]:   [TOTEM ] A processor joined or
> > left the membership and a new membership was formed.
> > Feb  6 01:39:32 node-2 corosync[27428]:   [CPG   ] chosen downlist:
> sender
> > r(0) ip(9.151.142.20) ; members(old:0 left:0)
> > Feb  6 01:39:32 node-2 corosync[27428]:   [MAIN  ] Completed service
> > synchronization, ready to provide service.
> > Feb  6 01:39:37 node-2 pacemakerd[27466]:     info: crm_log_init_worker:
> > Changed active directory to /var/lib/heartbeat/cores/root
> > Feb  6 01:39:37 node-2 pacemakerd[27466]:   notice: main: Starting
> Pacemaker
> > 1.1.7-6.el6 (Build: 148fccfd5985c5590cc601123c6c16e966b85d14):
> > generated-manpages agent-manpages ascii-docs publican-docs ncurses
> > trace-logging libqb  corosync-plugin cman
> > Feb  6 01:39:37 node-2 pacemakerd[27466]:     info: main: Maximum core
> file
> > size is: 18446744073709551615
> > Feb  6 01:39:37 node-2 pacemakerd[27466]:   notice:
> update_node_processes:
> > 0xb31fe0 Node 2 now known as node-2, was:
> > Feb  6 01:39:37 node-2 pacemakerd[27466]:     info: start_child: Forked
> > child 27470 for process cib
> > Feb  6 01:39:37 node-2 pacemakerd[27466]:     info: start_child: Forked
> > child 27471 for process stonith-ng
> > Feb  6 01:39:37 node-2 pacemakerd[27466]:     info: start_child: Forked
> > child 27472 for process lrmd
> > Feb  6 01:39:37 node-2 pacemakerd[27466]:     info: start_child: Forked
> > child 27473 for process attrd
> > Feb  6 01:39:37 node-2 pacemakerd[27466]:     info: start_child: Forked
> > child 27474 for process pengine
> > Feb  6 01:39:37 node-2 pacemakerd[27466]:     info: start_child: Forked
> > child 27475 for process crmd
> > Feb  6 01:39:37 node-2 pacemakerd[27466]:     info: main: Starting
> mainloop
> > Feb  6 01:39:37 node-2 lrmd: [27472]: info: G_main_add_SignalHandler:
> Added
> > signal handler for signal 15
> > Feb  6 01:39:37 node-2 stonith-ng[27471]:     info: crm_log_init_worker:
> > Changed active directory to /var/lib/heartbeat/cores/root
> > Feb  6 01:39:37 node-2 stonith-ng[27471]:     info: get_cluster_type:
> > Cluster type is: 'openais'
> > Feb  6 01:39:37 node-2 stonith-ng[27471]:   notice: crm_cluster_connect:
> > Connecting to cluster infrastructure: classic openais (with plugin)
> > Feb  6 01:39:37 node-2 stonith-ng[27471]:     info:
> > init_ais_connection_classic: Creating connection to our Corosync plugin
> > Feb  6 01:39:37 node-2 stonith-ng[27471]:     info:
> > init_ais_connection_classic: AIS connection established
> > Feb  6 01:39:37 node-2 stonith-ng[27471]:     info: get_ais_nodeid:
> Server
> > details: id=2 uname=node-2 cname=pcmk
> > Feb  6 01:39:37 node-2 stonith-ng[27471]:     info:
> > init_ais_connection_once: Connection to 'classic openais (with plugin)':
> > established
> > Feb  6 01:39:37 node-2 stonith-ng[27471]:     info: crm_new_peer: Node
> > node-2 now has id: 2
> > Feb  6 01:39:37 node-2 stonith-ng[27471]:     info: crm_new_peer: Node 2
> is
> > now known as node-2
> > Feb  6 01:39:37 node-2 crmd[27475]:     info: crm_log_init_worker:
> Changed
> > active directory to /var/lib/heartbeat/cores/hacluster
> > Feb  6 01:39:37 node-2 crmd[27475]:   notice: main: CRM Hg Version:
> > 148fccfd5985c5590cc601123c6c16e966b85d14
> > Feb  6 01:39:37 node-2 lrmd: [27472]: info: G_main_add_SignalHandler:
> Added
> > signal handler for signal 17
> > Feb  6 01:39:37 node-2 lrmd: [27472]: info: enabling coredumps
> > Feb  6 01:39:37 node-2 lrmd: [27472]: info: G_main_add_SignalHandler:
> Added
> > signal handler for signal 10
> > Feb  6 01:39:37 node-2 lrmd: [27472]: info: G_main_add_SignalHandler:
> Added
> > signal handler for signal 12
> > Feb  6 01:39:37 node-2 lrmd: [27472]: info: Started.
> > Feb  6 01:39:37 node-2 cib[27470]:     info: crm_log_init_worker: Changed
> > active directory to /var/lib/heartbeat/cores/hacluster
> > Feb  6 01:39:37 node-2 cib[27470]:     info: retrieveCib: Reading cluster
> > configuration from: /var/lib/heartbeat/crm/cib.xml (digest:
> > /var/lib/heartbeat/crm/cib.xml.sig)
> > Feb  6 01:39:37 node-2 attrd[27473]:   notice: crm_cluster_connect:
> > Connecting to cluster infrastructure: classic openais (with plugin)
> > Feb  6 01:39:37 node-2 attrd[27473]:   notice: main: Starting mainloop...
> > Feb  6 01:39:37 node-2 cib[27470]:     info: validate_with_relaxng:
> Creating
> > RNG parser context
> > Feb  6 01:39:37 node-2 cib[27470]:     info: startCib: CIB Initialization
> > completed successfully
> > Feb  6 01:39:37 node-2 cib[27470]:     info: get_cluster_type: Cluster
> type
> > is: 'openais'
> > Feb  6 01:39:37 node-2 cib[27470]:   notice: crm_cluster_connect:
> Connecting
> > to cluster infrastructure: classic openais (with plugin)
> > Feb  6 01:39:37 node-2 cib[27470]:     info: init_ais_connection_classic:
> > Creating connection to our Corosync plugin
> > Feb  6 01:39:37 node-2 cib[27470]:     info: init_ais_connection_classic:
> > AIS connection established
> > Feb  6 01:39:37 node-2 cib[27470]:     info: get_ais_nodeid: Server
> details:
> > id=2 uname=node-2 cname=pcmk
> > Feb  6 01:39:37 node-2 cib[27470]:     info: init_ais_connection_once:
> > Connection to 'classic openais (with plugin)': established
> > Feb  6 01:39:37 node-2 cib[27470]:     info: crm_new_peer: Node node-2
> now
> > has id: 2
> > Feb  6 01:39:37 node-2 cib[27470]:     info: crm_new_peer: Node 2 is now
> > known as node-2
> > Feb  6 01:39:37 node-2 cib[27470]:     info: cib_init: Starting cib
> mainloop
> > Feb  6 01:39:37 node-2 cib[27470]:     info: ais_dispatch_message:
> > Membership 900: quorum still lost
> > Feb  6 01:39:37 node-2 cib[27470]:     info: crm_update_peer: Node
> node-2:
> > id=2 state=member (new) addr=r(0) ip(9.151.142.20)  (new) votes=1 (new)
> > born=0 seen=900 proc=00000000000000000000000000000000
> > Feb  6 01:39:37 node-2 cib[27470]:     info: ais_dispatch_message:
> > Membership 904: quorum still lost
> > Feb  6 01:39:37 node-2 cib[27470]:     info: crm_new_peer: Node <null>
> now
> > has id: 1
> > Feb  6 01:39:37 node-2 cib[27470]:     info: crm_update_peer: Node
> (null):
> > id=1 state=member (new) addr=r(0) ip(9.151.142.10)  votes=0 born=0
> seen=904
> > proc=00000000000000000000000000000000
> > Feb  6 01:39:37 node-2 corosync[27428]:   [TOTEM ] A processor joined or
> > left the membership and a new membership was formed.
> > Feb  6 01:39:37 node-2 cib[27470]:   notice: ais_dispatch_message:
> > Membership 904: quorum acquired
> > Feb  6 01:39:37 node-2 cib[27470]:     info: crm_get_peer: Node 1 is now
> > known as node-1
> > Feb  6 01:39:37 node-2 cib[27470]:     info: crm_update_peer: Node
> node-1:
> > id=1 state=member addr=r(0) ip(9.151.142.10)  votes=1 (new) born=896
> > seen=904 proc=00000000000000000000000000000000
> > Feb  6 01:39:37 node-2 corosync[27428]:   [CPG   ] chosen downlist:
> sender
> > r(0) ip(9.151.142.10) ; members(old:1 left:0)
> > Feb  6 01:39:37 node-2 corosync[27428]:   [MAIN  ] Completed service
> > synchronization, ready to provide service.
> > Feb  6 01:39:38 node-2 cib[27470]:  warning: cib_peer_callback:
> Discarding
> > cib_apply_diff message (30) from node-1: not in our membership
> > Feb  6 01:39:38 node-2 cib[27470]:  warning: cib_peer_callback:
> Discarding
> > cib_apply_diff message (31) from node-1: not in our membership
> > Feb  6 01:39:38 node-2 cib[27470]:  warning: cib_peer_callback:
> Discarding
> > cib_apply_diff message (32) from node-1: not in our membership
> > Feb  6 01:39:38 node-2 cib[27470]:  warning: cib_peer_callback:
> Discarding
> > cib_apply_diff message (33) from node-1: not in our membership
> > Feb  6 01:39:38 node-2 cib[27470]:  warning: cib_peer_callback:
> Discarding
> > cib_apply_diff message (34) from node-1: not in our membership
> > Feb  6 01:39:38 node-2 cib[27470]:  warning: cib_peer_callback:
> Discarding
> > cib_apply_diff message (35) from node-1: not in our membership
> > Feb  6 01:39:38 node-2 cib[27470]:  warning: cib_peer_callback:
> Discarding
> > cib_apply_diff message (36) from node-1: not in our membership
> > Feb  6 01:39:38 node-2 cib[27470]:  warning: cib_peer_callback:
> Discarding
> > cib_apply_diff message (37) from node-1: not in our membership
> > Feb  6 01:39:38 node-2 cib[27470]:  warning: cib_peer_callback:
> Discarding
> > cib_apply_diff message (38) from node-1: not in our membership
> > Feb  6 01:39:38 node-2 stonith-ng[27471]:   notice: setup_cib: Watching
> for
> > stonith topology changes
> > Feb  6 01:39:38 node-2 stonith-ng[27471]:     info: main: Starting
> > stonith-ng mainloop
> > Feb  6 01:39:38 node-2 crmd[27475]:     info: do_cib_control: CIB
> connection
> > established
> > Feb  6 01:39:38 node-2 crmd[27475]:     info: get_cluster_type: Cluster
> type
> > is: 'openais'
> > Feb  6 01:39:38 node-2 crmd[27475]:   notice: crm_cluster_connect:
> > Connecting to cluster infrastructure: classic openais (with plugin)
> > Feb  6 01:39:38 node-2 crmd[27475]:     info:
> init_ais_connection_classic:
> > Creating connection to our Corosync plugin
> > Feb  6 01:39:38 node-2 crmd[27475]:     info:
> init_ais_connection_classic:
> > AIS connection established
> > Feb  6 01:39:38 node-2 crmd[27475]:     info: get_ais_nodeid: Server
> > details: id=2 uname=node-2 cname=pcmk
> > Feb  6 01:39:38 node-2 crmd[27475]:     info: init_ais_connection_once:
> > Connection to 'classic openais (with plugin)': established
> > Feb  6 01:39:38 node-2 crmd[27475]:     info: crm_new_peer: Node node-2
> now
> > has id: 2
> > Feb  6 01:39:38 node-2 crmd[27475]:     info: crm_new_peer: Node 2 is now
> > known as node-2
> > Feb  6 01:39:38 node-2 crmd[27475]:     info: ais_status_callback:
> status:
> > node-2 is now unknown
> > Feb  6 01:39:38 node-2 crmd[27475]:     info: do_ha_control: Connected to
> > the cluster
> > Feb  6 01:39:38 node-2 crmd[27475]:     info: do_started: Delaying
> start, no
> > membership data (0000000000100000)
> > Feb  6 01:39:38 node-2 crmd[27475]:   notice: ais_dispatch_message:
> > Membership 904: quorum acquired
> > Feb  6 01:39:38 node-2 crmd[27475]:     info: ais_status_callback:
> status:
> > node-2 is now member (was unknown)
> > Feb  6 01:39:38 node-2 crmd[27475]:     info: crm_update_peer: Node
> node-2:
> > id=2 state=member (new) addr=r(0) ip(9.151.142.20)  (new) votes=1 (new)
> > born=904 seen=904 proc=00000000000000000000000000000000
> > Feb  6 01:39:38 node-2 crmd[27475]:     info: crm_new_peer: Node node-1
> now
> > has id: 1
> > Feb  6 01:39:38 node-2 crmd[27475]:     info: crm_new_peer: Node 1 is now
> > known as node-1
> > Feb  6 01:39:38 node-2 crmd[27475]:     info: ais_status_callback:
> status:
> > node-1 is now unknown
> > Feb  6 01:39:38 node-2 crmd[27475]:     info: ais_status_callback:
> status:
> > node-1 is now member (was unknown)
> > Feb  6 01:39:38 node-2 crmd[27475]:     info: crm_update_peer: Node
> node-1:
> > id=1 state=member (new) addr=r(0) ip(9.151.142.10)  votes=1 born=896
> > seen=904 proc=00000000000000000000000000000000
> > Feb  6 01:39:38 node-2 crmd[27475]:     info: ais_dispatch_message:
> > Membership 904: quorum retained
> > Feb  6 01:39:38 node-2 crmd[27475]:   notice: do_started: The local CRM
> is
> > operational
> > Feb  6 01:39:38 node-2 crmd[27475]:   notice: do_state_transition: State
> > transition S_STARTING -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL
> > origin=do_started ]
> > Feb  6 01:39:38 node-2 pacemakerd[27466]:   notice:
> update_node_processes:
> > 0xb3cdd0 Node 1 now known as node-1, was:
> > Feb  6 01:39:38 node-2 stonith-ng[27471]:     info: crm_new_peer: Node
> > node-1 now has id: 1
> > Feb  6 01:39:38 node-2 stonith-ng[27471]:     info: crm_new_peer: Node 1
> is
> > now known as node-1
> > Feb  6 01:39:39 node-2 crmd[27475]:   notice: crmd_peer_update: Status
> > update: Client node-2/crmd now has status [online] (DC=<null>)
> > Feb  6 01:39:39 node-2 crmd[27475]:   notice: crmd_peer_update: Status
> > update: Client node-1/crmd now has status [online] (DC=<null>)
> > Feb  6 01:39:39 node-2 cib[27470]:     info: cib_replace_notify:
> Replaced:
> > -1.-1.-1 -> 0.108.191 from node-1
> > Feb  6 01:39:59 node-2 crmd[27475]:     info: crm_timer_popped: Election
> > Trigger (I_DC_TIMEOUT) just popped (20000ms)
> > Feb  6 01:39:59 node-2 crmd[27475]:  warning: do_log: FSA: Input
> > I_DC_TIMEOUT from crm_timer_popped() received in state S_PENDING
> > Feb  6 01:39:59 node-2 crmd[27475]:   notice: do_state_transition: State
> > transition S_PENDING -> S_ELECTION [ input=I_DC_TIMEOUT
> cause=C_TIMER_POPPED
> > origin=crm_timer_popped ]
> > Feb  6 01:39:59 node-2 crmd[27475]:     info: do_election_count_vote:
> > Election 3 (owner: node-1) lost: vote from node-1 (Uptime)
> > Feb  6 01:39:59 node-2 crmd[27475]:   notice: do_state_transition: State
> > transition S_ELECTION -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL
> > origin=do_election_count_vote ]
> > Feb  6 01:39:59 node-2 crmd[27475]:     info: do_dc_release: DC role
> > released
> > Feb  6 01:39:59 node-2 crmd[27475]:     info: do_te_control:
> Transitioner is
> > now inactive
> > Feb  6 01:40:19 node-2 crmd[27475]:     info: crm_timer_popped: Election
> > Trigger (I_DC_TIMEOUT) just popped (20000ms)
> > Feb  6 01:40:19 node-2 crmd[27475]:  warning: do_log: FSA: Input
> > I_DC_TIMEOUT from crm_timer_popped() received in state S_PENDING
> > Feb  6 01:40:19 node-2 crmd[27475]:   notice: do_state_transition: State
> > transition S_PENDING -> S_ELECTION [ input=I_DC_TIMEOUT
> cause=C_TIMER_POPPED
> > origin=crm_timer_popped ]
> > Feb  6 01:40:19 node-2 crmd[27475]:     info: do_election_count_vote:
> > Election 4 (owner: node-1) lost: vote from node-1 (Uptime)
> > Feb  6 01:40:19 node-2 crmd[27475]:   notice: do_state_transition: State
> > transition S_ELECTION -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL
> > origin=do_election_count_vote ]
> > Feb  6 01:40:19 node-2 crmd[27475]:     info: do_dc_release: DC role
> > released
> > Feb  6 01:40:19 node-2 crmd[27475]:     info: do_te_control:
> Transitioner is
> > now inactive
> > Feb  6 01:40:39 node-2 crmd[27475]:     info: crm_timer_popped: Election
> > Trigger (I_DC_TIMEOUT) just popped (20000ms)
> > Feb  6 01:40:39 node-2 crmd[27475]:  warning: do_log: FSA: Input
> > I_DC_TIMEOUT from crm_timer_popped() received in state S_PENDING
> > Feb  6 01:40:39 node-2 crmd[27475]:   notice: do_state_transition: State
> > transition S_PENDING -> S_ELECTION [ input=I_DC_TIMEOUT
> cause=C_TIMER_POPPED
> > origin=crm_timer_popped ]
> > Feb  6 01:40:39 node-2 crmd[27475]:     info: do_election_count_vote:
> > Election 5 (owner: node-1) lost: vote from node-1 (Uptime)
> > Feb  6 01:40:39 node-2 crmd[27475]:   notice: do_state_transition: State
> > transition S_ELECTION -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL
> > origin=do_election_count_vote ]
> > Feb  6 01:40:39 node-2 crmd[27475]:     info: do_dc_release: DC role
> > released
> > Feb  6 01:40:39 node-2 crmd[27475]:     info: do_te_control:
> Transitioner is
> > now inactive
> > Feb  6 01:40:59 node-2 crmd[27475]:     info: crm_timer_popped: Election
> > Trigger (I_DC_TIMEOUT) just popped (20000ms)
> > Feb  6 01:40:59 node-2 crmd[27475]:  warning: do_log: FSA: Input
> > I_DC_TIMEOUT from crm_timer_popped() received in state S_PENDING
> > Feb  6 01:40:59 node-2 crmd[27475]:   notice: do_state_transition: State
> > transition S_PENDING -> S_ELECTION [ input=I_DC_TIMEOUT
> cause=C_TIMER_POPPED
> > origin=crm_timer_popped ]
> > Feb  6 01:40:59 node-2 crmd[27475]:     info: do_election_count_vote:
> > Election 6 (owner: node-1) lost: vote from node-1 (Uptime)
> > Feb  6 01:40:59 node-2 crmd[27475]:   notice: do_state_transition: State
> > transition S_ELECTION -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL
> > origin=do_election_count_vote ]
> > Feb  6 01:40:59 node-2 crmd[27475]:     info: do_dc_release: DC role
> > released
> > Feb  6 01:40:59 node-2 crmd[27475]:     info: do_te_control:
> Transitioner is
> > now inactive
> >
> >
> > _______________________________________________
> > Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> > http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> >
> > Project Home: http://www.clusterlabs.org
> > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > Bugs: http://bugs.clusterlabs.org
> >
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/pacemaker/attachments/20130207/cbca0b50/attachment-0003.html>