----------------------------- Nov 04 11:08:19 [1937] kvm01.test-cluster1 crmd: info: crm_update_peer_expected: handle_request: Node kvm02.test-cluster1[2] - expected state is now down (was member) Nov 04 11:08:19 [1937] kvm01.test-cluster1 crmd: info: handle_shutdown_request: Creating shutdown request for kvm02.test-cluster1 (state=S_IDLE) Nov 04 11:08:19 [1935] kvm01.test-cluster1 attrd: info: attrd_peer_update: Setting shutdown[kvm02.test-cluster1]: 0 -> 1478254099 from kvm01.test-cluster1 Nov 04 11:08:19 [1929] kvm01.test-cluster1 cib: info: cib_perform_op: Diff: --- 0.23.18 2 Nov 04 11:08:19 [1929] kvm01.test-cluster1 cib: info: cib_perform_op: Diff: +++ 0.23.19 (null) Nov 04 11:08:19 [1929] kvm01.test-cluster1 cib: info: cib_perform_op: + /cib: @num_updates=19 Nov 04 11:08:19 [1929] kvm01.test-cluster1 cib: info: cib_perform_op: + /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-shutdown']: @value=1478254099 Nov 04 11:08:19 [1929] kvm01.test-cluster1 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=kvm02.test-cluster1/attrd/7, version=0.23.19) Nov 04 11:08:19 [1937] kvm01.test-cluster1 crmd: info: abort_transition_graph: Transition aborted by status-2-shutdown, shutdown=1478254099: Transient attribute change (modify cib=0.23.19, source=abort_unless_down:319, path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-shutdown'], 1) Nov 04 11:08:19 [1937] kvm01.test-cluster1 crmd: notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ] Nov 04 11:08:19 [1936] kvm01.test-cluster1 pengine: info: determine_online_status: Node kvm02.test-cluster1 is shutting down Nov 04 11:08:19 [1936] kvm01.test-cluster1 pengine: info: determine_online_status: Node kvm01.test-cluster1 is online Nov 04 11:08:19 [1936] kvm01.test-cluster1 pengine: info: native_print: kvm_samba1 (ocf::heartbeat:VirtualDomain): Started kvm01.test-cluster1 Nov 04 11:08:19 [1936] kvm01.test-cluster1 pengine: info: native_print: kvm_samba2 (ocf::heartbeat:VirtualDomain): Started kvm02.test-cluster1 Nov 04 11:08:19 [1936] kvm01.test-cluster1 pengine: info: RecurringOp: Start recurring monitor (10s) for kvm_samba2 on kvm01.test-cluster1 Nov 04 11:08:19 [1936] kvm01.test-cluster1 pengine: notice: stage6: Scheduling Node kvm02.test-cluster1 for shutdown Nov 04 11:08:19 [1936] kvm01.test-cluster1 pengine: info: LogActions: Leave kvm_samba1 (Started kvm01.test-cluster1) Nov 04 11:08:19 [1936] kvm01.test-cluster1 pengine: notice: LogActions: Migrate kvm_samba2 (Started kvm02.test-cluster1 -> kvm01.test-cluster1) Nov 04 11:08:19 [1936] kvm01.test-cluster1 pengine: notice: process_pe_message: Calculated Transition 7: /var/lib/pacemaker/pengine/pe-input-654.bz2 Nov 04 11:08:19 [1937] kvm01.test-cluster1 crmd: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ] Nov 04 11:08:19 [1937] kvm01.test-cluster1 crmd: info: do_te_invoke: Processing graph 7 (ref=pe_calc-dc-1478254099-31) derived from /var/lib/pacemaker/pengine/pe-input-654.bz2 Nov 04 11:08:19 [1937] kvm01.test-cluster1 crmd: notice: te_rsc_command: Initiating action 12: migrate_to kvm_samba2_migrate_to_0 on kvm02.test-cluster1 Nov 04 11:08:19 [1937] kvm01.test-cluster1 crmd: info: create_operation_update: cib_action_update: Updating resource kvm_samba2 after migrate_to op pending (interval=0) Nov 04 11:08:19 [1929] kvm01.test-cluster1 cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/45) Nov 04 11:08:19 [1929] kvm01.test-cluster1 cib: info: cib_perform_op: Diff: --- 0.23.19 2 Nov 04 11:08:19 [1929] kvm01.test-cluster1 cib: info: cib_perform_op: Diff: +++ 0.23.20 (null) Nov 04 11:08:19 [1929] kvm01.test-cluster1 cib: info: cib_perform_op: + /cib: @num_updates=20 Nov 04 11:08:19 [1929] kvm01.test-cluster1 cib: info: cib_perform_op: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='kvm_samba2']/lrm_rsc_op[@id='kvm_samba2_last_0']: @operation_key=kvm_samba2_migrate_to_0, @operation=migrate_to, @crm-debug-origin=cib_action_update, @transition-key=12:7:0:9d0841ad-adb6-44b4-a299-3cef745cc206, @transition-magic=-1:193;12:7:0:9d0841ad-adb6-44b4-a299-3cef745cc206, @call-id=-1, @rc-code=193, @op-status=-1, @last-run=1478254099, @las Nov 04 11:08:19 [1929] kvm01.test-cluster1 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=kvm01.test-cluster1/crmd/45, version=0.23.20) Nov 04 11:08:24 [1929] kvm01.test-cluster1 cib: info: cib_perform_op: Diff: --- 0.23.20 2 Nov 04 11:08:24 [1929] kvm01.test-cluster1 cib: info: cib_perform_op: Diff: +++ 0.23.21 (null) Nov 04 11:08:24 [1929] kvm01.test-cluster1 cib: info: cib_perform_op: + /cib: @num_updates=21 Nov 04 11:08:24 [1929] kvm01.test-cluster1 cib: info: cib_perform_op: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='kvm_samba2']/lrm_rsc_op[@id='kvm_samba2_last_0']: @crm-debug-origin=do_update_resource, @transition-magic=0:0;12:7:0:9d0841ad-adb6-44b4-a299-3cef745cc206, @call-id=13, @rc-code=0, @op-status=0, @exec-time=4483 Nov 04 11:08:24 [1929] kvm01.test-cluster1 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=kvm02.test-cluster1/crmd/15, version=0.23.21) Nov 04 11:08:24 [1937] kvm01.test-cluster1 crmd: info: match_graph_event: Action kvm_samba2_migrate_to_0 (12) confirmed on kvm02.test-cluster1 (rc=0) Nov 04 11:08:24 [1937] kvm01.test-cluster1 crmd: notice: te_rsc_command: Initiating action 13: migrate_from kvm_samba2_migrate_from_0 on kvm01.test-cluster1 (local) Nov 04 11:08:24 [1937] kvm01.test-cluster1 crmd: info: do_lrm_rsc_op: Performing key=13:7:0:9d0841ad-adb6-44b4-a299-3cef745cc206 op=kvm_samba2_migrate_from_0 Nov 04 11:08:24 [1934] kvm01.test-cluster1 lrmd: info: log_execute: executing - rsc:kvm_samba2 action:migrate_from call_id:12 VirtualDomain(kvm_samba2)[21154]: 2016/11/04_11:08:24 INFO: samba2: live migration from kvm02.test-cluster1 succeeded. Nov 04 11:08:24 [1934] kvm01.test-cluster1 lrmd: info: log_finished: finished - rsc:kvm_samba2 action:migrate_from call_id:12 pid:21154 exit-code:0 exec-time:175ms queue-time:0ms Nov 04 11:08:24 [1937] kvm01.test-cluster1 crmd: notice: process_lrm_event: Operation kvm_samba2_migrate_from_0: ok (node=kvm01.test-cluster1, call=12, rc=0, cib-update=46, confirmed=true) Nov 04 11:08:24 [1929] kvm01.test-cluster1 cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/46) Nov 04 11:08:24 [1929] kvm01.test-cluster1 cib: info: cib_perform_op: Diff: --- 0.23.21 2 Nov 04 11:08:24 [1929] kvm01.test-cluster1 cib: info: cib_perform_op: Diff: +++ 0.23.22 (null) Nov 04 11:08:24 [1929] kvm01.test-cluster1 cib: info: cib_perform_op: + /cib: @num_updates=22 Nov 04 11:08:24 [1929] kvm01.test-cluster1 cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='kvm_samba2']/lrm_rsc_op[@id='kvm_samba2_last_0']: @operation_key=kvm_samba2_migrate_from_0, @operation=migrate_from, @transition-key=13:7:0:9d0841ad-adb6-44b4-a299-3cef745cc206, @transition-magic=0:0;13:7:0:9d0841ad-adb6-44b4-a299-3cef745cc206, @call-id=12, @rc-code=0, @last-run=1478254104, @last-rc-change=1478254104, @exec-time=175, @migrate_sour Nov 04 11:08:24 [1929] kvm01.test-cluster1 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=kvm01.test-cluster1/crmd/46, version=0.23.22) Nov 04 11:08:24 [1937] kvm01.test-cluster1 crmd: info: match_graph_event: Action kvm_samba2_migrate_from_0 (13) confirmed on kvm01.test-cluster1 (rc=0) Nov 04 11:08:24 [1937] kvm01.test-cluster1 crmd: notice: te_rsc_command: Initiating action 9: stop kvm_samba2_stop_0 on kvm02.test-cluster1 Nov 04 11:08:24 [1929] kvm01.test-cluster1 cib: info: cib_perform_op: Diff: --- 0.23.22 2 Nov 04 11:08:24 [1929] kvm01.test-cluster1 cib: info: cib_perform_op: Diff: +++ 0.23.23 (null) Nov 04 11:08:24 [1929] kvm01.test-cluster1 cib: info: cib_perform_op: + /cib: @num_updates=23 Nov 04 11:08:24 [1929] kvm01.test-cluster1 cib: info: cib_perform_op: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='kvm_samba2']/lrm_rsc_op[@id='kvm_samba2_last_0']: @operation_key=kvm_samba2_stop_0, @operation=stop, @transition-key=9:7:0:9d0841ad-adb6-44b4-a299-3cef745cc206, @transition-magic=0:0;9:7:0:9d0841ad-adb6-44b4-a299-3cef745cc206, @call-id=14, @last-run=1478254104, @last-rc-change=1478254104, @exec-time=57 Nov 04 11:08:24 [1929] kvm01.test-cluster1 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=kvm02.test-cluster1/crmd/16, version=0.23.23) Nov 04 11:08:24 [1937] kvm01.test-cluster1 crmd: info: match_graph_event: Action kvm_samba2_stop_0 (9) confirmed on kvm02.test-cluster1 (rc=0) Nov 04 11:08:24 [1937] kvm01.test-cluster1 crmd: info: te_crm_command: Executing crm-event (15): do_shutdown on kvm02.test-cluster1 Nov 04 11:08:24 [1937] kvm01.test-cluster1 crmd: notice: te_rsc_command: Initiating action 11: monitor kvm_samba2_monitor_10000 on kvm01.test-cluster1 (local) Nov 04 11:08:24 [1937] kvm01.test-cluster1 crmd: info: do_lrm_rsc_op: Performing key=11:7:0:9d0841ad-adb6-44b4-a299-3cef745cc206 op=kvm_samba2_monitor_10000 Nov 04 11:08:24 [1937] kvm01.test-cluster1 crmd: info: pcmk_cpg_membership: Node 2 left group crmd (peer=kvm02.test-cluster1, counter=2.0) Nov 04 11:08:24 [1937] kvm01.test-cluster1 crmd: info: crm_update_peer_proc: pcmk_cpg_membership: Node kvm02.test-cluster1[2] - corosync-cpg is now offline Nov 04 11:08:24 [1937] kvm01.test-cluster1 crmd: info: peer_update_callback: Client kvm02.test-cluster1/peer now has status [offline] (DC=true, changed=4000000) Nov 04 11:08:24 [1937] kvm01.test-cluster1 crmd: info: peer_update_callback: Peer kvm02.test-cluster1 left us Nov 04 11:08:24 [1937] kvm01.test-cluster1 crmd: info: erase_status_tag: Deleting xpath: //node_state[@uname='kvm02.test-cluster1']/transient_attributes Nov 04 11:08:24 [1929] kvm01.test-cluster1 cib: info: cib_process_request: Forwarding cib_delete operation for section //node_state[@uname='kvm02.test-cluster1']/transient_attributes to master (origin=local/crmd/47) Nov 04 11:08:24 [1937] kvm01.test-cluster1 crmd: notice: peer_update_callback: do_shutdown of kvm02.test-cluster1 (op 15) is complete Nov 04 11:08:24 [1937] kvm01.test-cluster1 crmd: info: crm_update_peer_join: crmd_peer_down: Node kvm02.test-cluster1[2] - join-2 phase 4 -> 0 Nov 04 11:08:24 [1937] kvm01.test-cluster1 crmd: info: pcmk_cpg_membership: Node 1 still member of group crmd (peer=kvm01.test-cluster1, counter=2.0) Nov 04 11:08:24 [1929] kvm01.test-cluster1 cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/48) Nov 04 11:08:24 [1929] kvm01.test-cluster1 cib: info: cib_perform_op: Diff: --- 0.23.23 2 Nov 04 11:08:24 [1929] kvm01.test-cluster1 cib: info: cib_perform_op: Diff: +++ 0.23.24 2032dcea3b7299e948f6cadaf56b473b Nov 04 11:08:24 [1929] kvm01.test-cluster1 cib: info: cib_perform_op: -- /cib/status/node_state[@id='2']/transient_attributes[@id='2'] Nov 04 11:08:24 [1929] kvm01.test-cluster1 cib: info: cib_perform_op: + /cib: @num_updates=24 Nov 04 11:08:24 [1929] kvm01.test-cluster1 cib: info: cib_process_request: Completed cib_delete operation for section //node_state[@uname='kvm02.test-cluster1']/transient_attributes: OK (rc=0, origin=kvm01.test-cluster1/crmd/47, version=0.23.24) Nov 04 11:08:24 [1929] kvm01.test-cluster1 cib: info: cib_perform_op: Diff: --- 0.23.24 2 Nov 04 11:08:24 [1929] kvm01.test-cluster1 cib: info: cib_perform_op: Diff: +++ 0.23.25 (null) Nov 04 11:08:24 [1929] kvm01.test-cluster1 cib: info: cib_perform_op: + /cib: @num_updates=25 Nov 04 11:08:24 [1929] kvm01.test-cluster1 cib: info: cib_perform_op: + /cib/status/node_state[@id='2']: @crmd=offline, @crm-debug-origin=peer_update_callback, @join=down, @expected=down Nov 04 11:08:24 [1929] kvm01.test-cluster1 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=kvm01.test-cluster1/crmd/48, version=0.23.25) Nov 04 11:08:24 [1935] kvm01.test-cluster1 attrd: info: pcmk_cpg_membership: Node 2 left group attrd (peer=kvm02.test-cluster1, counter=2.0) Nov 04 11:08:24 [1935] kvm01.test-cluster1 attrd: info: crm_update_peer_proc: pcmk_cpg_membership: Node kvm02.test-cluster1[2] - corosync-cpg is now offline Nov 04 11:08:24 [1935] kvm01.test-cluster1 attrd: notice: crm_update_peer_state_iter: crm_update_peer_proc: Node kvm02.test-cluster1[2] - state is now lost (was member) Nov 04 11:08:24 [1935] kvm01.test-cluster1 attrd: notice: attrd_peer_remove: Removing all kvm02.test-cluster1 attributes for attrd_peer_change_cb Nov 04 11:08:24 [1935] kvm01.test-cluster1 attrd: notice: attrd_peer_change_cb: Lost attribute writer kvm02.test-cluster1 Nov 04 11:08:24 [1935] kvm01.test-cluster1 attrd: notice: crm_reap_dead_member: Removing kvm02.test-cluster1/2 from the membership list Nov 04 11:08:24 [1935] kvm01.test-cluster1 attrd: notice: reap_crm_member: Purged 1 peers with id=2 and/or uname=kvm02.test-cluster1 from the membership cache Nov 04 11:08:24 [1935] kvm01.test-cluster1 attrd: info: pcmk_cpg_membership: Node 1 still member of group attrd (peer=kvm01.test-cluster1, counter=2.0) Nov 04 11:08:24 [1933] kvm01.test-cluster1 stonith-ng: info: pcmk_cpg_membership: Node 2 left group stonith-ng (peer=kvm02.test-cluster1, counter=2.0) Nov 04 11:08:24 [1933] kvm01.test-cluster1 stonith-ng: info: crm_update_peer_proc: pcmk_cpg_membership: Node kvm02.test-cluster1[2] - corosync-cpg is now offline Nov 04 11:08:24 [1933] kvm01.test-cluster1 stonith-ng: notice: crm_update_peer_state_iter: crm_update_peer_proc: Node kvm02.test-cluster1[2] - state is now lost (was member) Nov 04 11:08:24 [1933] kvm01.test-cluster1 stonith-ng: notice: crm_reap_dead_member: Removing kvm02.test-cluster1/2 from the membership list Nov 04 11:08:24 [1933] kvm01.test-cluster1 stonith-ng: notice: reap_crm_member: Purged 1 peers with id=2 and/or uname=kvm02.test-cluster1 from the membership cache Nov 04 11:08:24 [1933] kvm01.test-cluster1 stonith-ng: info: pcmk_cpg_membership: Node 1 still member of group stonith-ng (peer=kvm01.test-cluster1, counter=2.0) Nov 04 11:08:24 [1929] kvm01.test-cluster1 cib: info: cib_process_shutdown_req: Shutdown REQ from kvm02.test-cluster1 Nov 04 11:08:24 [1929] kvm01.test-cluster1 cib: info: pcmk_cpg_membership: Node 2 left group cib (peer=kvm02.test-cluster1, counter=2.0) Nov 04 11:08:24 [1929] kvm01.test-cluster1 cib: info: crm_update_peer_proc: pcmk_cpg_membership: Node kvm02.test-cluster1[2] - corosync-cpg is now offline Nov 04 11:08:24 [1929] kvm01.test-cluster1 cib: notice: crm_update_peer_state_iter: crm_update_peer_proc: Node kvm02.test-cluster1[2] - state is now lost (was member) Nov 04 11:08:24 [1929] kvm01.test-cluster1 cib: notice: crm_reap_dead_member: Removing kvm02.test-cluster1/2 from the membership list Nov 04 11:08:24 [1929] kvm01.test-cluster1 cib: notice: reap_crm_member: Purged 1 peers with id=2 and/or uname=kvm02.test-cluster1 from the membership cache Nov 04 11:08:24 [1929] kvm01.test-cluster1 cib: info: pcmk_cpg_membership: Node 1 still member of group cib (peer=kvm01.test-cluster1, counter=2.0) Nov 04 11:08:24 [1908] kvm01.test-cluster1 pacemakerd: info: pcmk_cpg_membership: Node 2 left group pacemakerd (peer=kvm02.test-cluster1, counter=2.0) Nov 04 11:08:24 [1908] kvm01.test-cluster1 pacemakerd: info: crm_update_peer_proc: pcmk_cpg_membership: Node kvm02.test-cluster1[2] - corosync-cpg is now offline Nov 04 11:08:24 [1908] kvm01.test-cluster1 pacemakerd: info: pcmk_cpg_membership: Node 1 still member of group pacemakerd (peer=kvm01.test-cluster1, counter=2.0) Nov 04 11:08:24 [1908] kvm01.test-cluster1 pacemakerd: info: mcp_cpg_deliver: Ignoring process list sent by peer for local node Nov 04 11:08:24 [1937] kvm01.test-cluster1 crmd: info: process_lrm_event: Operation kvm_samba2_monitor_10000: ok (node=kvm01.test-cluster1, call=13, rc=0, cib-update=49, confirmed=false) Nov 04 11:08:24 [1929] kvm01.test-cluster1 cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/49) Nov 04 11:08:24 [1929] kvm01.test-cluster1 cib: info: cib_perform_op: Diff: --- 0.23.25 2 Nov 04 11:08:24 [1929] kvm01.test-cluster1 cib: info: cib_perform_op: Diff: +++ 0.23.26 (null) Nov 04 11:08:24 [1929] kvm01.test-cluster1 cib: info: cib_perform_op: + /cib: @num_updates=26 Nov 04 11:08:24 [1929] kvm01.test-cluster1 cib: info: cib_perform_op: ++ /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='kvm_samba2']: