[ClusterLabs] Trying to prevent instantaneous Failover / Failback at standby reconnect
O'Donovan, Garret
Garret.ODonovan at arris.com
Wed Jul 25 23:46:07 EDT 2018
Thanks for the responses, Andrei and others, and for taking the time to
look at this. Apologies for being slow getting back to you.
We realize that having a stonith device would help avoid split-brain, and
it is something we may look into in the future. In the meantime, we are
looking to eliminate the failover / failback in this particular test case.
We reproduced the same scenario and this time there are logs below from
both nodes.
Thanks again!
Node that was running as Primary:
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: pcmk_cpg_membership: Node 2 left group cib (peer=dvtvm0803, counter=4.0)
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: crm_update_peer_proc: pcmk_cpg_membership: Node dvtvm0803[2] - corosync-cpg is now offline
Jul 25 07:42:46 [1569] DVTVM0802.mse.am.mot.com pacemakerd: info: pcmk_cpg_membership: Node 2 left group pacemakerd (peer=dvtvm0803, counter=4.0)
Jul 25 07:42:46 [1569] DVTVM0802.mse.am.mot.com pacemakerd: info: crm_update_peer_proc: pcmk_cpg_membership: Node dvtvm0803[2] - corosync-cpg is now offline
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: notice: crm_update_peer_state_iter: Node dvtvm0803 state is now lost | nodeid=2 previous=member source=crm_update_peer_proc
Jul 25 07:42:46 [1569] DVTVM0802.mse.am.mot.com pacemakerd: info: pcmk_cpg_membership: Node 1 still member of group pacemakerd (peer=dvtvm0802, counter=4.0)
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: crm_reap_dead_member: Removing node with name dvtvm0803 and id 2 from membership cache
Jul 25 07:42:46 [1569] DVTVM0802.mse.am.mot.com pacemakerd: info: crm_cs_flush: Sent 0 CPG messages (1 remaining, last=10): Try again (6)
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: notice: reap_crm_member: Purged 1 peer with id=2 and/or uname=dvtvm0803 from the membership cache
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: pcmk_cpg_membership: Node 1 still member of group cib (peer=dvtvm0802, counter=4.0)
[1003] DVTVM0802.mse.am.mot.com corosyncnotice [QUORUM] Members[1]: 1
Jul 25 07:42:46 [1569] DVTVM0802.mse.am.mot.com pacemakerd: info: pcmk_quorum_notification: Quorum retained | membership=52 members=1
[1003] DVTVM0802.mse.am.mot.com corosyncnotice [MAIN ] Completed service synchronization, ready to provide service.
Jul 25 07:42:46 [1639] DVTVM0802.mse.am.mot.com stonith-ng: info: pcmk_cpg_membership: Node 2 left group stonith-ng (peer=dvtvm0803, counter=4.0)
Jul 25 07:42:46 [1639] DVTVM0802.mse.am.mot.com stonith-ng: info: crm_update_peer_proc: pcmk_cpg_membership: Node dvtvm0803[2] - corosync-cpg is now offline
Jul 25 07:42:46 [1569] DVTVM0802.mse.am.mot.com pacemakerd: notice: crm_update_peer_state_iter: Node dvtvm0803 state is now lost | nodeid=2 previous=member source=crm_reap_unseen_nodes
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: info: pcmk_quorum_notification: Quorum retained | membership=52 members=1
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: notice: crm_update_peer_state_iter: Node dvtvm0803 state is now lost | nodeid=2 previous=member source=crm_reap_unseen_nodes
Jul 25 07:42:46 [1639] DVTVM0802.mse.am.mot.com stonith-ng: notice: crm_update_peer_state_iter: Node dvtvm0803 state is now lost | nodeid=2 previous=member source=crm_update_peer_proc
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: info: peer_update_callback: Cluster node dvtvm0803 is now lost (was member)
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: warning: reap_dead_nodes: Our DC node (dvtvm0803) left the cluster
Jul 25 07:42:46 [1639] DVTVM0802.mse.am.mot.com stonith-ng: info: crm_reap_dead_member: Removing node with name dvtvm0803 and id 2 from membership cache
Jul 25 07:42:46 [1639] DVTVM0802.mse.am.mot.com stonith-ng: notice: reap_crm_member: Purged 1 peer with id=2 and/or uname=dvtvm0803 from the membership cache
Jul 25 07:42:46 [1639] DVTVM0802.mse.am.mot.com stonith-ng: info: pcmk_cpg_membership: Node 1 still member of group stonith-ng (peer=dvtvm0802, counter=4.0)
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: notice: do_state_transition: State transition S_NOT_DC -> S_ELECTION | input=I_ELECTION cause=C_FSA_INTERNAL origin=reap_dead_nodes
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: info: update_dc: Unset DC. Was dvtvm0803
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: info: pcmk_cpg_membership: Node 2 left group crmd (peer=dvtvm0803, counter=4.0)
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: info: crm_update_peer_proc: pcmk_cpg_membership: Node dvtvm0803[2] - corosync-cpg is now offline
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: info: peer_update_callback: Client dvtvm0803/peer now has status [offline] (DC=<null>, changed=4000000)
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: info: pcmk_cpg_membership: Node 1 still member of group crmd (peer=dvtvm0802, counter=4.0)
Jul 25 07:42:46 [1641] DVTVM0802.mse.am.mot.com attrd: info: pcmk_cpg_membership: Node 2 left group attrd (peer=dvtvm0803, counter=4.0)
Jul 25 07:42:46 [1641] DVTVM0802.mse.am.mot.com attrd: info: crm_update_peer_proc: pcmk_cpg_membership: Node dvtvm0803[2] - corosync-cpg is now offline
Jul 25 07:42:46 [1641] DVTVM0802.mse.am.mot.com attrd: notice: crm_update_peer_state_iter: Node dvtvm0803 state is now lost | nodeid=2 previous=member source=crm_update_peer_proc
Jul 25 07:42:46 [1641] DVTVM0802.mse.am.mot.com attrd: notice: attrd_peer_remove: Removing all dvtvm0803 attributes for peer loss
Jul 25 07:42:46 [1641] DVTVM0802.mse.am.mot.com attrd: notice: attrd_peer_change_cb: Lost attribute writer dvtvm0803
Jul 25 07:42:46 [1641] DVTVM0802.mse.am.mot.com attrd: info: crm_reap_dead_member: Removing node with name dvtvm0803 and id 2 from membership cache
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: info: election_complete: Election election-0 complete
Jul 25 07:42:46 [1641] DVTVM0802.mse.am.mot.com attrd: notice: reap_crm_member: Purged 1 peer with id=2 and/or uname=dvtvm0803 from the membership cache
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: info: election_timeout_popped: Election failed: Declaring ourselves the winner
Jul 25 07:42:46 [1641] DVTVM0802.mse.am.mot.com attrd: info: pcmk_cpg_membership: Node 1 still member of group attrd (peer=dvtvm0802, counter=4.0)
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: info: do_log: Input I_ELECTION_DC received in state S_ELECTION from election_timeout_popped
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: notice: do_state_transition: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_TIMER_POPPED origin=election_timeout_popped
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: info: do_te_control: Registering TE UUID: 49795742-d293-405c-a429-61113ad512c0
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: info: set_graph_functions: Setting custom graph functions
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: info: do_dc_takeover: Taking over DC status for this partition
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_readwrite: We are now in R/W mode
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/61, version=0.20.63)
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section cib to all (origin=local/crmd/62)
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section cib: OK (rc=0, origin=dvtvm0802/crmd/62, version=0.20.63)
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/64)
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section crm_config: OK (rc=0, origin=dvtvm0802/crmd/64, version=0.20.63)
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/66)
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: info: crm_update_peer_join: initialize_join: Node dvtvm0802[1] - join-3 phase confirmed -> none
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: info: join_make_offer: Making join offers based on membership 52
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: info: join_make_offer: join-3: Sending offer to dvtvm0802
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: info: crm_update_peer_join: join_make_offer: Node dvtvm0802[1] - join-3 phase none -> welcomed
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: info: join_make_offer: Not making an offer to dvtvm0803: not active (lost)
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: info: do_dc_join_offer_all: join-3: Waiting on 1 outstanding join acks
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: warning: do_log: Input I_ELECTION_DC received in state S_INTEGRATION from do_election_check
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: info: crm_update_peer_join: initialize_join: Node dvtvm0802[1] - join-4 phase welcomed -> none
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: info: join_make_offer: join-4: Sending offer to dvtvm0802
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: info: crm_update_peer_join: join_make_offer: Node dvtvm0802[1] - join-4 phase none -> welcomed
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: info: join_make_offer: Not making an offer to dvtvm0803: not active (lost)
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: info: do_dc_join_offer_all: join-4: Waiting on 1 outstanding join acks
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: info: update_dc: Set DC to dvtvm0802 (3.0.14)
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section crm_config: OK (rc=0, origin=dvtvm0802/crmd/66, version=0.20.63)
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/68)
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section crm_config: OK (rc=0, origin=dvtvm0802/crmd/68, version=0.20.63)
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: info: crm_update_peer_join: do_dc_join_filter_offer: Node dvtvm0802[1] - join-4 phase welcomed -> integrated
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: info: crmd_join_phase_log: join-4: dvtvm0802=integrated
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: info: crmd_join_phase_log: join-4: dvtvm0803=none
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: info: do_dc_join_finalize: join-4: Syncing our CIB to the rest of the cluster
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: info: crm_update_peer_join: finalize_join_for: Node dvtvm0802[1] - join-4 phase integrated -> finalized
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_replace: Digest matched on replace from dvtvm0802: 0bc3cfc4c7b71bb6b009f97aa31175b8
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_replace: Replaced 0.20.63 with 0.20.63 from dvtvm0802
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_replace operation for section 'all': OK (rc=0, origin=dvtvm0802/crmd/72, version=0.20.63)
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/73)
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: info: action_synced_wait: Managed edgemanager_meta-data_0 process 12189 exited with rc=4
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: error: lrmd_api_get_metadata: Failed to receive meta-data for ocf:vendor:application
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: warning: build_operation_update: Failed to get metadata for application (ocf:vendor:application)
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: info: action_synced_wait: Managed edgemanager_meta-data_0 process 12191 exited with rc=4
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: error: lrmd_api_get_metadata: Failed to receive meta-data for ocf:vendor:application
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: warning: build_operation_update: Failed to get metadata for application (ocf:vendor:application)
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=dvtvm0802/crmd/73, version=0.20.63)
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: info: crm_update_peer_join: do_dc_join_ack: Node dvtvm0802[1] - join-4 phase finalized -> confirmed
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: info: do_dc_join_ack: join-4: Updating node state to member for dvtvm0802
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: info: erase_status_tag: Deleting lrm status entries for dvtvm0802 | xpath=//node_state[@uname='dvtvm0802']/lrm
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_file_backup: Archived previous version as /var/lib/pacemaker/cib/cib-28.raw
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_delete operation for section //node_state[@uname='dvtvm0802']/lrm to all (origin=local/crmd/74)
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.63 2
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.64 (null)
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: -- /cib/status/node_state[@id='1']/lrm[@id='1']
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=64
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_file_write_with_digest: Wrote version 0.20.0 of the CIB to disk (digest: 38d399e886fdf463121cda517374a888)
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_file_write_with_digest: Reading cluster configuration file /var/lib/pacemaker/cib/cib.mC21GJ (digest: /var/lib/pacemaker/cib/cib.n9gaMp)
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_delete operation for section //node_state[@uname='dvtvm0802']/lrm: OK (rc=0, origin=dvtvm0802/crmd/74, version=0.20.64)
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: info: action_synced_wait: Managed edgemanager_meta-data_0 process 12192 exited with rc=4
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: error: lrmd_api_get_metadata: Failed to receive meta-data for ocf:vendor:application
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: warning: build_operation_update: Failed to get metadata for application (ocf:vendor:application)
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: info: action_synced_wait: Managed edgemanager_meta-data_0 process 12193 exited with rc=4
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: error: lrmd_api_get_metadata: Failed to receive meta-data for ocf:vendor:application
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: warning: build_operation_update: Failed to get metadata for application (ocf:vendor:application)
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/75)
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.64 2
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.65 (null)
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=65
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_lrm_query_internal
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ /cib/status/node_state[@id='1']: <lrm id="1"/>
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_resources>
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_resource id="Connected" type="ping" class="ocf" provider="pacemaker">
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_rsc_op id="Connected_last_0" operation_key="Connected_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="10:0:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" transition-magic="0:0;10:0:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" exit-reason="" on_node="dvtvm0802" call-id="15" rc-code="0" op-status="0" interval="0" last-run="1532518326
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_rsc_op id="Connected_monitor_3000" operation_key="Connected_monitor_3000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="11:0:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" transition-magic="0:0;11:0:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" exit-reason="" on_node="dvtvm0802" call-id="16" rc-code="0" op-status="0" interval="3000" last
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ </lrm_resource>
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_resource id="ClusterIP" type="IPaddr2" class="ocf" provider="heartbeat">
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_rsc_op id="ClusterIP_last_0" operation_key="ClusterIP_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="4:1:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" transition-magic="0:0;4:1:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" exit-reason="" on_node="dvtvm0802" call-id="17" rc-code="0" op-status="0" interval="0" last-run="1532518330"
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_rsc_op id="ClusterIP_monitor_1000" operation_key="ClusterIP_monitor_1000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="5:1:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" transition-magic="0:0;5:1:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" exit-reason="" on_node="dvtvm0802" call-id="18" rc-code="0" op-status="0" interval="1000" last-r
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ </lrm_resource>
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_resource id="application" type="application" class="ocf" provider="vendor">
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_rsc_op id="edgemanager_last_0" operation_key="edgemanager_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="14:1:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" transition-magic="0:0;14:1:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" exit-reason="" on_node="dvtvm0802" call-id="19" rc-code="0" op-status="0" interval="0" last-run="153251
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_rsc_op id="edgemanager_monitor_1000" operation_key="edgemanager_monitor_1000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="15:1:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" transition-magic="0:0;15:1:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" exit-reason="" on_node="dvtvm0802" call-id="20" rc-code="0" op-status="0" interval="1000"
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ </lrm_resource>
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ </lrm_resources>
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ </lrm>
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0802/crmd/75, version=0.20.65)
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: info: do_state_transition: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: info: abort_transition_graph: Transition aborted: Peer Cancelled | source=do_te_invoke:143 complete=true
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/78)
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/79)
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section cib to all (origin=local/crmd/80)
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=dvtvm0802/crmd/78, version=0.20.65)
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.65 2
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.66 (null)
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=66
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='2']: @in_ccm=false, @crmd=offline, @join=down
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0802/crmd/79, version=0.20.66)
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.66 2
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.67 (null)
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=67, @dc-uuid=1
Jul 25 07:42:46 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section cib: OK (rc=0, origin=dvtvm0802/crmd/80, version=0.20.67)
Jul 25 07:42:46 [1643] DVTVM0802.mse.am.mot.com pengine: notice: unpack_config: On loss of CCM Quorum: Ignore
Jul 25 07:42:46 [1643] DVTVM0802.mse.am.mot.com pengine: info: determine_online_status: Node dvtvm0802 is online
Jul 25 07:42:46 [1643] DVTVM0802.mse.am.mot.com pengine: info: unpack_node_loop: Node 1 is already processed
Jul 25 07:42:46 [1643] DVTVM0802.mse.am.mot.com pengine: info: unpack_node_loop: Node 1 is already processed
Jul 25 07:42:46 [1643] DVTVM0802.mse.am.mot.com pengine: info: common_print: ClusterIP (ocf::heartbeat:IPaddr2): Started dvtvm0802
Jul 25 07:42:46 [1643] DVTVM0802.mse.am.mot.com pengine: info: clone_print: Clone Set: Connected-clone [Connected]
Jul 25 07:42:46 [1643] DVTVM0802.mse.am.mot.com pengine: info: short_print: Started: [ dvtvm0802 ]
Jul 25 07:42:46 [1643] DVTVM0802.mse.am.mot.com pengine: info: short_print: Stopped: [ dvtvm0803 ]
Jul 25 07:42:46 [1643] DVTVM0802.mse.am.mot.com pengine: info: common_print: application (ocf::vendor:application): Started dvtvm0802
Jul 25 07:42:46 [1643] DVTVM0802.mse.am.mot.com pengine: info: native_color: Resource Connected:1 cannot run anywhere
Jul 25 07:42:46 [1643] DVTVM0802.mse.am.mot.com pengine: info: LogActions: Leave ClusterIP (Started dvtvm0802)
Jul 25 07:42:46 [1643] DVTVM0802.mse.am.mot.com pengine: info: LogActions: Leave Connected:0 (Started dvtvm0802)
Jul 25 07:42:46 [1643] DVTVM0802.mse.am.mot.com pengine: info: LogActions: Leave Connected:1 (Stopped)
Jul 25 07:42:46 [1643] DVTVM0802.mse.am.mot.com pengine: info: LogActions: Leave application (Started dvtvm0802)
Jul 25 07:42:46 [1643] DVTVM0802.mse.am.mot.com pengine: notice: process_pe_message: Calculated transition 1, saving inputs in /var/lib/pacemaker/pengine/pe-input-5.bz2
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: info: do_te_invoke: Processing graph 1 (ref=pe_calc-dc-1532518966-33) derived from /var/lib/pacemaker/pengine/pe-input-5.bz2
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: notice: run_graph: Transition 1 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-5.bz2): Complete
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: info: do_log: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd
Jul 25 07:42:46 [1645] DVTVM0802.mse.am.mot.com crmd: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd
Jul 25 07:42:46 [1569] DVTVM0802.mse.am.mot.com pacemakerd: info: mcp_cpg_deliver: Ignoring process list sent by peer for local node
Jul 25 07:42:49 [1641] DVTVM0802.mse.am.mot.com attrd: info: attrd_client_update: Starting an election to determine the writer
Jul 25 07:42:49 [1641] DVTVM0802.mse.am.mot.com attrd: info: election_complete: Election election-attrd complete
Jul 25 07:42:49 [1641] DVTVM0802.mse.am.mot.com attrd: info: write_attribute: Processed 1 private change for #attrd-protocol, id=<n/a>, set=(null)
Jul 25 07:42:49 [1641] DVTVM0802.mse.am.mot.com attrd: info: write_attribute: Sent update 5 with 1 changes for pingd, id=<n/a>, set=(null)
Jul 25 07:42:49 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/attrd/5)
Jul 25 07:42:49 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0802/attrd/5, version=0.20.67)
Jul 25 07:42:49 [1641] DVTVM0802.mse.am.mot.com attrd: info: attrd_cib_callback: Update 5 for pingd: OK (0)
Jul 25 07:42:49 [1641] DVTVM0802.mse.am.mot.com attrd: info: attrd_cib_callback: Update 5 for pingd[dvtvm0802]=2000: OK (0)
Jul 25 07:42:54 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_ping: Reporting our current digest to dvtvm0802: d4226f5b1384916e105bdeab21320802 for 0.20.67 (0x560c396fc420 0)
[1003] DVTVM0802.mse.am.mot.com corosyncnotice [TOTEM ] A new membership (10.144.102.32:56) was formed. Members joined: 2
Jul 25 07:43:09 [1639] DVTVM0802.mse.am.mot.com stonith-ng: info: pcmk_cpg_membership: Node 2 joined group stonith-ng (counter=5.0)
Jul 25 07:43:09 [1645] DVTVM0802.mse.am.mot.com crmd: info: pcmk_cpg_membership: Node 2 joined group crmd (counter=5.0)
Jul 25 07:43:09 [1641] DVTVM0802.mse.am.mot.com attrd: info: pcmk_cpg_membership: Node 2 joined group attrd (counter=5.0)
Jul 25 07:43:09 [1639] DVTVM0802.mse.am.mot.com stonith-ng: info: pcmk_cpg_membership: Node 1 still member of group stonith-ng (peer=dvtvm0802, counter=5.0)
Jul 25 07:43:09 [1645] DVTVM0802.mse.am.mot.com crmd: info: pcmk_cpg_membership: Node 1 still member of group crmd (peer=dvtvm0802, counter=5.0)
Jul 25 07:43:09 [1641] DVTVM0802.mse.am.mot.com attrd: info: pcmk_cpg_membership: Node 1 still member of group attrd (peer=dvtvm0802, counter=5.0)
Jul 25 07:43:09 [1645] DVTVM0802.mse.am.mot.com crmd: info: pcmk_cpg_membership: Node 2 still member of group crmd (peer=dvtvm0803, counter=5.1)
Jul 25 07:43:09 [1645] DVTVM0802.mse.am.mot.com crmd: info: crm_update_peer_proc: pcmk_cpg_membership: Node dvtvm0803[2] - corosync-cpg is now online
Jul 25 07:43:09 [1645] DVTVM0802.mse.am.mot.com crmd: info: peer_update_callback: Client dvtvm0803/peer now has status [online] (DC=true, changed=4000000)
Jul 25 07:43:09 [1569] DVTVM0802.mse.am.mot.com pacemakerd: info: pcmk_cpg_membership: Node 2 joined group pacemakerd (counter=5.0)
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: pcmk_cpg_membership: Node 2 joined group cib (counter=5.0)
Jul 25 07:43:09 [1569] DVTVM0802.mse.am.mot.com pacemakerd: info: pcmk_cpg_membership: Node 1 still member of group pacemakerd (peer=dvtvm0802, counter=5.0)
Jul 25 07:43:09 [1645] DVTVM0802.mse.am.mot.com crmd: error: pcmk_cpg_membership: Node dvtvm0803[2] appears to be online even though we think it is dead
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: pcmk_cpg_membership: Node 1 still member of group cib (peer=dvtvm0802, counter=5.0)
Jul 25 07:43:09 [1569] DVTVM0802.mse.am.mot.com pacemakerd: info: pcmk_cpg_membership: Node 2 still member of group pacemakerd (peer=dvtvm0803, counter=5.1)
Jul 25 07:43:09 [1569] DVTVM0802.mse.am.mot.com pacemakerd: info: crm_update_peer_proc: pcmk_cpg_membership: Node dvtvm0803[2] - corosync-cpg is now online
Jul 25 07:43:09 [1645] DVTVM0802.mse.am.mot.com crmd: notice: crm_update_peer_state_iter: Node dvtvm0803 state is now member | nodeid=2 previous=lost source=pcmk_cpg_membership
Jul 25 07:43:09 [1645] DVTVM0802.mse.am.mot.com crmd: info: peer_update_callback: Cluster node dvtvm0803 is now member (was lost)
Jul 25 07:43:09 [1569] DVTVM0802.mse.am.mot.com pacemakerd: error: pcmk_cpg_membership: Node dvtvm0803[2] appears to be online even though we think it is dead
Jul 25 07:43:09 [1569] DVTVM0802.mse.am.mot.com pacemakerd: notice: crm_update_peer_state_iter: Node dvtvm0803 state is now member | nodeid=2 previous=lost source=pcmk_cpg_membership
Jul 25 07:43:09 [1569] DVTVM0802.mse.am.mot.com pacemakerd: info: crm_cs_flush: Sent 0 CPG messages (1 remaining, last=11): Try again (6)
[1003] DVTVM0802.mse.am.mot.com corosyncnotice [QUORUM] Members[2]: 1 2
Jul 25 07:43:09 [1569] DVTVM0802.mse.am.mot.com pacemakerd: info: pcmk_quorum_notification: Quorum retained | membership=56 members=2
[1003] DVTVM0802.mse.am.mot.com corosyncnotice [MAIN ] Completed service synchronization, ready to provide service.
Jul 25 07:43:09 [1641] DVTVM0802.mse.am.mot.com attrd: info: crm_get_peer: Created entry c0e26d80-254b-4ef5-92d6-feb3e4d79467/0x561724cc6f00 for node dvtvm0803/2 (2 total)
Jul 25 07:43:09 [1641] DVTVM0802.mse.am.mot.com attrd: info: crm_get_peer: Node 2 is now known as dvtvm0803
Jul 25 07:43:09 [1641] DVTVM0802.mse.am.mot.com attrd: info: crm_get_peer: Node 2 has uuid 2
Jul 25 07:43:09 [1641] DVTVM0802.mse.am.mot.com attrd: info: pcmk_cpg_membership: Node 2 still member of group attrd (peer=dvtvm0803, counter=5.1)
Jul 25 07:43:09 [1641] DVTVM0802.mse.am.mot.com attrd: info: crm_update_peer_proc: pcmk_cpg_membership: Node dvtvm0803[2] - corosync-cpg is now online
Jul 25 07:43:09 [1641] DVTVM0802.mse.am.mot.com attrd: notice: crm_update_peer_state_iter: Node dvtvm0803 state is now member | nodeid=2 previous=unknown source=crm_update_peer_proc
Jul 25 07:43:09 [1645] DVTVM0802.mse.am.mot.com crmd: info: pcmk_quorum_notification: Quorum retained | membership=56 members=2
Jul 25 07:43:09 [1639] DVTVM0802.mse.am.mot.com stonith-ng: info: crm_get_peer: Created entry 794822b3-61e5-4cda-85c6-6c42c3a14471/0x55f69ac31e30 for node dvtvm0803/2 (2 total)
Jul 25 07:43:09 [1639] DVTVM0802.mse.am.mot.com stonith-ng: info: crm_get_peer: Node 2 is now known as dvtvm0803
Jul 25 07:43:09 [1639] DVTVM0802.mse.am.mot.com stonith-ng: info: crm_get_peer: Node 2 has uuid 2
Jul 25 07:43:09 [1639] DVTVM0802.mse.am.mot.com stonith-ng: info: pcmk_cpg_membership: Node 2 still member of group stonith-ng (peer=dvtvm0803, counter=5.1)
Jul 25 07:43:09 [1639] DVTVM0802.mse.am.mot.com stonith-ng: info: crm_update_peer_proc: pcmk_cpg_membership: Node dvtvm0803[2] - corosync-cpg is now online
Jul 25 07:43:09 [1639] DVTVM0802.mse.am.mot.com stonith-ng: notice: crm_update_peer_state_iter: Node dvtvm0803 state is now member | nodeid=2 previous=unknown source=crm_update_peer_proc
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: crm_get_peer: Created entry 3dc6fe56-2790-4a6f-a17b-88696608ddc3/0x560c39bc9380 for node dvtvm0803/2 (2 total)
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: crm_get_peer: Node 2 is now known as dvtvm0803
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: crm_get_peer: Node 2 has uuid 2
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: pcmk_cpg_membership: Node 2 still member of group cib (peer=dvtvm0803, counter=5.1)
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: crm_update_peer_proc: pcmk_cpg_membership: Node dvtvm0803[2] - corosync-cpg is now online
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: notice: crm_update_peer_state_iter: Node dvtvm0803 state is now member | nodeid=2 previous=unknown source=crm_update_peer_proc
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/82)
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/83)
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/86)
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/87)
Jul 25 07:43:09 [1645] DVTVM0802.mse.am.mot.com crmd: info: register_fsa_error_adv: Resetting the current action list
Jul 25 07:43:09 [1645] DVTVM0802.mse.am.mot.com crmd: warning: crmd_ha_msg_filter: Another DC detected: dvtvm0803 (op=noop)
Jul 25 07:43:09 [1645] DVTVM0802.mse.am.mot.com crmd: notice: do_state_transition: State transition S_IDLE -> S_ELECTION | input=I_ELECTION cause=C_FSA_INTERNAL origin=crmd_ha_msg_filter
Jul 25 07:43:09 [1645] DVTVM0802.mse.am.mot.com crmd: info: update_dc: Unset DC. Was dvtvm0802
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.67 2
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.68 (null)
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=68
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='2']: @crmd=online, @crm-debug-origin=peer_update_callback
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0802/crmd/82, version=0.20.68)
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0802/crmd/83, version=0.20.68)
Jul 25 07:43:09 [1641] DVTVM0802.mse.am.mot.com attrd: notice: attrd_peer_message: Detected another attribute writer: dvtvm0803
Jul 25 07:43:09 [1641] DVTVM0802.mse.am.mot.com attrd: info: attrd_peer_message: Processing sync-response from dvtvm0803
Jul 25 07:43:09 [1641] DVTVM0802.mse.am.mot.com attrd: info: attrd_peer_update: Setting #attrd-protocol[dvtvm0803]: (null) -> 2 from dvtvm0803
Jul 25 07:43:09 [1641] DVTVM0802.mse.am.mot.com attrd: info: attrd_peer_update: Setting pingd[dvtvm0803]: (null) -> 0 from dvtvm0803
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=dvtvm0802/crmd/86, version=0.20.68)
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.68 2
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.69 (null)
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=69
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']: @crm-debug-origin=post_cache_update
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='2']: @in_ccm=true, @crm-debug-origin=post_cache_update
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0802/crmd/87, version=0.20.69)
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.69 2
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.70 (null)
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=70
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']: @crm-debug-origin=peer_update_callback
Jul 25 07:43:09 [1645] DVTVM0802.mse.am.mot.com crmd: info: election_count_vote: Election 6 (owner: 2) lost: vote from dvtvm0803 (Uptime)
Jul 25 07:43:09 [1645] DVTVM0802.mse.am.mot.com crmd: notice: do_state_transition: State transition S_ELECTION -> S_RELEASE_DC | input=I_RELEASE_DC cause=C_FSA_INTERNAL origin=do_election_count_vote
Jul 25 07:43:09 [1645] DVTVM0802.mse.am.mot.com crmd: info: do_dc_release: DC role released
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0803/crmd/129, version=0.20.70)
Jul 25 07:43:09 [1645] DVTVM0802.mse.am.mot.com crmd: info: pe_ipc_destroy: Connection to the Policy Engine released
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0803/crmd/130, version=0.20.70)
Jul 25 07:43:09 [1645] DVTVM0802.mse.am.mot.com crmd: info: do_te_control: Transitioner is now inactive
Jul 25 07:43:09 [1645] DVTVM0802.mse.am.mot.com crmd: info: do_log: Input I_RELEASE_SUCCESS received in state S_RELEASE_DC from do_dc_release
Jul 25 07:43:09 [1645] DVTVM0802.mse.am.mot.com crmd: info: do_state_transition: State transition S_RELEASE_DC -> S_PENDING | input=I_RELEASE_SUCCESS cause=C_FSA_INTERNAL origin=do_dc_release
Jul 25 07:43:09 [1645] DVTVM0802.mse.am.mot.com crmd: info: election_count_vote: Election 7 (owner: 2) lost: vote from dvtvm0803 (Uptime)
Jul 25 07:43:09 [1645] DVTVM0802.mse.am.mot.com crmd: info: do_log: Input I_PENDING received in state S_PENDING from do_election_count_vote
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=dvtvm0803/crmd/133, version=0.20.70)
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.70 2
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.71 (null)
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=71
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']: @crm-debug-origin=post_cache_update
Jul 25 07:43:09 [1641] DVTVM0802.mse.am.mot.com attrd: info: election_count_vote: Election 4 (owner: 2) lost: vote from dvtvm0803 (Uptime)
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0803/crmd/134, version=0.20.71)
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_readwrite: We are now in R/O mode
Jul 25 07:43:09 [1641] DVTVM0802.mse.am.mot.com attrd: info: election_count_vote: Election 5 (owner: 2) lost: vote from dvtvm0803 (Uptime)
Jul 25 07:43:09 [1641] DVTVM0802.mse.am.mot.com attrd: info: attrd_peer_message: Processing sync-response from dvtvm0803
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section cib: OK (rc=0, origin=dvtvm0803/crmd/136, version=0.20.71)
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section crm_config: OK (rc=0, origin=dvtvm0803/crmd/138, version=0.20.71)
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section crm_config: OK (rc=0, origin=dvtvm0803/crmd/140, version=0.20.71)
Jul 25 07:43:09 [1645] DVTVM0802.mse.am.mot.com crmd: info: update_dc: Set DC to dvtvm0803 (3.0.14)
Jul 25 07:43:09 [1645] DVTVM0802.mse.am.mot.com crmd: info: election_count_vote: Election 8 (owner: 2) lost: vote from dvtvm0803 (Uptime)
Jul 25 07:43:09 [1645] DVTVM0802.mse.am.mot.com crmd: info: update_dc: Unset DC. Was dvtvm0803
Jul 25 07:43:09 [1645] DVTVM0802.mse.am.mot.com crmd: info: do_log: Input I_PENDING received in state S_PENDING from do_election_count_vote
Jul 25 07:43:09 [1645] DVTVM0802.mse.am.mot.com crmd: info: update_dc: Set DC to dvtvm0803 (3.0.14)
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section crm_config: OK (rc=0, origin=dvtvm0803/crmd/142, version=0.20.71)
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_replace: Digest matched on replace from dvtvm0803: e5254bf4e58c636caa44908bdc7e0a94
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_replace: Replaced 0.20.71 with 0.20.83 from dvtvm0803
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.71 2
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.83 (null)
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=83, @dc-uuid=2
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='2']: @join=member
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-pingd']: @value=0
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='ClusterIP']/lrm_rsc_op[@id='ClusterIP_last_0']: @crm-debug-origin=do_update_resource, @transition-key=5:7:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @transition-magic=0:0;5:7:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @call-id=32, @last-run=1532518973, @last-rc-change=1532518973, @exec-time=74
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='ClusterIP']: <lrm_rsc_op id="ClusterIP_monitor_1000" operation_key="ClusterIP_monitor_1000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="4:6:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" transition-magic="0:0;4:6:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" exit-reason="" on_node="dvtvm0803" call-id=
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='edgemanager']/lrm_rsc_op[@id='edgemanager_last_0']: @crm-debug-origin=do_update_resource, @transition-key=12:7:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @transition-magic=0:0;12:7:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @call-id=30, @last-run=1532518973, @last-rc-change=1532518973, @exec-time=34
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='edgemanager']: <lrm_rsc_op id="edgemanager_monitor_1000" operation_key="edgemanager_monitor_1000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="12:6:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" transition-magic="0:0;12:6:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" exit-reason="" on_node="dvtvm0803"
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_replace_notify: Replaced: 0.20.71 -> 0.20.83 from dvtvm0803
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_replace operation for section 'all': OK (rc=0, origin=dvtvm0803/crmd/146, version=0.20.83)
Jul 25 07:43:09 [1645] DVTVM0802.mse.am.mot.com crmd: info: action_synced_wait: Managed edgemanager_meta-data_0 process 13694 exited with rc=4
Jul 25 07:43:09 [1645] DVTVM0802.mse.am.mot.com crmd: error: lrmd_api_get_metadata: Failed to receive meta-data for ocf:vendor:application
Jul 25 07:43:09 [1645] DVTVM0802.mse.am.mot.com crmd: warning: build_operation_update: Failed to get metadata for application (ocf:vendor:application)
Jul 25 07:43:09 [1641] DVTVM0802.mse.am.mot.com attrd: notice: attrd_cib_replaced_cb: Updating all attributes after cib_refresh_notify event
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=dvtvm0803/crmd/147, version=0.20.83)
Jul 25 07:43:09 [1645] DVTVM0802.mse.am.mot.com crmd: info: action_synced_wait: Managed edgemanager_meta-data_0 process 13696 exited with rc=4
Jul 25 07:43:09 [1645] DVTVM0802.mse.am.mot.com crmd: error: lrmd_api_get_metadata: Failed to receive meta-data for ocf:vendor:application
Jul 25 07:43:09 [1645] DVTVM0802.mse.am.mot.com crmd: warning: build_operation_update: Failed to get metadata for application (ocf:vendor:application)
Jul 25 07:43:09 [1645] DVTVM0802.mse.am.mot.com crmd: info: do_log: Input I_NOT_DC received in state S_PENDING from do_cl_join_finalize_respond
Jul 25 07:43:09 [1645] DVTVM0802.mse.am.mot.com crmd: notice: do_state_transition: State transition S_PENDING -> S_NOT_DC | input=I_NOT_DC cause=C_HA_MESSAGE origin=do_cl_join_finalize_respond
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=dvtvm0803/crmd/148, version=0.20.83)
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.83 2
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.84 (null)
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: -- /cib/status/node_state[@id='2']/lrm[@id='2']
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=84
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_file_backup: Archived previous version as /var/lib/pacemaker/cib/cib-29.raw
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_delete operation for section //node_state[@uname='dvtvm0803']/lrm: OK (rc=0, origin=dvtvm0803/crmd/149, version=0.20.84)
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.84 2
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.85 (null)
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=85
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_lrm_query_internal
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ /cib/status/node_state[@id='2']: <lrm id="2"/>
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_resources>
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_resource id="Connected" type="ping" class="ocf" provider="pacemaker">
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_rsc_op id="Connected_last_0" operation_key="Connected_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="8:0:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" transition-magic="0:0;8:0:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" exit-reason="" on_node="dvtvm0803" call-id="15" rc-code="0" op-status="0" interval="0" last-run="1532518326"
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_rsc_op id="Connected_monitor_3000" operation_key="Connected_monitor_3000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="9:0:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" transition-magic="0:0;9:0:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" exit-reason="" on_node="dvtvm0803" call-id="16" rc-code="0" op-status="0" interval="3000" last-r
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ </lrm_resource>
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_resource id="ClusterIP" type="IPaddr2" class="ocf" provider="heartbeat">
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_rsc_op id="ClusterIP_last_0" operation_key="ClusterIP_stop_0" operation="stop" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="5:7:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" transition-magic="0:0;5:7:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" exit-reason="" on_node="dvtvm0803" call-id="32" rc-code="0" op-status="0" interval="0" last-run="1532518973" la
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ </lrm_resource>
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_resource id="application" type="application" class="ocf" provider="vendor">
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_rsc_op id="edgemanager_last_0" operation_key="edgemanager_stop_0" operation="stop" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="12:7:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" transition-magic="0:0;12:7:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" exit-reason="" on_node="dvtvm0803" call-id="30" rc-code="0" op-status="0" interval="0" last-run="15325189
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ </lrm_resource>
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ </lrm_resources>
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ </lrm>
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_file_write_with_digest: Wrote version 0.20.0 of the CIB to disk (digest: d281e07c969310cc6525c6f46ec6a307)
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0803/crmd/150, version=0.20.85)
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_file_write_with_digest: Reading cluster configuration file /var/lib/pacemaker/cib/cib.rzefQ3 (digest: /var/lib/pacemaker/cib/cib.Pm42VJ)
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.85 2
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.86 (null)
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: -- /cib/status/node_state[@id='1']/lrm[@id='1']
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=86
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_delete operation for section //node_state[@uname='dvtvm0802']/lrm: OK (rc=0, origin=dvtvm0803/crmd/151, version=0.20.86)
Jul 25 07:43:09 [1641] DVTVM0802.mse.am.mot.com attrd: info: attrd_peer_update: Setting pingd[dvtvm0803]: 0 -> 1000 from dvtvm0803
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.86 2
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.87 (null)
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=87
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_lrm_query_internal
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ /cib/status/node_state[@id='1']: <lrm id="1"/>
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_resources>
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_resource id="Connected" type="ping" class="ocf" provider="pacemaker">
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_rsc_op id="Connected_last_0" operation_key="Connected_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="10:0:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" transition-magic="0:0;10:0:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" exit-reason="" on_node="dvtvm0802" call-id="15" rc-code="0" op-status="0" interval="0" last-run="1532518326
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_rsc_op id="Connected_monitor_3000" operation_key="Connected_monitor_3000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="11:0:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" transition-magic="0:0;11:0:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" exit-reason="" on_node="dvtvm0802" call-id="16" rc-code="0" op-status="0" interval="3000" last
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ </lrm_resource>
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_resource id="ClusterIP" type="IPaddr2" class="ocf" provider="heartbeat">
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_rsc_op id="ClusterIP_last_0" operation_key="ClusterIP_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="4:1:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" transition-magic="0:0;4:1:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" exit-reason="" on_node="dvtvm0802" call-id="17" rc-code="0" op-status="0" interval="0" last-run="1532518330"
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_rsc_op id="ClusterIP_monitor_1000" operation_key="ClusterIP_monitor_1000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="5:1:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" transition-magic="0:0;5:1:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" exit-reason="" on_node="dvtvm0802" call-id="18" rc-code="0" op-status="0" interval="1000" last-r
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ </lrm_resource>
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_resource id="application" type="application" class="ocf" provider="vendor">
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_rsc_op id="edgemanager_last_0" operation_key="edgemanager_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="14:1:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" transition-magic="0:0;14:1:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" exit-reason="" on_node="dvtvm0802" call-id="19" rc-code="0" op-status="0" interval="0" last-run="153251
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_rsc_op id="edgemanager_monitor_1000" operation_key="edgemanager_monitor_1000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="15:1:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" transition-magic="0:0;15:1:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" exit-reason="" on_node="dvtvm0802" call-id="20" rc-code="0" op-status="0" interval="1000"
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ </lrm_resource>
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ </lrm_resources>
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ </lrm>
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0803/crmd/152, version=0.20.87)
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=dvtvm0803/crmd/155, version=0.20.87)
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.87 2
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.88 (null)
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=88
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0803/crmd/156, version=0.20.88)
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section cib: OK (rc=0, origin=dvtvm0803/crmd/157, version=0.20.88)
Jul 25 07:43:09 [1640] DVTVM0802.mse.am.mot.com lrmd: info: cancel_recurring_action: Cancelling ocf operation edgemanager_monitor_1000
Jul 25 07:43:09 [1645] DVTVM0802.mse.am.mot.com crmd: info: do_lrm_rsc_op: Performing key=15:8:0:1cec18ac-2aae-4c5c-8513-d6851bc10191 op=edgemanager_stop_0
Jul 25 07:43:09 [1640] DVTVM0802.mse.am.mot.com lrmd: info: log_execute: executing - rsc:application action:stop call_id:22
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/92)
Jul 25 07:43:09 [1645] DVTVM0802.mse.am.mot.com crmd: info: process_lrm_event: Result of monitor operation for application on dvtvm0802: Cancelled | call=20 key=edgemanager_monitor_1000 confirmed=true
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.88 2
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.89 (null)
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=89
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_update_resource
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='edgemanager']/lrm_rsc_op[@id='edgemanager_last_0']: @operation_key=edgemanager_stop_0, @operation=stop, @crm-debug-origin=do_update_resource, @transition-key=15:8:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @transition-magic=-1:193;15:8:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @call-id=-1, @rc-code=193, @op-status=-1, @last-run=1532518989, @last-r
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0802/crmd/92, version=0.20.89)
Jul 25 07:43:09 application(application)[13697]: INFO: Stopping APP - setting to Standby
Jul 25 07:43:09 [1640] DVTVM0802.mse.am.mot.com lrmd: notice: operation_finished: edgemanager_stop_0:13697:stderr [ /usr/local/vendor/bin/PrimaryMode.sh: line 27: $Filename: ambiguous redirect ]
Jul 25 07:43:09 [1640] DVTVM0802.mse.am.mot.com lrmd: info: log_finished: finished - rsc:application action:stop call_id:22 pid:13697 exit-code:0 exec-time:34ms queue-time:0ms
Jul 25 07:43:09 [1645] DVTVM0802.mse.am.mot.com crmd: notice: process_lrm_event: Result of stop operation for application on dvtvm0802: 0 (ok) | call=22 key=edgemanager_stop_0 confirmed=true cib-update=93
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/93)
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.89 2
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.90 (null)
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=90
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='edgemanager']/lrm_rsc_op[@id='edgemanager_last_0']: @transition-magic=0:0;15:8:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @call-id=22, @rc-code=0, @op-status=0, @exec-time=34
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0802/crmd/93, version=0.20.90)
Jul 25 07:43:09 [1640] DVTVM0802.mse.am.mot.com lrmd: info: cancel_recurring_action: Cancelling ocf operation ClusterIP_monitor_1000
Jul 25 07:43:09 [1645] DVTVM0802.mse.am.mot.com crmd: info: do_lrm_rsc_op: Performing key=6:8:0:1cec18ac-2aae-4c5c-8513-d6851bc10191 op=ClusterIP_stop_0
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/94)
Jul 25 07:43:09 [1640] DVTVM0802.mse.am.mot.com lrmd: info: log_execute: executing - rsc:ClusterIP action:stop call_id:24
Jul 25 07:43:09 [1645] DVTVM0802.mse.am.mot.com crmd: info: process_lrm_event: Result of monitor operation for ClusterIP on dvtvm0802: Cancelled | call=18 key=ClusterIP_monitor_1000 confirmed=true
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.90 2
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.91 (null)
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=91
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='ClusterIP']/lrm_rsc_op[@id='ClusterIP_last_0']: @operation_key=ClusterIP_stop_0, @operation=stop, @crm-debug-origin=do_update_resource, @transition-key=6:8:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @transition-magic=-1:193;6:8:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @call-id=-1, @rc-code=193, @op-status=-1, @last-run=1532518989, @last-rc-change
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0802/crmd/94, version=0.20.91)
Jul 25 07:43:09 IPaddr2(ClusterIP)[13712]: INFO: IP status = ok, IP_CIP=
Jul 25 07:43:09 [1640] DVTVM0802.mse.am.mot.com lrmd: info: log_finished: finished - rsc:ClusterIP action:stop call_id:24 pid:13712 exit-code:0 exec-time:76ms queue-time:0ms
Jul 25 07:43:09 [1645] DVTVM0802.mse.am.mot.com crmd: notice: process_lrm_event: Result of stop operation for ClusterIP on dvtvm0802: 0 (ok) | call=24 key=ClusterIP_stop_0 confirmed=true cib-update=95
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/95)
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.91 2
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.92 (null)
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=92
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='ClusterIP']/lrm_rsc_op[@id='ClusterIP_last_0']: @transition-magic=0:0;6:8:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @call-id=24, @rc-code=0, @op-status=0, @exec-time=76
Jul 25 07:43:09 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0802/crmd/95, version=0.20.92)
Jul 25 07:43:09 [1569] DVTVM0802.mse.am.mot.com pacemakerd: info: mcp_cpg_deliver: Ignoring process list sent by peer for local node
Jul 25 07:43:10 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.92 2
Jul 25 07:43:10 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.93 (null)
Jul 25 07:43:10 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=93
Jul 25 07:43:10 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ /cib/status/node_state[@id='1']: <transient_attributes id="1"/>
Jul 25 07:43:10 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ <instance_attributes id="status-1">
Jul 25 07:43:10 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ <nvpair id="status-1-pingd" name="pingd" value="2000"/>
Jul 25 07:43:10 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ </instance_attributes>
Jul 25 07:43:10 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: ++ </transient_attributes>
Jul 25 07:43:10 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-pingd']: @value=1000
Jul 25 07:43:10 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0803/attrd/9, version=0.20.93)
Jul 25 07:43:10 [1645] DVTVM0802.mse.am.mot.com crmd: info: do_lrm_rsc_op: Performing key=4:9:0:1cec18ac-2aae-4c5c-8513-d6851bc10191 op=ClusterIP_start_0
Jul 25 07:43:10 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/96)
Jul 25 07:43:10 [1640] DVTVM0802.mse.am.mot.com lrmd: info: log_execute: executing - rsc:ClusterIP action:start call_id:25
Jul 25 07:43:10 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.93 2
Jul 25 07:43:10 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.94 (null)
Jul 25 07:43:10 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=94
Jul 25 07:43:10 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='ClusterIP']/lrm_rsc_op[@id='ClusterIP_last_0']: @operation_key=ClusterIP_start_0, @operation=start, @transition-key=4:9:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @transition-magic=-1:193;4:9:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @call-id=-1, @rc-code=193, @op-status=-1, @last-run=1532518990, @last-rc-change=1532518990, @exec-time=0
Jul 25 07:43:10 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0802/crmd/96, version=0.20.94)
Jul 25 07:43:10 IPaddr2(ClusterIP)[13774]: INFO: Adding inet address 10.144.102.230/24 with broadcast address 10.144.102.255 to device eth0
Jul 25 07:43:10 IPaddr2(ClusterIP)[13774]: INFO: Bringing device eth0 up
Jul 25 07:43:10 IPaddr2(ClusterIP)[13774]: INFO: /usr/libexec/heartbeat/send_arp -i 200 -c 5 -p /var/run/resource-agents/send_arp-10.144.102.230 -I eth0 -m auto 10.144.102.230
Jul 25 07:43:14 [1641] DVTVM0802.mse.am.mot.com attrd: info: attrd_peer_update: Setting pingd[dvtvm0803]: 1000 -> 2000 from dvtvm0803
Jul 25 07:43:14 [1640] DVTVM0802.mse.am.mot.com lrmd: info: log_finished: finished - rsc:ClusterIP action:start call_id:25 pid:13774 exit-code:0 exec-time:4105ms queue-time:1ms
Jul 25 07:43:14 [1645] DVTVM0802.mse.am.mot.com crmd: notice: process_lrm_event: Result of start operation for ClusterIP on dvtvm0802: 0 (ok) | call=25 key=ClusterIP_start_0 confirmed=true cib-update=97
Jul 25 07:43:14 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/97)
Jul 25 07:43:14 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.94 2
Jul 25 07:43:14 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.95 (null)
Jul 25 07:43:14 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=95
Jul 25 07:43:14 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='ClusterIP']/lrm_rsc_op[@id='ClusterIP_last_0']: @transition-magic=0:0;4:9:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @call-id=25, @rc-code=0, @op-status=0, @exec-time=4105, @queue-time=1
Jul 25 07:43:14 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0802/crmd/97, version=0.20.95)
Jul 25 07:43:14 [1645] DVTVM0802.mse.am.mot.com crmd: info: do_lrm_rsc_op: Performing key=5:9:0:1cec18ac-2aae-4c5c-8513-d6851bc10191 op=ClusterIP_monitor_1000
Jul 25 07:43:14 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/98)
Jul 25 07:43:14 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.95 2
Jul 25 07:43:14 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.96 (null)
Jul 25 07:43:14 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=96
Jul 25 07:43:14 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='ClusterIP']/lrm_rsc_op[@id='ClusterIP_monitor_1000']: @crm-debug-origin=do_update_resource, @transition-key=5:9:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @transition-magic=-1:193;5:9:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1532518994, @exec-time=0
Jul 25 07:43:14 [1645] DVTVM0802.mse.am.mot.com crmd: info: do_lrm_rsc_op: Performing key=14:9:0:1cec18ac-2aae-4c5c-8513-d6851bc10191 op=edgemanager_start_0
Jul 25 07:43:14 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0802/crmd/98, version=0.20.96)
Jul 25 07:43:14 [1645] DVTVM0802.mse.am.mot.com crmd: info: action_synced_wait: Managed edgemanager_meta-data_0 process 13924 exited with rc=4
Jul 25 07:43:14 [1645] DVTVM0802.mse.am.mot.com crmd: error: lrmd_api_get_metadata: Failed to receive meta-data for ocf:vendor:application
Jul 25 07:43:14 [1645] DVTVM0802.mse.am.mot.com crmd: warning: build_operation_update: Failed to get metadata for application (ocf:vendor:application)
Jul 25 07:43:14 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/99)
Jul 25 07:43:14 [1640] DVTVM0802.mse.am.mot.com lrmd: info: log_execute: executing - rsc:application action:start call_id:27
Jul 25 07:43:14 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.96 2
Jul 25 07:43:14 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.97 (null)
Jul 25 07:43:14 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=97
Jul 25 07:43:14 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='edgemanager']/lrm_rsc_op[@id='edgemanager_last_0']: @operation_key=edgemanager_start_0, @operation=start, @transition-key=14:9:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @transition-magic=-1:193;14:9:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @call-id=-1, @rc-code=193, @op-status=-1, @last-run=1532518994, @last-rc-change=1532518994, @exec-time=0
Jul 25 07:43:14 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0802/crmd/99, version=0.20.97)
Jul 25 07:43:14 application(edgemanager)[13925]: INFO: Starting APP - setting to Primary
Jul 25 07:43:14 [1640] DVTVM0802.mse.am.mot.com lrmd: info: log_finished: finished - rsc:application action:start call_id:27 pid:13925 exit-code:0 exec-time:37ms queue-time:0ms
Jul 25 07:43:14 [1645] DVTVM0802.mse.am.mot.com crmd: info: action_synced_wait: Managed edgemanager_meta-data_0 process 13955 exited with rc=4
Jul 25 07:43:14 [1645] DVTVM0802.mse.am.mot.com crmd: error: lrmd_api_get_metadata: Failed to receive meta-data for ocf:vendor:application
Jul 25 07:43:14 [1645] DVTVM0802.mse.am.mot.com crmd: warning: build_operation_update: Failed to get metadata for application (ocf:vendor:application)
Jul 25 07:43:14 [1645] DVTVM0802.mse.am.mot.com crmd: notice: process_lrm_event: Result of start operation for application on dvtvm0802: 0 (ok) | call=27 key=edgemanager_start_0 confirmed=true cib-update=100
Jul 25 07:43:14 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/100)
Jul 25 07:43:14 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.97 2
Jul 25 07:43:14 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.98 (null)
Jul 25 07:43:14 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=98
Jul 25 07:43:14 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='edgemanager']/lrm_rsc_op[@id='edgemanager_last_0']: @transition-magic=0:0;14:9:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @call-id=27, @rc-code=0, @op-status=0, @exec-time=37
Jul 25 07:43:14 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0802/crmd/100, version=0.20.98)
Jul 25 07:43:14 [1645] DVTVM0802.mse.am.mot.com crmd: info: do_lrm_rsc_op: Performing key=15:9:0:1cec18ac-2aae-4c5c-8513-d6851bc10191 op=edgemanager_monitor_1000
Jul 25 07:43:14 [1645] DVTVM0802.mse.am.mot.com crmd: info: action_synced_wait: Managed edgemanager_meta-data_0 process 13962 exited with rc=4
Jul 25 07:43:14 [1645] DVTVM0802.mse.am.mot.com crmd: error: lrmd_api_get_metadata: Failed to receive meta-data for ocf:vendor:application
Jul 25 07:43:14 [1645] DVTVM0802.mse.am.mot.com crmd: warning: build_operation_update: Failed to get metadata for application (ocf:vendor:application)
Jul 25 07:43:14 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/101)
Jul 25 07:43:14 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.98 2
Jul 25 07:43:14 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.99 (null)
Jul 25 07:43:14 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=99
Jul 25 07:43:14 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='edgemanager']/lrm_rsc_op[@id='edgemanager_monitor_1000']: @crm-debug-origin=do_update_resource, @transition-key=15:9:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @transition-magic=-1:193;15:9:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1532518994, @exec-time=0, @queue-time=0
Jul 25 07:43:14 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0802/crmd/101, version=0.20.99)
Jul 25 07:43:14 [1645] DVTVM0802.mse.am.mot.com crmd: info: action_synced_wait: Managed edgemanager_meta-data_0 process 13995 exited with rc=4
Jul 25 07:43:14 [1645] DVTVM0802.mse.am.mot.com crmd: error: lrmd_api_get_metadata: Failed to receive meta-data for ocf:vendor:application
Jul 25 07:43:14 [1645] DVTVM0802.mse.am.mot.com crmd: warning: build_operation_update: Failed to get metadata for application (ocf:vendor:application)
Jul 25 07:43:14 [1645] DVTVM0802.mse.am.mot.com crmd: info: process_lrm_event: Result of monitor operation for application on dvtvm0802: 0 (ok) | call=28 key=edgemanager_monitor_1000 confirmed=false cib-update=102
Jul 25 07:43:14 [1645] DVTVM0802.mse.am.mot.com crmd: info: process_lrm_event: Result of monitor operation for ClusterIP on dvtvm0802: 0 (ok) | call=26 key=ClusterIP_monitor_1000 confirmed=false cib-update=103
Jul 25 07:43:14 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/102)
Jul 25 07:43:14 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/103)
Jul 25 07:43:14 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.99 2
Jul 25 07:43:14 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.100 (null)
Jul 25 07:43:14 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=100
Jul 25 07:43:14 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='edgemanager']/lrm_rsc_op[@id='edgemanager_monitor_1000']: @transition-magic=0:0;15:9:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @call-id=28, @rc-code=0, @op-status=0, @exec-time=18
Jul 25 07:43:14 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0802/crmd/102, version=0.20.100)
Jul 25 07:43:14 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.100 2
Jul 25 07:43:14 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.101 (null)
Jul 25 07:43:14 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=101
Jul 25 07:43:14 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='ClusterIP']/lrm_rsc_op[@id='ClusterIP_monitor_1000']: @transition-magic=0:0;5:9:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @call-id=26, @rc-code=0, @op-status=0, @exec-time=71
Jul 25 07:43:14 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0802/crmd/103, version=0.20.101)
Jul 25 07:43:15 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.101 2
Jul 25 07:43:15 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.102 (null)
Jul 25 07:43:15 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=102
Jul 25 07:43:15 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-pingd']: @value=2000
Jul 25 07:43:15 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0803/attrd/10, version=0.20.102)
Jul 25 07:43:20 [1638] DVTVM0802.mse.am.mot.com cib: info: cib_process_ping: Reporting our current digest to dvtvm0803: 6aaa74aa220cce54790a731cd758f8e3 for 0.20.102 (0x560c399ebe20 0)
Node that was running as standby:
Jul 25 07:42:46 [1624] DVTVM0803.mse.am.mot.com crmd: info: pcmk_cpg_membership: Node 1 left group crmd (peer=dvtvm0802, counter=3.0)
Jul 25 07:42:46 [1624] DVTVM0803.mse.am.mot.com crmd: info: crm_update_peer_proc: pcmk_cpg_membership: Node dvtvm0802[1] - corosync-cpg is now offline
Jul 25 07:42:46 [1624] DVTVM0803.mse.am.mot.com crmd: info: peer_update_callback: Client dvtvm0802/peer now has status [offline] (DC=true, changed=4000000)
Jul 25 07:42:46 [1624] DVTVM0803.mse.am.mot.com crmd: info: peer_update_callback: Peer dvtvm0802 left us
[994] DVTVM0803.mse.am.mot.com corosyncnotice [QUORUM] Members[1]: 2
Jul 25 07:42:46 [1624] DVTVM0803.mse.am.mot.com crmd: info: erase_status_tag: Deleting transient_attributes status entries for dvtvm0802 | xpath=//node_state[@uname='dvtvm0802']/transient_attributes
[994] DVTVM0803.mse.am.mot.com corosyncnotice [MAIN ] Completed service synchronization, ready to provide service.
Jul 25 07:42:46 [1624] DVTVM0803.mse.am.mot.com crmd: warning: match_down_event: No reason to expect node 1 to be down
Jul 25 07:42:46 [1624] DVTVM0803.mse.am.mot.com crmd: notice: peer_update_callback: Stonith/shutdown of dvtvm0802 not matched
Jul 25 07:42:46 [1624] DVTVM0803.mse.am.mot.com crmd: info: crm_update_peer_join: peer_update_callback: Node dvtvm0802[1] - join-4 phase confirmed -> none
Jul 25 07:42:46 [1624] DVTVM0803.mse.am.mot.com crmd: info: abort_transition_graph: Transition aborted: Node failure | source=peer_update_callback:258 complete=true
Jul 25 07:42:46 [1624] DVTVM0803.mse.am.mot.com crmd: info: pcmk_cpg_membership: Node 2 still member of group crmd (peer=dvtvm0803, counter=3.0)
Jul 25 07:42:46 [1624] DVTVM0803.mse.am.mot.com crmd: notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph
Jul 25 07:42:46 [1624] DVTVM0803.mse.am.mot.com crmd: info: pcmk_quorum_notification: Quorum retained | membership=52 members=1
Jul 25 07:42:46 [1624] DVTVM0803.mse.am.mot.com crmd: notice: crm_update_peer_state_iter: Node dvtvm0802 state is now lost | nodeid=1 previous=member source=crm_reap_unseen_nodes
Jul 25 07:42:46 [1624] DVTVM0803.mse.am.mot.com crmd: info: peer_update_callback: Cluster node dvtvm0802 is now lost (was member)
Jul 25 07:42:46 [1624] DVTVM0803.mse.am.mot.com crmd: warning: match_down_event: No reason to expect node 1 to be down
Jul 25 07:42:46 [1624] DVTVM0803.mse.am.mot.com crmd: notice: peer_update_callback: Stonith/shutdown of dvtvm0802 not matched
Jul 25 07:42:46 [1624] DVTVM0803.mse.am.mot.com crmd: info: abort_transition_graph: Transition aborted: Node failure | source=peer_update_callback:258 complete=true
Jul 25 07:42:46 [1622] DVTVM0803.mse.am.mot.com attrd: info: pcmk_cpg_membership: Node 1 left group attrd (peer=dvtvm0802, counter=3.0)
Jul 25 07:42:46 [1622] DVTVM0803.mse.am.mot.com attrd: info: crm_update_peer_proc: pcmk_cpg_membership: Node dvtvm0802[1] - corosync-cpg is now offline
Jul 25 07:42:46 [1622] DVTVM0803.mse.am.mot.com attrd: notice: crm_update_peer_state_iter: Node dvtvm0802 state is now lost | nodeid=1 previous=member source=crm_update_peer_proc
Jul 25 07:42:46 [1556] DVTVM0803.mse.am.mot.com pacemakerd: info: pcmk_quorum_notification: Quorum retained | membership=52 members=1
Jul 25 07:42:46 [1622] DVTVM0803.mse.am.mot.com attrd: notice: attrd_peer_remove: Removing all dvtvm0802 attributes for peer loss
Jul 25 07:42:46 [1622] DVTVM0803.mse.am.mot.com attrd: info: crm_reap_dead_member: Removing node with name dvtvm0802 and id 1 from membership cache
Jul 25 07:42:46 [1556] DVTVM0803.mse.am.mot.com pacemakerd: notice: crm_update_peer_state_iter: Node dvtvm0802 state is now lost | nodeid=1 previous=member source=crm_reap_unseen_nodes
Jul 25 07:42:46 [1622] DVTVM0803.mse.am.mot.com attrd: notice: reap_crm_member: Purged 1 peer with id=1 and/or uname=dvtvm0802 from the membership cache
Jul 25 07:42:46 [1556] DVTVM0803.mse.am.mot.com pacemakerd: info: pcmk_cpg_membership: Node 1 left group pacemakerd (peer=dvtvm0802, counter=3.0)
Jul 25 07:42:46 [1622] DVTVM0803.mse.am.mot.com attrd: info: pcmk_cpg_membership: Node 2 still member of group attrd (peer=dvtvm0803, counter=3.0)
Jul 25 07:42:46 [1556] DVTVM0803.mse.am.mot.com pacemakerd: info: crm_update_peer_proc: pcmk_cpg_membership: Node dvtvm0802[1] - corosync-cpg is now offline
Jul 25 07:42:46 [1556] DVTVM0803.mse.am.mot.com pacemakerd: info: pcmk_cpg_membership: Node 2 still member of group pacemakerd (peer=dvtvm0803, counter=3.0)
Jul 25 07:42:46 [1618] DVTVM0803.mse.am.mot.com cib: info: pcmk_cpg_membership: Node 1 left group cib (peer=dvtvm0802, counter=3.0)
Jul 25 07:42:46 [1618] DVTVM0803.mse.am.mot.com cib: info: crm_update_peer_proc: pcmk_cpg_membership: Node dvtvm0802[1] - corosync-cpg is now offline
Jul 25 07:42:46 [1618] DVTVM0803.mse.am.mot.com cib: notice: crm_update_peer_state_iter: Node dvtvm0802 state is now lost | nodeid=1 previous=member source=crm_update_peer_proc
Jul 25 07:42:46 [1618] DVTVM0803.mse.am.mot.com cib: info: crm_reap_dead_member: Removing node with name dvtvm0802 and id 1 from membership cache
Jul 25 07:42:46 [1618] DVTVM0803.mse.am.mot.com cib: notice: reap_crm_member: Purged 1 peer with id=1 and/or uname=dvtvm0802 from the membership cache
Jul 25 07:42:46 [1618] DVTVM0803.mse.am.mot.com cib: info: pcmk_cpg_membership: Node 2 still member of group cib (peer=dvtvm0803, counter=3.0)
Jul 25 07:42:46 [1556] DVTVM0803.mse.am.mot.com pacemakerd: info: mcp_cpg_deliver: Ignoring process list sent by peer for local node
Jul 25 07:42:46 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_delete operation for section //node_state[@uname='dvtvm0802']/transient_attributes to all (origin=local/crmd/106)
Jul 25 07:42:46 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/107)
Jul 25 07:42:46 [1619] DVTVM0803.mse.am.mot.com stonith-ng: info: pcmk_cpg_membership: Node 1 left group stonith-ng (peer=dvtvm0802, counter=3.0)
Jul 25 07:42:46 [1619] DVTVM0803.mse.am.mot.com stonith-ng: info: crm_update_peer_proc: pcmk_cpg_membership: Node dvtvm0802[1] - corosync-cpg is now offline
Jul 25 07:42:46 [1619] DVTVM0803.mse.am.mot.com stonith-ng: notice: crm_update_peer_state_iter: Node dvtvm0802 state is now lost | nodeid=1 previous=member source=crm_update_peer_proc
Jul 25 07:42:46 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/109)
Jul 25 07:42:46 [1619] DVTVM0803.mse.am.mot.com stonith-ng: info: crm_reap_dead_member: Removing node with name dvtvm0802 and id 1 from membership cache
Jul 25 07:42:46 [1619] DVTVM0803.mse.am.mot.com stonith-ng: notice: reap_crm_member: Purged 1 peer with id=1 and/or uname=dvtvm0802 from the membership cache
Jul 25 07:42:46 [1619] DVTVM0803.mse.am.mot.com stonith-ng: info: pcmk_cpg_membership: Node 2 still member of group stonith-ng (peer=dvtvm0803, counter=3.0)
Jul 25 07:42:46 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.63 2
Jul 25 07:42:46 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.64 4514de4adeee3556fca57c6e9787b266
Jul 25 07:42:46 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']
Jul 25 07:42:46 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=64
Jul 25 07:42:46 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_delete operation for section //node_state[@uname='dvtvm0802']/transient_attributes: OK (rc=0, origin=dvtvm0803/crmd/106, version=0.20.64)
Jul 25 07:42:46 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.64 2
Jul 25 07:42:46 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.65 (null)
Jul 25 07:42:46 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=65
Jul 25 07:42:46 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']: @crmd=offline, @crm-debug-origin=peer_update_callback
Jul 25 07:42:46 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0803/crmd/107, version=0.20.65)
Jul 25 07:42:46 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0803/crmd/109, version=0.20.65)
Jul 25 07:42:46 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/112)
Jul 25 07:42:46 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/113)
Jul 25 07:42:46 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=dvtvm0803/crmd/112, version=0.20.65)
Jul 25 07:42:46 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.65 2
Jul 25 07:42:46 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.66 (null)
Jul 25 07:42:46 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=66
Jul 25 07:42:46 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']: @in_ccm=false, @crm-debug-origin=post_cache_update
Jul 25 07:42:46 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='2']: @crm-debug-origin=post_cache_update
Jul 25 07:42:46 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0803/crmd/113, version=0.20.66)
Jul 25 07:42:46 [1624] DVTVM0803.mse.am.mot.com crmd: info: abort_transition_graph: Transition aborted by deletion of transient_attributes[@id='1']: Transient attribute change | cib=0.20.64 source=abort_unless_down:357 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1'] complete=true
Jul 25 07:42:47 [1623] DVTVM0803.mse.am.mot.com pengine: notice: unpack_config: On loss of CCM Quorum: Ignore
Jul 25 07:42:47 [1623] DVTVM0803.mse.am.mot.com pengine: info: determine_online_status: Node dvtvm0803 is online
Jul 25 07:42:47 [1623] DVTVM0803.mse.am.mot.com pengine: info: unpack_node_loop: Node 2 is already processed
Jul 25 07:42:47 [1623] DVTVM0803.mse.am.mot.com pengine: info: unpack_node_loop: Node 2 is already processed
Jul 25 07:42:47 [1623] DVTVM0803.mse.am.mot.com pengine: info: common_print: ClusterIP (ocf::heartbeat:IPaddr2): Stopped
Jul 25 07:42:47 [1623] DVTVM0803.mse.am.mot.com pengine: info: clone_print: Clone Set: Connected-clone [Connected]
Jul 25 07:42:47 [1623] DVTVM0803.mse.am.mot.com pengine: info: short_print: Started: [ dvtvm0803 ]
Jul 25 07:42:47 [1623] DVTVM0803.mse.am.mot.com pengine: info: short_print: Stopped: [ dvtvm0802 ]
Jul 25 07:42:47 [1623] DVTVM0803.mse.am.mot.com pengine: info: common_print: application (ocf::vendor:application): Stopped
Jul 25 07:42:47 [1623] DVTVM0803.mse.am.mot.com pengine: info: native_color: Resource Connected:1 cannot run anywhere
Jul 25 07:42:47 [1623] DVTVM0803.mse.am.mot.com pengine: info: RecurringOp: Start recurring monitor (1s) for ClusterIP on dvtvm0803
Jul 25 07:42:47 [1623] DVTVM0803.mse.am.mot.com pengine: info: RecurringOp: Start recurring monitor (1s) for application on dvtvm0803
Jul 25 07:42:47 [1623] DVTVM0803.mse.am.mot.com pengine: notice: LogAction: * Start ClusterIP ( dvtvm0803 )
Jul 25 07:42:47 [1623] DVTVM0803.mse.am.mot.com pengine: info: LogActions: Leave Connected:0 (Started dvtvm0803)
Jul 25 07:42:47 [1623] DVTVM0803.mse.am.mot.com pengine: info: LogActions: Leave Connected:1 (Stopped)
Jul 25 07:42:47 [1623] DVTVM0803.mse.am.mot.com pengine: notice: LogAction: * Start application ( dvtvm0803 )
Jul 25 07:42:47 [1623] DVTVM0803.mse.am.mot.com pengine: notice: process_pe_message: Calculated transition 6, saving inputs in /var/lib/pacemaker/pengine/pe-input-16.bz2
Jul 25 07:42:47 [1624] DVTVM0803.mse.am.mot.com crmd: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
Jul 25 07:42:47 [1624] DVTVM0803.mse.am.mot.com crmd: info: do_te_invoke: Processing graph 6 (ref=pe_calc-dc-1532518967-53) derived from /var/lib/pacemaker/pengine/pe-input-16.bz2
Jul 25 07:42:47 [1624] DVTVM0803.mse.am.mot.com crmd: notice: te_rsc_command: Initiating start operation ClusterIP_start_0 locally on dvtvm0803 | action 3
Jul 25 07:42:47 [1624] DVTVM0803.mse.am.mot.com crmd: info: do_lrm_rsc_op: Performing key=3:6:0:1cec18ac-2aae-4c5c-8513-d6851bc10191 op=ClusterIP_start_0
Jul 25 07:42:47 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/116)
Jul 25 07:42:47 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.66 2
Jul 25 07:42:47 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.67 (null)
Jul 25 07:42:47 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=67
Jul 25 07:42:47 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_update_resource
Jul 25 07:42:47 [1620] DVTVM0803.mse.am.mot.com lrmd: info: log_execute: executing - rsc:ClusterIP action:start call_id:25
Jul 25 07:42:47 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='ClusterIP']/lrm_rsc_op[@id='ClusterIP_last_0']: @operation_key=ClusterIP_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=3:6:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @transition-magic=-1:193;3:6:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @call-id=-1, @rc-code=193, @op-status=-1, @last-run=1532518967, @last-rc-chan
Jul 25 07:42:47 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0803/crmd/116, version=0.20.67)
Jul 25 07:42:47 IPaddr2(ClusterIP)[12697]: INFO: Adding inet address 10.144.102.230/24 with broadcast address 10.144.102.255 to device eth0
Jul 25 07:42:47 IPaddr2(ClusterIP)[12697]: INFO: Bringing device eth0 up
Jul 25 07:42:47 IPaddr2(ClusterIP)[12697]: INFO: /usr/libexec/heartbeat/send_arp -i 200 -c 5 -p /var/run/resource-agents/send_arp-10.144.102.230 -I eth0 -m auto 10.144.102.230
Jul 25 07:42:49 ping(Connected)[12678]: WARNING: 10.144.102.5 is inactive: PING 10.144.102.5 (10.144.102.5) 56(84) bytes of data.
--- 10.144.102.5 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 999ms
Jul 25 07:42:51 [1620] DVTVM0803.mse.am.mot.com lrmd: info: log_finished: finished - rsc:ClusterIP action:start call_id:25 pid:12697 exit-code:0 exec-time:4106ms queue-time:0ms
Jul 25 07:42:51 [1624] DVTVM0803.mse.am.mot.com crmd: notice: process_lrm_event: Result of start operation for ClusterIP on dvtvm0803: 0 (ok) | call=25 key=ClusterIP_start_0 confirmed=true cib-update=117
Jul 25 07:42:51 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/117)
Jul 25 07:42:51 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.67 2
Jul 25 07:42:51 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.68 (null)
Jul 25 07:42:51 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=68
Jul 25 07:42:51 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='ClusterIP']/lrm_rsc_op[@id='ClusterIP_last_0']: @transition-magic=0:0;3:6:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @call-id=25, @rc-code=0, @op-status=0, @exec-time=4106
Jul 25 07:42:51 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0803/crmd/117, version=0.20.68)
Jul 25 07:42:51 [1624] DVTVM0803.mse.am.mot.com crmd: info: match_graph_event: Action ClusterIP_start_0 (3) confirmed on dvtvm0803 (rc=0)
Jul 25 07:42:51 [1624] DVTVM0803.mse.am.mot.com crmd: notice: te_rsc_command: Initiating monitor operation ClusterIP_monitor_1000 locally on dvtvm0803 | action 4
Jul 25 07:42:51 [1624] DVTVM0803.mse.am.mot.com crmd: info: do_lrm_rsc_op: Performing key=4:6:0:1cec18ac-2aae-4c5c-8513-d6851bc10191 op=ClusterIP_monitor_1000
Jul 25 07:42:51 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/118)
Jul 25 07:42:51 [1624] DVTVM0803.mse.am.mot.com crmd: notice: te_rsc_command: Initiating start operation edgemanager_start_0 locally on dvtvm0803 | action 11
Jul 25 07:42:51 [1624] DVTVM0803.mse.am.mot.com crmd: info: do_lrm_rsc_op: Performing key=11:6:0:1cec18ac-2aae-4c5c-8513-d6851bc10191 op=edgemanager_start_0
Jul 25 07:42:51 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.68 2
Jul 25 07:42:51 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.69 (null)
Jul 25 07:42:51 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=69
Jul 25 07:42:51 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='ClusterIP']: <lrm_rsc_op id="ClusterIP_monitor_1000" operation_key="ClusterIP_monitor_1000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="4:6:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" transition-magic="-1:193;4:6:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" exit-reason="" on_node="dvtvm0803" call-
Jul 25 07:42:51 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0803/crmd/118, version=0.20.69)
Jul 25 07:42:51 [1624] DVTVM0803.mse.am.mot.com crmd: info: action_synced_wait: Managed edgemanager_meta-data_0 process 12828 exited with rc=4
Jul 25 07:42:51 [1624] DVTVM0803.mse.am.mot.com crmd: error: lrmd_api_get_metadata: Failed to receive meta-data for ocf:vendor:application
Jul 25 07:42:51 [1624] DVTVM0803.mse.am.mot.com crmd: warning: build_operation_update: Failed to get metadata for application (ocf:vendor:application)
Jul 25 07:42:51 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/119)
Jul 25 07:42:51 [1620] DVTVM0803.mse.am.mot.com lrmd: info: log_execute: executing - rsc:application action:start call_id:27
Jul 25 07:42:51 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.69 2
Jul 25 07:42:51 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.70 (null)
Jul 25 07:42:51 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=70
Jul 25 07:42:51 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='edgemanager']/lrm_rsc_op[@id='edgemanager_last_0']: @operation_key=edgemanager_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=11:6:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @transition-magic=-1:193;11:6:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @call-id=-1, @rc-code=193, @op-status=-1, @last-run=1532518971, @last
Jul 25 07:42:51 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0803/crmd/119, version=0.20.70)
Jul 25 07:42:51 application(application)[12829]: INFO: Starting APP - setting to Primary
Jul 25 07:42:51 [1620] DVTVM0803.mse.am.mot.com lrmd: info: log_finished: finished - rsc:application action:start call_id:27 pid:12829 exit-code:0 exec-time:33ms queue-time:0ms
Jul 25 07:42:51 [1624] DVTVM0803.mse.am.mot.com crmd: info: action_synced_wait: Managed edgemanager_meta-data_0 process 12865 exited with rc=4
Jul 25 07:42:51 [1624] DVTVM0803.mse.am.mot.com crmd: error: lrmd_api_get_metadata: Failed to receive meta-data for ocf:vendor:application
Jul 25 07:42:51 [1624] DVTVM0803.mse.am.mot.com crmd: warning: build_operation_update: Failed to get metadata for application (ocf:vendor:application)
Jul 25 07:42:51 [1624] DVTVM0803.mse.am.mot.com crmd: notice: process_lrm_event: Result of start operation for application on dvtvm0803: 0 (ok) | call=27 key=edgemanager_start_0 confirmed=true cib-update=120
Jul 25 07:42:51 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/120)
Jul 25 07:42:51 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.70 2
Jul 25 07:42:51 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.71 (null)
Jul 25 07:42:51 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=71
Jul 25 07:42:51 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='edgemanager']/lrm_rsc_op[@id='edgemanager_last_0']: @transition-magic=0:0;11:6:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @call-id=27, @rc-code=0, @op-status=0, @exec-time=33
Jul 25 07:42:51 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0803/crmd/120, version=0.20.71)
Jul 25 07:42:51 [1624] DVTVM0803.mse.am.mot.com crmd: info: match_graph_event: Action edgemanager_start_0 (11) confirmed on dvtvm0803 (rc=0)
Jul 25 07:42:51 [1624] DVTVM0803.mse.am.mot.com crmd: notice: te_rsc_command: Initiating monitor operation edgemanager_monitor_1000 locally on dvtvm0803 | action 12
Jul 25 07:42:51 [1624] DVTVM0803.mse.am.mot.com crmd: info: do_lrm_rsc_op: Performing key=12:6:0:1cec18ac-2aae-4c5c-8513-d6851bc10191 op=edgemanager_monitor_1000
Jul 25 07:42:51 [1624] DVTVM0803.mse.am.mot.com crmd: info: action_synced_wait: Managed edgemanager_meta-data_0 process 12869 exited with rc=4
Jul 25 07:42:51 [1624] DVTVM0803.mse.am.mot.com crmd: error: lrmd_api_get_metadata: Failed to receive meta-data for ocf:vendor:application
Jul 25 07:42:51 [1624] DVTVM0803.mse.am.mot.com crmd: warning: build_operation_update: Failed to get metadata for application (ocf:vendor:application)
Jul 25 07:42:51 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/121)
Jul 25 07:42:51 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.71 2
Jul 25 07:42:51 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.72 (null)
Jul 25 07:42:51 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=72
Jul 25 07:42:51 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: ++ /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='edgemanager']: <lrm_rsc_op id="edgemanager_monitor_1000" operation_key="edgemanager_monitor_1000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.14" transition-key="12:6:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" transition-magic="-1:193;12:6:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" exit-reason="" on_node="dvtvm080
Jul 25 07:42:51 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0803/crmd/121, version=0.20.72)
Jul 25 07:42:51 [1624] DVTVM0803.mse.am.mot.com crmd: info: process_lrm_event: Result of monitor operation for ClusterIP on dvtvm0803: 0 (ok) | call=26 key=ClusterIP_monitor_1000 confirmed=false cib-update=122
Jul 25 07:42:51 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/122)
Jul 25 07:42:51 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.72 2
Jul 25 07:42:51 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.73 (null)
Jul 25 07:42:51 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=73
Jul 25 07:42:51 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='ClusterIP']/lrm_rsc_op[@id='ClusterIP_monitor_1000']: @transition-magic=0:0;4:6:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @call-id=26, @rc-code=0, @op-status=0, @exec-time=56
Jul 25 07:42:51 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0803/crmd/122, version=0.20.73)
Jul 25 07:42:51 [1624] DVTVM0803.mse.am.mot.com crmd: info: match_graph_event: Action ClusterIP_monitor_1000 (4) confirmed on dvtvm0803 (rc=0)
Jul 25 07:42:51 [1624] DVTVM0803.mse.am.mot.com crmd: info: action_synced_wait: Managed edgemanager_meta-data_0 process 12896 exited with rc=4
Jul 25 07:42:51 [1624] DVTVM0803.mse.am.mot.com crmd: error: lrmd_api_get_metadata: Failed to receive meta-data for ocf:vendor:application
Jul 25 07:42:51 [1624] DVTVM0803.mse.am.mot.com crmd: warning: build_operation_update: Failed to get metadata for application (ocf:vendor:application)
Jul 25 07:42:51 [1624] DVTVM0803.mse.am.mot.com crmd: info: process_lrm_event: Result of monitor operation for application on dvtvm0803: 0 (ok) | call=28 key=edgemanager_monitor_1000 confirmed=false cib-update=123
Jul 25 07:42:51 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/123)
Jul 25 07:42:51 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.73 2
Jul 25 07:42:51 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.74 (null)
Jul 25 07:42:51 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=74
Jul 25 07:42:51 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='edgemanager']/lrm_rsc_op[@id='edgemanager_monitor_1000']: @transition-magic=0:0;12:6:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @call-id=28, @rc-code=0, @op-status=0, @exec-time=16
Jul 25 07:42:51 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0803/crmd/123, version=0.20.74)
Jul 25 07:42:51 [1624] DVTVM0803.mse.am.mot.com crmd: info: match_graph_event: Action edgemanager_monitor_1000 (12) confirmed on dvtvm0803 (rc=0)
Jul 25 07:42:51 [1624] DVTVM0803.mse.am.mot.com crmd: notice: run_graph: Transition 6 (Complete=4, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-16.bz2): Complete
Jul 25 07:42:51 [1624] DVTVM0803.mse.am.mot.com crmd: info: do_log: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd
Jul 25 07:42:51 [1624] DVTVM0803.mse.am.mot.com crmd: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd
Jul 25 07:42:52 ping(Connected)[12678]: WARNING: 10.144.102.152 is inactive: PING 10.144.102.152 (10.144.102.152) 56(84) bytes of data.
--- 10.144.102.152 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 1000ms
Jul 25 07:42:52 [1622] DVTVM0803.mse.am.mot.com attrd: info: attrd_peer_update: Setting pingd[dvtvm0803]: 2000 -> 0 from dvtvm0803
Jul 25 07:42:53 [1622] DVTVM0803.mse.am.mot.com attrd: info: write_attribute: Sent update 8 with 1 changes for pingd, id=<n/a>, set=(null)
Jul 25 07:42:53 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/attrd/8)
Jul 25 07:42:53 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.74 2
Jul 25 07:42:53 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.75 (null)
Jul 25 07:42:53 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=75
Jul 25 07:42:53 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-pingd']: @value=0
Jul 25 07:42:53 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0803/attrd/8, version=0.20.75)
Jul 25 07:42:53 [1624] DVTVM0803.mse.am.mot.com crmd: info: abort_transition_graph: Transition aborted by status-2-pingd doing modify pingd=0: Transient attribute change | cib=0.20.75 source=abort_unless_down:343 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-pingd'] complete=true
Jul 25 07:42:53 [1624] DVTVM0803.mse.am.mot.com crmd: notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph
Jul 25 07:42:53 [1622] DVTVM0803.mse.am.mot.com attrd: info: attrd_cib_callback: Update 8 for pingd: OK (0)
Jul 25 07:42:53 [1622] DVTVM0803.mse.am.mot.com attrd: info: attrd_cib_callback: Update 8 for pingd[dvtvm0803]=0: OK (0)
Jul 25 07:42:53 [1623] DVTVM0803.mse.am.mot.com pengine: notice: unpack_config: On loss of CCM Quorum: Ignore
Jul 25 07:42:53 [1623] DVTVM0803.mse.am.mot.com pengine: info: determine_online_status: Node dvtvm0803 is online
Jul 25 07:42:53 [1623] DVTVM0803.mse.am.mot.com pengine: info: unpack_node_loop: Node 2 is already processed
Jul 25 07:42:53 [1623] DVTVM0803.mse.am.mot.com pengine: info: unpack_node_loop: Node 2 is already processed
Jul 25 07:42:53 [1623] DVTVM0803.mse.am.mot.com pengine: info: common_print: ClusterIP (ocf::heartbeat:IPaddr2): Started dvtvm0803
Jul 25 07:42:53 [1623] DVTVM0803.mse.am.mot.com pengine: info: clone_print: Clone Set: Connected-clone [Connected]
Jul 25 07:42:53 [1623] DVTVM0803.mse.am.mot.com pengine: info: short_print: Started: [ dvtvm0803 ]
Jul 25 07:42:53 [1623] DVTVM0803.mse.am.mot.com pengine: info: short_print: Stopped: [ dvtvm0802 ]
Jul 25 07:42:53 [1623] DVTVM0803.mse.am.mot.com pengine: info: common_print: application (ocf::vendor:application): Started dvtvm0803
Jul 25 07:42:53 [1623] DVTVM0803.mse.am.mot.com pengine: info: rsc_merge_weights: application: Rolling back scores from ClusterIP
Jul 25 07:42:53 [1623] DVTVM0803.mse.am.mot.com pengine: info: native_color: Resource ClusterIP cannot run anywhere
Jul 25 07:42:53 [1623] DVTVM0803.mse.am.mot.com pengine: info: native_color: Resource Connected:1 cannot run anywhere
Jul 25 07:42:53 [1623] DVTVM0803.mse.am.mot.com pengine: notice: LogAction: * Stop ClusterIP ( dvtvm0803 ) due to node availability
Jul 25 07:42:53 [1623] DVTVM0803.mse.am.mot.com pengine: info: LogActions: Leave Connected:0 (Started dvtvm0803)
Jul 25 07:42:53 [1623] DVTVM0803.mse.am.mot.com pengine: info: LogActions: Leave Connected:1 (Stopped)
Jul 25 07:42:53 [1623] DVTVM0803.mse.am.mot.com pengine: notice: LogAction: * Stop application ( dvtvm0803 ) due to required ClusterIP start
Jul 25 07:42:53 [1623] DVTVM0803.mse.am.mot.com pengine: notice: process_pe_message: Calculated transition 7, saving inputs in /var/lib/pacemaker/pengine/pe-input-17.bz2
Jul 25 07:42:53 [1624] DVTVM0803.mse.am.mot.com crmd: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
Jul 25 07:42:53 [1624] DVTVM0803.mse.am.mot.com crmd: info: do_te_invoke: Processing graph 7 (ref=pe_calc-dc-1532518973-58) derived from /var/lib/pacemaker/pengine/pe-input-17.bz2
Jul 25 07:42:53 [1624] DVTVM0803.mse.am.mot.com crmd: notice: te_rsc_command: Initiating stop operation edgemanager_stop_0 locally on dvtvm0803 | action 12
Jul 25 07:42:53 [1620] DVTVM0803.mse.am.mot.com lrmd: info: cancel_recurring_action: Cancelling ocf operation edgemanager_monitor_1000
Jul 25 07:42:53 [1624] DVTVM0803.mse.am.mot.com crmd: info: do_lrm_rsc_op: Performing key=12:7:0:1cec18ac-2aae-4c5c-8513-d6851bc10191 op=edgemanager_stop_0
Jul 25 07:42:53 [1620] DVTVM0803.mse.am.mot.com lrmd: info: log_execute: executing - rsc:application action:stop call_id:30
Jul 25 07:42:53 [1624] DVTVM0803.mse.am.mot.com crmd: info: process_lrm_event: Result of monitor operation for application on dvtvm0803: Cancelled | call=28 key=edgemanager_monitor_1000 confirmed=true
Jul 25 07:42:53 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/125)
Jul 25 07:42:53 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.75 2
Jul 25 07:42:53 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.76 (null)
Jul 25 07:42:53 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=76
Jul 25 07:42:53 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='edgemanager']/lrm_rsc_op[@id='edgemanager_last_0']: @operation_key=edgemanager_stop_0, @operation=stop, @transition-key=12:7:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @transition-magic=-1:193;12:7:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @call-id=-1, @rc-code=193, @op-status=-1, @last-run=1532518973, @last-rc-change=1532518973, @exec-time=0
Jul 25 07:42:53 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0803/crmd/125, version=0.20.76)
Jul 25 07:42:53 application(application)[12958]: INFO: Stopping APP - setting to Standby
Jul 25 07:42:53 [1620] DVTVM0803.mse.am.mot.com lrmd: notice: operation_finished: edgemanager_stop_0:12958:stderr [ /usr/local/vendor/bin/PrimaryMode.sh: line 27: $Filename: ambiguous redirect ]
Jul 25 07:42:53 [1620] DVTVM0803.mse.am.mot.com lrmd: info: log_finished: finished - rsc:application action:stop call_id:30 pid:12958 exit-code:0 exec-time:34ms queue-time:0ms
Jul 25 07:42:53 [1624] DVTVM0803.mse.am.mot.com crmd: notice: process_lrm_event: Result of stop operation for application on dvtvm0803: 0 (ok) | call=30 key=edgemanager_stop_0 confirmed=true cib-update=126
Jul 25 07:42:53 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/126)
Jul 25 07:42:53 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.76 2
Jul 25 07:42:53 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.77 (null)
Jul 25 07:42:53 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=77
Jul 25 07:42:53 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='edgemanager']/lrm_rsc_op[@id='edgemanager_last_0']: @transition-magic=0:0;12:7:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @call-id=30, @rc-code=0, @op-status=0, @exec-time=34
Jul 25 07:42:53 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0803/crmd/126, version=0.20.77)
Jul 25 07:42:53 [1624] DVTVM0803.mse.am.mot.com crmd: info: match_graph_event: Action edgemanager_stop_0 (12) confirmed on dvtvm0803 (rc=0)
Jul 25 07:42:53 [1624] DVTVM0803.mse.am.mot.com crmd: notice: te_rsc_command: Initiating stop operation ClusterIP_stop_0 locally on dvtvm0803 | action 5
Jul 25 07:42:53 [1620] DVTVM0803.mse.am.mot.com lrmd: info: cancel_recurring_action: Cancelling ocf operation ClusterIP_monitor_1000
Jul 25 07:42:53 [1624] DVTVM0803.mse.am.mot.com crmd: info: do_lrm_rsc_op: Performing key=5:7:0:1cec18ac-2aae-4c5c-8513-d6851bc10191 op=ClusterIP_stop_0
Jul 25 07:42:53 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/127)
Jul 25 07:42:53 [1620] DVTVM0803.mse.am.mot.com lrmd: info: log_execute: executing - rsc:ClusterIP action:stop call_id:32
Jul 25 07:42:53 [1624] DVTVM0803.mse.am.mot.com crmd: info: process_lrm_event: Result of monitor operation for ClusterIP on dvtvm0803: Cancelled | call=26 key=ClusterIP_monitor_1000 confirmed=true
Jul 25 07:42:53 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.77 2
Jul 25 07:42:53 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.78 (null)
Jul 25 07:42:53 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=78
Jul 25 07:42:53 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='ClusterIP']/lrm_rsc_op[@id='ClusterIP_last_0']: @operation_key=ClusterIP_stop_0, @operation=stop, @transition-key=5:7:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @transition-magic=-1:193;5:7:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @call-id=-1, @rc-code=193, @op-status=-1, @last-run=1532518973, @last-rc-change=1532518973, @exec-time=0
Jul 25 07:42:53 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0803/crmd/127, version=0.20.78)
Jul 25 07:42:53 IPaddr2(ClusterIP)[12970]: INFO: IP status = ok, IP_CIP=
Jul 25 07:42:53 [1620] DVTVM0803.mse.am.mot.com lrmd: info: log_finished: finished - rsc:ClusterIP action:stop call_id:32 pid:12970 exit-code:0 exec-time:74ms queue-time:0ms
Jul 25 07:42:53 [1624] DVTVM0803.mse.am.mot.com crmd: notice: process_lrm_event: Result of stop operation for ClusterIP on dvtvm0803: 0 (ok) | call=32 key=ClusterIP_stop_0 confirmed=true cib-update=128
Jul 25 07:42:53 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/128)
Jul 25 07:42:53 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.78 2
Jul 25 07:42:53 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.79 (null)
Jul 25 07:42:53 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=79
Jul 25 07:42:53 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='ClusterIP']/lrm_rsc_op[@id='ClusterIP_last_0']: @transition-magic=0:0;5:7:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @call-id=32, @rc-code=0, @op-status=0, @exec-time=74
Jul 25 07:42:53 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0803/crmd/128, version=0.20.79)
Jul 25 07:42:53 [1624] DVTVM0803.mse.am.mot.com crmd: info: match_graph_event: Action ClusterIP_stop_0 (5) confirmed on dvtvm0803 (rc=0)
Jul 25 07:42:53 [1624] DVTVM0803.mse.am.mot.com crmd: notice: run_graph: Transition 7 (Complete=3, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-17.bz2): Complete
Jul 25 07:42:53 [1624] DVTVM0803.mse.am.mot.com crmd: info: do_log: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd
Jul 25 07:42:53 [1624] DVTVM0803.mse.am.mot.com crmd: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd
Jul 25 07:42:58 ping(Connected)[13080]: WARNING: 10.144.102.5 is inactive: PING 10.144.102.5 (10.144.102.5) 56(84) bytes of data.
--- 10.144.102.5 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 999ms
Jul 25 07:42:58 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_ping: Reporting our current digest to dvtvm0803: c20c09f5d9e99f10d879bc76de5ac6bb for 0.20.79 (0x55d4164ad140 0)
Jul 25 07:43:01 ping(Connected)[13080]: WARNING: 10.144.102.152 is inactive: PING 10.144.102.152 (10.144.102.152) 56(84) bytes of data.
--- 10.144.102.152 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 999ms
Jul 25 07:43:07 ping(Connected)[13164]: WARNING: 10.144.102.5 is inactive: PING 10.144.102.5 (10.144.102.5) 56(84) bytes of data.
--- 10.144.102.5 ping statistics ---
2 packets transmitted, 0 received, 100% packet loss, time 999ms
[994] DVTVM0803.mse.am.mot.com corosyncnotice [TOTEM ] A new membership (10.144.102.32:56) was formed. Members joined: 1
Jul 25 07:43:09 [1619] DVTVM0803.mse.am.mot.com stonith-ng: info: pcmk_cpg_membership: Node 1 joined group stonith-ng (counter=4.0)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: pcmk_cpg_membership: Node 1 joined group cib (counter=4.0)
Jul 25 07:43:09 [1556] DVTVM0803.mse.am.mot.com pacemakerd: info: pcmk_cpg_membership: Node 1 joined group pacemakerd (counter=4.0)
Jul 25 07:43:09 [1556] DVTVM0803.mse.am.mot.com pacemakerd: info: pcmk_cpg_membership: Node 1 still member of group pacemakerd (peer=dvtvm0802, counter=4.0)
Jul 25 07:43:09 [1556] DVTVM0803.mse.am.mot.com pacemakerd: info: crm_update_peer_proc: pcmk_cpg_membership: Node dvtvm0802[1] - corosync-cpg is now online
Jul 25 07:43:09 [1556] DVTVM0803.mse.am.mot.com pacemakerd: error: pcmk_cpg_membership: Node dvtvm0802[1] appears to be online even though we think it is dead
Jul 25 07:43:09 [1556] DVTVM0803.mse.am.mot.com pacemakerd: notice: crm_update_peer_state_iter: Node dvtvm0802 state is now member | nodeid=1 previous=lost source=pcmk_cpg_membership
Jul 25 07:43:09 [1556] DVTVM0803.mse.am.mot.com pacemakerd: info: pcmk_cpg_membership: Node 2 still member of group pacemakerd (peer=dvtvm0803, counter=4.1)
Jul 25 07:43:09 [1556] DVTVM0803.mse.am.mot.com pacemakerd: info: crm_cs_flush: Sent 0 CPG messages (1 remaining, last=10): Try again (6)
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: pcmk_cpg_membership: Node 1 joined group crmd (counter=4.0)
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: pcmk_cpg_membership: Node 1 still member of group crmd (peer=dvtvm0802, counter=4.0)
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: crm_update_peer_proc: pcmk_cpg_membership: Node dvtvm0802[1] - corosync-cpg is now online
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: peer_update_callback: Client dvtvm0802/peer now has status [online] (DC=true, changed=4000000)
Jul 25 07:43:09 [1622] DVTVM0803.mse.am.mot.com attrd: info: pcmk_cpg_membership: Node 1 joined group attrd (counter=4.0)
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: error: pcmk_cpg_membership: Node dvtvm0802[1] appears to be online even though we think it is dead
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: notice: crm_update_peer_state_iter: Node dvtvm0802 state is now member | nodeid=1 previous=lost source=pcmk_cpg_membership
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: peer_update_callback: Cluster node dvtvm0802 is now member (was lost)
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: pcmk_cpg_membership: Node 2 still member of group crmd (peer=dvtvm0803, counter=4.1)
[994] DVTVM0803.mse.am.mot.com corosyncnotice [QUORUM] Members[2]: 1 2
[994] DVTVM0803.mse.am.mot.com corosyncnotice [MAIN ] Completed service synchronization, ready to provide service.
Jul 25 07:43:09 [1619] DVTVM0803.mse.am.mot.com stonith-ng: info: crm_get_peer: Created entry a3f26acd-11c5-4e8c-a839-fa51ac07f3c0/0x559fa4ef9300 for node dvtvm0802/1 (2 total)
Jul 25 07:43:09 [1619] DVTVM0803.mse.am.mot.com stonith-ng: info: crm_get_peer: Node 1 is now known as dvtvm0802
Jul 25 07:43:09 [1619] DVTVM0803.mse.am.mot.com stonith-ng: info: crm_get_peer: Node 1 has uuid 1
Jul 25 07:43:09 [1619] DVTVM0803.mse.am.mot.com stonith-ng: info: pcmk_cpg_membership: Node 1 still member of group stonith-ng (peer=dvtvm0802, counter=4.0)
Jul 25 07:43:09 [1619] DVTVM0803.mse.am.mot.com stonith-ng: info: crm_update_peer_proc: pcmk_cpg_membership: Node dvtvm0802[1] - corosync-cpg is now online
Jul 25 07:43:09 [1619] DVTVM0803.mse.am.mot.com stonith-ng: notice: crm_update_peer_state_iter: Node dvtvm0802 state is now member | nodeid=1 previous=unknown source=crm_update_peer_proc
Jul 25 07:43:09 [1619] DVTVM0803.mse.am.mot.com stonith-ng: info: pcmk_cpg_membership: Node 2 still member of group stonith-ng (peer=dvtvm0803, counter=4.1)
Jul 25 07:43:09 [1556] DVTVM0803.mse.am.mot.com pacemakerd: info: pcmk_quorum_notification: Quorum retained | membership=56 members=2
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: pcmk_quorum_notification: Quorum retained | membership=56 members=2
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: crm_get_peer: Created entry 9ac08c1f-5b08-4e91-9cf1-69e9b55e7307/0x55d4165461d0 for node dvtvm0802/1 (2 total)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: crm_get_peer: Node 1 is now known as dvtvm0802
Jul 25 07:43:09 [1622] DVTVM0803.mse.am.mot.com attrd: info: crm_get_peer: Created entry 9d8450fc-da7a-4f92-8fc9-06a7a83f3334/0x56375c6bc590 for node dvtvm0802/1 (2 total)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: crm_get_peer: Node 1 has uuid 1
Jul 25 07:43:09 [1622] DVTVM0803.mse.am.mot.com attrd: info: crm_get_peer: Node 1 is now known as dvtvm0802
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: pcmk_cpg_membership: Node 1 still member of group cib (peer=dvtvm0802, counter=4.0)
Jul 25 07:43:09 [1622] DVTVM0803.mse.am.mot.com attrd: info: crm_get_peer: Node 1 has uuid 1
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: crm_update_peer_proc: pcmk_cpg_membership: Node dvtvm0802[1] - corosync-cpg is now online
Jul 25 07:43:09 [1622] DVTVM0803.mse.am.mot.com attrd: info: pcmk_cpg_membership: Node 1 still member of group attrd (peer=dvtvm0802, counter=4.0)
Jul 25 07:43:09 [1622] DVTVM0803.mse.am.mot.com attrd: info: crm_update_peer_proc: pcmk_cpg_membership: Node dvtvm0802[1] - corosync-cpg is now online
Jul 25 07:43:09 [1622] DVTVM0803.mse.am.mot.com attrd: notice: crm_update_peer_state_iter: Node dvtvm0802 state is now member | nodeid=1 previous=unknown source=crm_update_peer_proc
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: notice: crm_update_peer_state_iter: Node dvtvm0802 state is now member | nodeid=1 previous=unknown source=crm_update_peer_proc
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: pcmk_cpg_membership: Node 2 still member of group cib (peer=dvtvm0803, counter=4.1)
Jul 25 07:43:09 [1622] DVTVM0803.mse.am.mot.com attrd: info: pcmk_cpg_membership: Node 2 still member of group attrd (peer=dvtvm0803, counter=4.1)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/129)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/130)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/133)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/134)
Jul 25 07:43:09 [1622] DVTVM0803.mse.am.mot.com attrd: notice: attrd_peer_message: Detected another attribute writer: dvtvm0802
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: register_fsa_error_adv: Resetting the current action list
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: warning: crmd_ha_msg_filter: Another DC detected: dvtvm0802 (op=noop)
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: notice: do_state_transition: State transition S_IDLE -> S_ELECTION | input=I_ELECTION cause=C_FSA_INTERNAL origin=crmd_ha_msg_filter
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: update_dc: Unset DC. Was dvtvm0803
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.79 2
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.80 (null)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=80
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='2']: @crm-debug-origin=peer_update_callback
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0802/crmd/82, version=0.20.80)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0802/crmd/83, version=0.20.80)
Jul 25 07:43:09 [1622] DVTVM0803.mse.am.mot.com attrd: info: attrd_peer_message: Processing sync-response from dvtvm0802
Jul 25 07:43:09 [1622] DVTVM0803.mse.am.mot.com attrd: info: attrd_peer_update: Setting #attrd-protocol[dvtvm0802]: (null) -> 2 from dvtvm0802
Jul 25 07:43:09 [1622] DVTVM0803.mse.am.mot.com attrd: info: attrd_peer_update: Setting pingd[dvtvm0802]: (null) -> 2000 from dvtvm0802
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: election_count_vote: Election 7 (owner: 1) pass: vote from dvtvm0802 (Uptime)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=dvtvm0802/crmd/86, version=0.20.80)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.80 2
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.81 (null)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=81
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']: @in_ccm=true, @crmd=online
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='2']: @crm-debug-origin=post_cache_update
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0802/crmd/87, version=0.20.81)
Jul 25 07:43:09 [1622] DVTVM0803.mse.am.mot.com attrd: info: election_count_vote: Election 4 (owner: 1) pass: vote from dvtvm0802 (Uptime)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.81 2
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.82 (null)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=82
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']: @crm-debug-origin=peer_update_callback
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0803/crmd/129, version=0.20.82)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0803/crmd/130, version=0.20.82)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=dvtvm0803/crmd/133, version=0.20.82)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.82 2
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.83 (null)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=83
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']: @crm-debug-origin=post_cache_update
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0803/crmd/134, version=0.20.83)
Jul 25 07:43:09 [1622] DVTVM0803.mse.am.mot.com attrd: info: election_complete: Election election-attrd complete
Jul 25 07:43:09 [1622] DVTVM0803.mse.am.mot.com attrd: info: write_attribute: Processed 2 private changes for #attrd-protocol, id=<n/a>, set=(null)
Jul 25 07:43:09 [1622] DVTVM0803.mse.am.mot.com attrd: info: write_attribute: Write out of 'pingd' delayed: timer is running
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: election_complete: Election election-0 complete
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: election_timeout_popped: Election failed: Declaring ourselves the winner
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: do_log: Input I_ELECTION_DC received in state S_ELECTION from election_timeout_popped
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: notice: do_state_transition: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_TIMER_POPPED origin=election_timeout_popped
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: do_dc_takeover: Taking over DC status for this partition
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/135, version=0.20.83)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section cib to all (origin=local/crmd/136)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/138)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section cib: OK (rc=0, origin=dvtvm0803/crmd/136, version=0.20.83)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section crm_config: OK (rc=0, origin=dvtvm0803/crmd/138, version=0.20.83)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/140)
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: crm_update_peer_join: initialize_join: Node dvtvm0803[2] - join-5 phase confirmed -> none
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: join_make_offer: Making join offers based on membership 56
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: join_make_offer: join-5: Sending offer to dvtvm0802
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: crm_update_peer_join: join_make_offer: Node dvtvm0802[1] - join-5 phase none -> welcomed
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: join_make_offer: join-5: Sending offer to dvtvm0803
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: crm_update_peer_join: join_make_offer: Node dvtvm0803[2] - join-5 phase none -> welcomed
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: do_dc_join_offer_all: join-5: Waiting on 2 outstanding join acks
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: warning: do_log: Input I_ELECTION_DC received in state S_INTEGRATION from do_election_check
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: crm_update_peer_join: initialize_join: Node dvtvm0802[1] - join-6 phase welcomed -> none
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: crm_update_peer_join: initialize_join: Node dvtvm0803[2] - join-6 phase welcomed -> none
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: join_make_offer: join-6: Sending offer to dvtvm0802
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: crm_update_peer_join: join_make_offer: Node dvtvm0802[1] - join-6 phase none -> welcomed
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: join_make_offer: join-6: Sending offer to dvtvm0803
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: crm_update_peer_join: join_make_offer: Node dvtvm0803[2] - join-6 phase none -> welcomed
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: do_dc_join_offer_all: join-6: Waiting on 2 outstanding join acks
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section crm_config: OK (rc=0, origin=dvtvm0803/crmd/140, version=0.20.83)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/142)
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: update_dc: Set DC to dvtvm0803 (3.0.14)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section crm_config: OK (rc=0, origin=dvtvm0803/crmd/142, version=0.20.83)
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: crm_update_peer_join: do_dc_join_filter_offer: Node dvtvm0803[2] - join-6 phase welcomed -> integrated
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: crm_update_peer_join: do_dc_join_filter_offer: Node dvtvm0802[1] - join-6 phase welcomed -> integrated
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: crmd_join_phase_log: join-6: dvtvm0802=integrated
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: crmd_join_phase_log: join-6: dvtvm0803=integrated
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: do_dc_join_finalize: join-6: Syncing our CIB to the rest of the cluster
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: crm_update_peer_join: finalize_join_for: Node dvtvm0802[1] - join-6 phase integrated -> finalized
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: crm_update_peer_join: finalize_join_for: Node dvtvm0803[2] - join-6 phase integrated -> finalized
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_replace: Digest matched on replace from dvtvm0803: e5254bf4e58c636caa44908bdc7e0a94
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_replace: Replaced 0.20.83 with 0.20.83 from dvtvm0803
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: crm_update_peer_join: do_dc_join_ack: Node dvtvm0803[2] - join-6 phase finalized -> confirmed
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: do_dc_join_ack: join-6: Updating node state to member for dvtvm0803
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: erase_status_tag: Deleting lrm status entries for dvtvm0803 | xpath=//node_state[@uname='dvtvm0803']/lrm
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_replace operation for section 'all': OK (rc=0, origin=dvtvm0803/crmd/146, version=0.20.83)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/147)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/148)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_delete operation for section //node_state[@uname='dvtvm0803']/lrm to all (origin=local/crmd/149)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/150)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=dvtvm0803/crmd/147, version=0.20.83)
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: crm_update_peer_join: do_dc_join_ack: Node dvtvm0802[1] - join-6 phase finalized -> confirmed
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: do_dc_join_ack: join-6: Updating node state to member for dvtvm0802
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: erase_status_tag: Deleting lrm status entries for dvtvm0802 | xpath=//node_state[@uname='dvtvm0802']/lrm
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=dvtvm0803/crmd/148, version=0.20.83)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.83 2
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.84 (null)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: -- /cib/status/node_state[@id='2']/lrm[@id='2']
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=84
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_delete operation for section //node_state[@uname='dvtvm0803']/lrm: OK (rc=0, origin=dvtvm0803/crmd/149, version=0.20.84)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.84 2
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.85 (null)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=85
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_lrm_query_internal
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: ++ /cib/status/node_state[@id='2']: <lrm id="2"/>
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_resources>
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_resource id="Connected" type="ping" class="ocf" provider="pacemaker">
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_rsc_op id="Connected_last_0" operation_key="Connected_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="8:0:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" transition-magic="0:0;8:0:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" exit-reason="" on_node="dvtvm0803" call-id="15" rc-code="0" op-status="0" interval="0" last-run="1532518326"
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_rsc_op id="Connected_monitor_3000" operation_key="Connected_monitor_3000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="9:0:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" transition-magic="0:0;9:0:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" exit-reason="" on_node="dvtvm0803" call-id="16" rc-code="0" op-status="0" interval="3000" last-r
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: ++ </lrm_resource>
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_resource id="ClusterIP" type="IPaddr2" class="ocf" provider="heartbeat">
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_rsc_op id="ClusterIP_last_0" operation_key="ClusterIP_stop_0" operation="stop" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="5:7:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" transition-magic="0:0;5:7:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" exit-reason="" on_node="dvtvm0803" call-id="32" rc-code="0" op-status="0" interval="0" last-run="1532518973" la
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: ++ </lrm_resource>
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_resource id="application" type="application" class="ocf" provider="vendor">
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_rsc_op id="edgemanager_last_0" operation_key="edgemanager_stop_0" operation="stop" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="12:7:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" transition-magic="0:0;12:7:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" exit-reason="" on_node="dvtvm0803" call-id="30" rc-code="0" op-status="0" interval="0" last-run="15325189
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: ++ </lrm_resource>
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: ++ </lrm_resources>
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: ++ </lrm>
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0803/crmd/150, version=0.20.85)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_delete operation for section //node_state[@uname='dvtvm0802']/lrm to all (origin=local/crmd/151)
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: do_state_transition: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/152)
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: abort_transition_graph: Transition aborted: Peer Cancelled | source=do_te_invoke:143 complete=true
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.85 2
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.86 (null)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: -- /cib/status/node_state[@id='1']/lrm[@id='1']
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=86
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_delete operation for section //node_state[@uname='dvtvm0802']/lrm: OK (rc=0, origin=dvtvm0803/crmd/151, version=0.20.86)
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: abort_transition_graph: Transition aborted by deletion of lrm[@id='1']: Resource state removal | cib=0.20.86 source=abort_unless_down:357 path=/cib/status/node_state[@id='1']/lrm[@id='1'] complete=true
Jul 25 07:43:09 [1622] DVTVM0803.mse.am.mot.com attrd: info: attrd_peer_update: Setting pingd[dvtvm0803]: 0 -> 1000 from dvtvm0803
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.86 2
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.87 (null)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=87
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_lrm_query_internal
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: ++ /cib/status/node_state[@id='1']: <lrm id="1"/>
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_resources>
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_resource id="Connected" type="ping" class="ocf" provider="pacemaker">
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_rsc_op id="Connected_last_0" operation_key="Connected_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="10:0:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" transition-magic="0:0;10:0:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" exit-reason="" on_node="dvtvm0802" call-id="15" rc-code="0" op-status="0" interval="0" last-run="1532518326
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_rsc_op id="Connected_monitor_3000" operation_key="Connected_monitor_3000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="11:0:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" transition-magic="0:0;11:0:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" exit-reason="" on_node="dvtvm0802" call-id="16" rc-code="0" op-status="0" interval="3000" last
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: ++ </lrm_resource>
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_resource id="ClusterIP" type="IPaddr2" class="ocf" provider="heartbeat">
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_rsc_op id="ClusterIP_last_0" operation_key="ClusterIP_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="4:1:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" transition-magic="0:0;4:1:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" exit-reason="" on_node="dvtvm0802" call-id="17" rc-code="0" op-status="0" interval="0" last-run="1532518330"
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_rsc_op id="ClusterIP_monitor_1000" operation_key="ClusterIP_monitor_1000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="5:1:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" transition-magic="0:0;5:1:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" exit-reason="" on_node="dvtvm0802" call-id="18" rc-code="0" op-status="0" interval="1000" last-r
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: ++ </lrm_resource>
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_resource id="application" type="application" class="ocf" provider="vendor">
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_rsc_op id="edgemanager_last_0" operation_key="edgemanager_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="14:1:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" transition-magic="0:0;14:1:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" exit-reason="" on_node="dvtvm0802" call-id="19" rc-code="0" op-status="0" interval="0" last-run="153251
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_rsc_op id="edgemanager_monitor_1000" operation_key="edgemanager_monitor_1000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="15:1:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" transition-magic="0:0;15:1:0:1cec18ac-2aae-4c5c-8513-d6851bc10191" exit-reason="" on_node="dvtvm0802" call-id="20" rc-code="0" op-status="0" interval="1000"
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: ++ </lrm_resource>
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: ++ </lrm_resources>
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: ++ </lrm>
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0803/crmd/152, version=0.20.87)
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: abort_transition_graph: Transition aborted: LRM Refresh | source=process_resource_updates:278 complete=true
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/155)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/156)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section cib to all (origin=local/crmd/157)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=dvtvm0803/crmd/155, version=0.20.87)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.87 2
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.88 (null)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=88
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0803/crmd/156, version=0.20.88)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section cib: OK (rc=0, origin=dvtvm0803/crmd/157, version=0.20.88)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_file_backup: Archived previous version as /var/lib/pacemaker/cib/cib-26.raw
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_file_write_with_digest: Wrote version 0.20.0 of the CIB to disk (digest: d281e07c969310cc6525c6f46ec6a307)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_file_write_with_digest: Reading cluster configuration file /var/lib/pacemaker/cib/cib.6H2SW1 (digest: /var/lib/pacemaker/cib/cib.RmM7iH)
Jul 25 07:43:09 [1623] DVTVM0803.mse.am.mot.com pengine: notice: unpack_config: On loss of CCM Quorum: Ignore
Jul 25 07:43:09 [1623] DVTVM0803.mse.am.mot.com pengine: info: determine_online_status: Node dvtvm0802 is online
Jul 25 07:43:09 [1623] DVTVM0803.mse.am.mot.com pengine: info: determine_online_status: Node dvtvm0803 is online
Jul 25 07:43:09 [1623] DVTVM0803.mse.am.mot.com pengine: info: unpack_node_loop: Node 1 is already processed
Jul 25 07:43:09 [1623] DVTVM0803.mse.am.mot.com pengine: info: unpack_node_loop: Node 2 is already processed
Jul 25 07:43:09 [1623] DVTVM0803.mse.am.mot.com pengine: info: unpack_node_loop: Node 1 is already processed
Jul 25 07:43:09 [1623] DVTVM0803.mse.am.mot.com pengine: info: unpack_node_loop: Node 2 is already processed
Jul 25 07:43:09 [1623] DVTVM0803.mse.am.mot.com pengine: info: common_print: ClusterIP (ocf::heartbeat:IPaddr2): Started dvtvm0802
Jul 25 07:43:09 [1623] DVTVM0803.mse.am.mot.com pengine: info: clone_print: Clone Set: Connected-clone [Connected]
Jul 25 07:43:09 [1623] DVTVM0803.mse.am.mot.com pengine: info: short_print: Started: [ dvtvm0802 dvtvm0803 ]
Jul 25 07:43:09 [1623] DVTVM0803.mse.am.mot.com pengine: info: common_print: application (ocf::vendor:application): Started dvtvm0802
Jul 25 07:43:09 [1623] DVTVM0803.mse.am.mot.com pengine: info: rsc_merge_weights: application: Rolling back scores from ClusterIP
Jul 25 07:43:09 [1623] DVTVM0803.mse.am.mot.com pengine: info: native_color: Resource ClusterIP cannot run anywhere
Jul 25 07:43:09 [1623] DVTVM0803.mse.am.mot.com pengine: notice: LogAction: * Stop ClusterIP ( dvtvm0802 ) due to node availability
Jul 25 07:43:09 [1623] DVTVM0803.mse.am.mot.com pengine: info: LogActions: Leave Connected:0 (Started dvtvm0802)
Jul 25 07:43:09 [1623] DVTVM0803.mse.am.mot.com pengine: info: LogActions: Leave Connected:1 (Started dvtvm0803)
Jul 25 07:43:09 [1623] DVTVM0803.mse.am.mot.com pengine: notice: LogAction: * Stop application ( dvtvm0802 ) due to required ClusterIP start
Jul 25 07:43:09 [1623] DVTVM0803.mse.am.mot.com pengine: notice: process_pe_message: Calculated transition 8, saving inputs in /var/lib/pacemaker/pengine/pe-input-18.bz2
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: do_te_invoke: Processing graph 8 (ref=pe_calc-dc-1532518989-73) derived from /var/lib/pacemaker/pengine/pe-input-18.bz2
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: notice: te_rsc_command: Initiating stop operation edgemanager_stop_0 on dvtvm0802 | action 15
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.88 2
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.89 (null)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=89
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_update_resource
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='edgemanager']/lrm_rsc_op[@id='edgemanager_last_0']: @operation_key=edgemanager_stop_0, @operation=stop, @crm-debug-origin=do_update_resource, @transition-key=15:8:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @transition-magic=-1:193;15:8:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @call-id=-1, @rc-code=193, @op-status=-1, @last-run=1532518989, @last-r
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0802/crmd/92, version=0.20.89)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.89 2
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.90 (null)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=90
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='edgemanager']/lrm_rsc_op[@id='edgemanager_last_0']: @transition-magic=0:0;15:8:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @call-id=22, @rc-code=0, @op-status=0, @exec-time=34
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0802/crmd/93, version=0.20.90)
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: match_graph_event: Action edgemanager_stop_0 (15) confirmed on dvtvm0802 (rc=0)
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: notice: te_rsc_command: Initiating stop operation ClusterIP_stop_0 on dvtvm0802 | action 6
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.90 2
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.91 (null)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=91
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='ClusterIP']/lrm_rsc_op[@id='ClusterIP_last_0']: @operation_key=ClusterIP_stop_0, @operation=stop, @crm-debug-origin=do_update_resource, @transition-key=6:8:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @transition-magic=-1:193;6:8:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @call-id=-1, @rc-code=193, @op-status=-1, @last-run=1532518989, @last-rc-change
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0802/crmd/94, version=0.20.91)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.91 2
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.92 (null)
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=92
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='ClusterIP']/lrm_rsc_op[@id='ClusterIP_last_0']: @transition-magic=0:0;6:8:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @call-id=24, @rc-code=0, @op-status=0, @exec-time=76
Jul 25 07:43:09 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0802/crmd/95, version=0.20.92)
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: match_graph_event: Action ClusterIP_stop_0 (6) confirmed on dvtvm0802 (rc=0)
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: notice: run_graph: Transition 8 (Complete=3, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-18.bz2): Complete
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: info: do_log: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd
Jul 25 07:43:09 [1624] DVTVM0803.mse.am.mot.com crmd: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd
Jul 25 07:43:09 [1556] DVTVM0803.mse.am.mot.com pacemakerd: info: mcp_cpg_deliver: Ignoring process list sent by peer for local node
Jul 25 07:43:10 [1622] DVTVM0803.mse.am.mot.com attrd: info: write_attribute: Sent update 9 with 2 changes for pingd, id=<n/a>, set=(null)
Jul 25 07:43:10 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/attrd/9)
Jul 25 07:43:10 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.92 2
Jul 25 07:43:10 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.93 (null)
Jul 25 07:43:10 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=93
Jul 25 07:43:10 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: ++ /cib/status/node_state[@id='1']: <transient_attributes id="1"/>
Jul 25 07:43:10 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: ++ <instance_attributes id="status-1">
Jul 25 07:43:10 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: ++ <nvpair id="status-1-pingd" name="pingd" value="2000"/>
Jul 25 07:43:10 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: ++ </instance_attributes>
Jul 25 07:43:10 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: ++ </transient_attributes>
Jul 25 07:43:10 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-pingd']: @value=1000
Jul 25 07:43:10 [1624] DVTVM0803.mse.am.mot.com crmd: info: abort_transition_graph: Transition aborted by transient_attributes.1 'create': Transient attribute change | cib=0.20.93 source=abort_unless_down:343 path=/cib/status/node_state[@id='1'] complete=true
Jul 25 07:43:10 [1624] DVTVM0803.mse.am.mot.com crmd: notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph
Jul 25 07:43:10 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0803/attrd/9, version=0.20.93)
Jul 25 07:43:10 [1622] DVTVM0803.mse.am.mot.com attrd: info: attrd_cib_callback: Update 9 for pingd: OK (0)
Jul 25 07:43:10 [1622] DVTVM0803.mse.am.mot.com attrd: info: attrd_cib_callback: Update 9 for pingd[dvtvm0802]=2000: OK (0)
Jul 25 07:43:10 [1622] DVTVM0803.mse.am.mot.com attrd: info: attrd_cib_callback: Update 9 for pingd[dvtvm0803]=1000: OK (0)
Jul 25 07:43:10 [1623] DVTVM0803.mse.am.mot.com pengine: notice: unpack_config: On loss of CCM Quorum: Ignore
Jul 25 07:43:10 [1623] DVTVM0803.mse.am.mot.com pengine: info: determine_online_status: Node dvtvm0802 is online
Jul 25 07:43:10 [1623] DVTVM0803.mse.am.mot.com pengine: info: determine_online_status: Node dvtvm0803 is online
Jul 25 07:43:10 [1623] DVTVM0803.mse.am.mot.com pengine: info: unpack_node_loop: Node 1 is already processed
Jul 25 07:43:10 [1623] DVTVM0803.mse.am.mot.com pengine: info: unpack_node_loop: Node 2 is already processed
Jul 25 07:43:10 [1623] DVTVM0803.mse.am.mot.com pengine: info: unpack_node_loop: Node 1 is already processed
Jul 25 07:43:10 [1623] DVTVM0803.mse.am.mot.com pengine: info: unpack_node_loop: Node 2 is already processed
Jul 25 07:43:10 [1623] DVTVM0803.mse.am.mot.com pengine: info: common_print: ClusterIP (ocf::heartbeat:IPaddr2): Stopped
Jul 25 07:43:10 [1623] DVTVM0803.mse.am.mot.com pengine: info: clone_print: Clone Set: Connected-clone [Connected]
Jul 25 07:43:10 [1623] DVTVM0803.mse.am.mot.com pengine: info: short_print: Started: [ dvtvm0802 dvtvm0803 ]
Jul 25 07:43:10 [1623] DVTVM0803.mse.am.mot.com pengine: info: common_print: application (ocf::vendor:application): Stopped
Jul 25 07:43:10 [1623] DVTVM0803.mse.am.mot.com pengine: info: RecurringOp: Start recurring monitor (1s) for ClusterIP on dvtvm0802
Jul 25 07:43:10 [1623] DVTVM0803.mse.am.mot.com pengine: info: RecurringOp: Start recurring monitor (1s) for application on dvtvm0802
Jul 25 07:43:10 [1623] DVTVM0803.mse.am.mot.com pengine: notice: LogAction: * Start ClusterIP ( dvtvm0802 )
Jul 25 07:43:10 [1623] DVTVM0803.mse.am.mot.com pengine: info: LogActions: Leave Connected:0 (Started dvtvm0802)
Jul 25 07:43:10 [1623] DVTVM0803.mse.am.mot.com pengine: info: LogActions: Leave Connected:1 (Started dvtvm0803)
Jul 25 07:43:10 [1623] DVTVM0803.mse.am.mot.com pengine: notice: LogAction: * Start application ( dvtvm0802 )
Jul 25 07:43:10 [1623] DVTVM0803.mse.am.mot.com pengine: notice: process_pe_message: Calculated transition 9, saving inputs in /var/lib/pacemaker/pengine/pe-input-19.bz2
Jul 25 07:43:10 [1624] DVTVM0803.mse.am.mot.com crmd: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
Jul 25 07:43:10 [1624] DVTVM0803.mse.am.mot.com crmd: info: do_te_invoke: Processing graph 9 (ref=pe_calc-dc-1532518990-76) derived from /var/lib/pacemaker/pengine/pe-input-19.bz2
Jul 25 07:43:10 [1624] DVTVM0803.mse.am.mot.com crmd: notice: te_rsc_command: Initiating start operation ClusterIP_start_0 on dvtvm0802 | action 4
Jul 25 07:43:10 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.93 2
Jul 25 07:43:10 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.94 (null)
Jul 25 07:43:10 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=94
Jul 25 07:43:10 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='ClusterIP']/lrm_rsc_op[@id='ClusterIP_last_0']: @operation_key=ClusterIP_start_0, @operation=start, @transition-key=4:9:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @transition-magic=-1:193;4:9:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @call-id=-1, @rc-code=193, @op-status=-1, @last-run=1532518990, @last-rc-change=1532518990, @exec-time=0
Jul 25 07:43:10 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0802/crmd/96, version=0.20.94)
Jul 25 07:43:14 [1622] DVTVM0803.mse.am.mot.com attrd: info: attrd_peer_update: Setting pingd[dvtvm0803]: 1000 -> 2000 from dvtvm0803
Jul 25 07:43:14 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.94 2
Jul 25 07:43:14 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.95 (null)
Jul 25 07:43:14 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=95
Jul 25 07:43:14 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='ClusterIP']/lrm_rsc_op[@id='ClusterIP_last_0']: @transition-magic=0:0;4:9:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @call-id=25, @rc-code=0, @op-status=0, @exec-time=4105, @queue-time=1
Jul 25 07:43:14 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0802/crmd/97, version=0.20.95)
Jul 25 07:43:14 [1624] DVTVM0803.mse.am.mot.com crmd: info: match_graph_event: Action ClusterIP_start_0 (4) confirmed on dvtvm0802 (rc=0)
Jul 25 07:43:14 [1624] DVTVM0803.mse.am.mot.com crmd: notice: te_rsc_command: Initiating monitor operation ClusterIP_monitor_1000 on dvtvm0802 | action 5
Jul 25 07:43:14 [1624] DVTVM0803.mse.am.mot.com crmd: notice: te_rsc_command: Initiating start operation edgemanager_start_0 on dvtvm0802 | action 14
Jul 25 07:43:14 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.95 2
Jul 25 07:43:14 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.96 (null)
Jul 25 07:43:14 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=96
Jul 25 07:43:14 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='ClusterIP']/lrm_rsc_op[@id='ClusterIP_monitor_1000']: @crm-debug-origin=do_update_resource, @transition-key=5:9:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @transition-magic=-1:193;5:9:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1532518994, @exec-time=0
Jul 25 07:43:14 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0802/crmd/98, version=0.20.96)
Jul 25 07:43:14 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.96 2
Jul 25 07:43:14 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.97 (null)
Jul 25 07:43:14 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=97
Jul 25 07:43:14 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='edgemanager']/lrm_rsc_op[@id='edgemanager_last_0']: @operation_key=edgemanager_start_0, @operation=start, @transition-key=14:9:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @transition-magic=-1:193;14:9:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @call-id=-1, @rc-code=193, @op-status=-1, @last-run=1532518994, @last-rc-change=1532518994, @exec-time=0
Jul 25 07:43:14 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0802/crmd/99, version=0.20.97)
Jul 25 07:43:14 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.97 2
Jul 25 07:43:14 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.98 (null)
Jul 25 07:43:14 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=98
Jul 25 07:43:14 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='edgemanager']/lrm_rsc_op[@id='edgemanager_last_0']: @transition-magic=0:0;14:9:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @call-id=27, @rc-code=0, @op-status=0, @exec-time=37
Jul 25 07:43:14 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0802/crmd/100, version=0.20.98)
Jul 25 07:43:14 [1624] DVTVM0803.mse.am.mot.com crmd: info: match_graph_event: Action edgemanager_start_0 (14) confirmed on dvtvm0802 (rc=0)
Jul 25 07:43:14 [1624] DVTVM0803.mse.am.mot.com crmd: notice: te_rsc_command: Initiating monitor operation edgemanager_monitor_1000 on dvtvm0802 | action 15
Jul 25 07:43:14 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.98 2
Jul 25 07:43:14 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.99 (null)
Jul 25 07:43:14 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=99
Jul 25 07:43:14 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='edgemanager']/lrm_rsc_op[@id='edgemanager_monitor_1000']: @crm-debug-origin=do_update_resource, @transition-key=15:9:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @transition-magic=-1:193;15:9:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1532518994, @exec-time=0, @queue-time=0
Jul 25 07:43:14 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0802/crmd/101, version=0.20.99)
Jul 25 07:43:14 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.99 2
Jul 25 07:43:14 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.100 (null)
Jul 25 07:43:14 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=100
Jul 25 07:43:14 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='edgemanager']/lrm_rsc_op[@id='edgemanager_monitor_1000']: @transition-magic=0:0;15:9:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @call-id=28, @rc-code=0, @op-status=0, @exec-time=18
Jul 25 07:43:14 [1624] DVTVM0803.mse.am.mot.com crmd: info: match_graph_event: Action edgemanager_monitor_1000 (15) confirmed on dvtvm0802 (rc=0)
Jul 25 07:43:14 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0802/crmd/102, version=0.20.100)
Jul 25 07:43:14 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.100 2
Jul 25 07:43:14 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.101 (null)
Jul 25 07:43:14 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=101
Jul 25 07:43:14 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='ClusterIP']/lrm_rsc_op[@id='ClusterIP_monitor_1000']: @transition-magic=0:0;5:9:0:1cec18ac-2aae-4c5c-8513-d6851bc10191, @call-id=26, @rc-code=0, @op-status=0, @exec-time=71
Jul 25 07:43:14 [1624] DVTVM0803.mse.am.mot.com crmd: info: match_graph_event: Action ClusterIP_monitor_1000 (5) confirmed on dvtvm0802 (rc=0)
Jul 25 07:43:14 [1624] DVTVM0803.mse.am.mot.com crmd: notice: run_graph: Transition 9 (Complete=4, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-19.bz2): Complete
Jul 25 07:43:14 [1624] DVTVM0803.mse.am.mot.com crmd: info: do_log: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd
Jul 25 07:43:14 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0802/crmd/103, version=0.20.101)
Jul 25 07:43:14 [1624] DVTVM0803.mse.am.mot.com crmd: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd
Jul 25 07:43:15 [1622] DVTVM0803.mse.am.mot.com attrd: info: write_attribute: Sent update 10 with 2 changes for pingd, id=<n/a>, set=(null)
Jul 25 07:43:15 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/attrd/10)
Jul 25 07:43:15 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.20.101 2
Jul 25 07:43:15 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.20.102 (null)
Jul 25 07:43:15 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=102
Jul 25 07:43:15 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-pingd']: @value=2000
Jul 25 07:43:15 [1624] DVTVM0803.mse.am.mot.com crmd: info: abort_transition_graph: Transition aborted by status-2-pingd doing modify pingd=2000: Transient attribute change | cib=0.20.102 source=abort_unless_down:343 path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-pingd'] complete=true
Jul 25 07:43:15 [1624] DVTVM0803.mse.am.mot.com crmd: notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph
Jul 25 07:43:15 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0803/attrd/10, version=0.20.102)
Jul 25 07:43:15 [1622] DVTVM0803.mse.am.mot.com attrd: info: attrd_cib_callback: Update 10 for pingd: OK (0)
Jul 25 07:43:15 [1622] DVTVM0803.mse.am.mot.com attrd: info: attrd_cib_callback: Update 10 for pingd[dvtvm0802]=2000: OK (0)
Jul 25 07:43:15 [1622] DVTVM0803.mse.am.mot.com attrd: info: attrd_cib_callback: Update 10 for pingd[dvtvm0803]=2000: OK (0)
Jul 25 07:43:15 [1623] DVTVM0803.mse.am.mot.com pengine: notice: unpack_config: On loss of CCM Quorum: Ignore
Jul 25 07:43:15 [1623] DVTVM0803.mse.am.mot.com pengine: info: determine_online_status: Node dvtvm0802 is online
Jul 25 07:43:15 [1623] DVTVM0803.mse.am.mot.com pengine: info: determine_online_status: Node dvtvm0803 is online
Jul 25 07:43:15 [1623] DVTVM0803.mse.am.mot.com pengine: info: unpack_node_loop: Node 1 is already processed
Jul 25 07:43:15 [1623] DVTVM0803.mse.am.mot.com pengine: info: unpack_node_loop: Node 2 is already processed
Jul 25 07:43:15 [1623] DVTVM0803.mse.am.mot.com pengine: info: unpack_node_loop: Node 1 is already processed
Jul 25 07:43:15 [1623] DVTVM0803.mse.am.mot.com pengine: info: unpack_node_loop: Node 2 is already processed
Jul 25 07:43:15 [1623] DVTVM0803.mse.am.mot.com pengine: info: common_print: ClusterIP (ocf::heartbeat:IPaddr2): Started dvtvm0802
Jul 25 07:43:15 [1623] DVTVM0803.mse.am.mot.com pengine: info: clone_print: Clone Set: Connected-clone [Connected]
Jul 25 07:43:15 [1623] DVTVM0803.mse.am.mot.com pengine: info: short_print: Started: [ dvtvm0802 dvtvm0803 ]
Jul 25 07:43:15 [1623] DVTVM0803.mse.am.mot.com pengine: info: common_print: application (ocf::vendor:application): Started dvtvm0802
Jul 25 07:43:15 [1623] DVTVM0803.mse.am.mot.com pengine: info: LogActions: Leave ClusterIP (Started dvtvm0802)
Jul 25 07:43:15 [1623] DVTVM0803.mse.am.mot.com pengine: info: LogActions: Leave Connected:0 (Started dvtvm0802)
Jul 25 07:43:15 [1623] DVTVM0803.mse.am.mot.com pengine: info: LogActions: Leave Connected:1 (Started dvtvm0803)
Jul 25 07:43:15 [1623] DVTVM0803.mse.am.mot.com pengine: info: LogActions: Leave application (Started dvtvm0802)
Jul 25 07:43:15 [1623] DVTVM0803.mse.am.mot.com pengine: notice: process_pe_message: Calculated transition 10, saving inputs in /var/lib/pacemaker/pengine/pe-input-20.bz2
Jul 25 07:43:15 [1624] DVTVM0803.mse.am.mot.com crmd: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
Jul 25 07:43:15 [1624] DVTVM0803.mse.am.mot.com crmd: info: do_te_invoke: Processing graph 10 (ref=pe_calc-dc-1532518995-81) derived from /var/lib/pacemaker/pengine/pe-input-20.bz2
Jul 25 07:43:15 [1624] DVTVM0803.mse.am.mot.com crmd: notice: run_graph: Transition 10 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-20.bz2): Complete
Jul 25 07:43:15 [1624] DVTVM0803.mse.am.mot.com crmd: info: do_log: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd
Jul 25 07:43:15 [1624] DVTVM0803.mse.am.mot.com crmd: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd
Jul 25 07:43:20 [1618] DVTVM0803.mse.am.mot.com cib: info: cib_process_ping: Reporting our current digest to dvtvm0803: 6aaa74aa220cce54790a731cd758f8e3 for 0.20.102 (0x55d416545f90 0)
On 7/24/18, 2:09 PM, "Users on behalf of Andrei Borzenkov"
<users-bounces at clusterlabs.org on behalf of arvidjaar at gmail.com> wrote:
>24.07.2018 20:59, O'Donovan, Garret ?????:
>> Hello, and thank you for adding me to the list.
>>
>> We are using Pacemaker in a two-node hot-warm redundancy configuration.
>> Both nodes run ocf:pacemaker:ping (cloned) to monitor a ping group of
>>devices. The nodes share a virtual IP using ocf:heartbeat:IPAddr2. Our
>>applications run in either Primary mode (does all the work and sends
>>status updates to its peer), or Standby mode (receives status updates
>>and is ready to take over if current primary fails). We have
>>constraints set up so that IPAddr2 fails over on any failure Pacemaker
>>detects (including ping group failure), and the applications follow it.
>>
>> This works great for most use cases, but we have issues in one test
>>case where we disconnect the node in standby (by yanking the eth cable)
>>for about 30 seconds to a minute, and then reconnect it. The problem is
>>that Pacemaker seems to put the Primary into standby for a very short
>>time while reconnecting the two nodes, and then making it primary again.
>>
>> Is there any way to prevent Pacemaker from doing this? Detailed config
>>info and log file snippet are below.
>>
>> - Regards
>> - Garret O'Donovan
>>
>>
>> PLATFORM
>> This is all running this on CentOS 7
>>(centos-release-7-4.1708.el7.centos.x86_64) on VM's (VMware ESXi 5.5).
>>The two nodes are hosted on physically different servers.
>>
>> VERSION INFO
>> corosync-2.4.3-2.el7_5.1.x86_64.rpm
>> pacemaker-1.1.18-11.el7_5.2.x86_64.rpm
>> pcs-0.9.162-5.el7.centos.1.x86_64.rpm
>> resource-agents-3.9.5-124.el7.x86_64.rpm
>>
>> PACEMAKER CONFIGURATION
>> [root at DVTVM0302 ~]# pcs config show
>> Cluster Name: vendor1
>> Corosync Nodes:
>> dvtvm0302 dvtvm0303
>> Pacemaker Nodes:
>> dvtvm0302 dvtvm0303
>>
>> Resources:
>> Resource: ClusterIP (class=ocf provider=heartbeat type=IPaddr2)
>> Attributes: cidr_netmask=24 ip=10.144.101.210 nic=eth0
>> Operations: monitor interval=1s (ClusterIP-monitor-interval-1s)
>> start interval=0s timeout=20s
>>(ClusterIP-start-interval-0s)
>> stop interval=0s timeout=20s (ClusterIP-stop-interval-0s)
>> Resource: application (class=ocf provider=vendor type=application)
>> Operations: migrate_from interval=0s timeout=20
>>(application-migrate_from-interval-0s)
>> migrate_to interval=0s timeout=20
>>(application-migrate_to-interval-0s)
>> monitor interval=1s (application-monitor-interval-1s)
>> reload interval=0s timeout=20
>>(application-reload-interval-0s)
>> start interval=0s timeout=20
>>(application-start-interval-0s)
>> stop interval=0s timeout=20 (application-stop-interval-0s)
>> Clone: Connected-clone
>> Meta Attrs: interleave=true
>> Resource: Connected (class=ocf provider=pacemaker type=ping)
>> Attributes: attempts=2 dampen=1s debug=true host_list="10.10.24.5
>>10.10.24.18" multiplier=1000
>> Operations: monitor interval=3s timeout=10
>>(Connected-monitor-interval-3s)
>> start interval=0 timeout=3 (Connected-start-interval-0)
>> stop interval=0s timeout=20 (Connected-stop-interval-0s)
>>
>> Stonith Devices:
>
>You are risking real split brain here.
>
>> Fencing Levels:
>>
>> Location Constraints:
>> Resource: ClusterIP
>> Constraint: location-ClusterIP
>> Rule: boolean-op=or score=-INFINITY (id:location-ClusterIP-rule)
>> Expression: pingd lt 500 (id:location-ClusterIP-rule-expr)
>> Expression: not_defined pingd
>>(id:location-ClusterIP-rule-expr-1)
>> Ordering Constraints:
>> start ClusterIP then start application (kind:Mandatory)
>> Colocation Constraints:
>> ClusterIP with application (score:INFINITY)
>> Ticket Constraints:
>>
>> Alerts:
>> No alerts defined
>>
>> Resources Defaults:
>> migration-threshold: 1
>> failure-timeout: 5s
>> cluster-recheck-interval: 5s
>> resource-stickiness: INFINITY
>> Operations Defaults:
>> No defaults set
>>
>> Cluster Properties:
>> cluster-infrastructure: corosync
>> cluster-name: vendor1
>> dc-version: 1.1.18-11.el7_5.2-2b07d5c5a9
>> have-watchdog: false
>> no-quorum-policy: ignore
>> stonith-enabled: false
>>
>> Quorum:
>> Options:
>>
>> COROSYNC CONFIG FILE:
>>
>> [root at DVTVM0302 corosync]# cat corosync.conf
>> totem {
>> version: 2
>> cluster_name: vendor1
>> secauth: off
>> transport: udpu
>> }
>>
>> nodelist {
>> node {
>> ring0_addr: dvtvm0302
>> nodeid: 1
>> }
>>
>> node {
>> ring0_addr: dvtvm0303
>> nodeid: 2
>> }
>> }
>>
>> quorum {
>> provider: corosync_votequorum
>> two_node: 1
>> }
>>
>> logging {
>> to_logfile: yes
>> logfile: /var/log/cluster/corosync.log
>> to_syslog: yes
>> }
>>
>>
>> LOGFILE:
>> /var/log/cluster/corosync.log
>>
>
>
More information about the Users
mailing list