[ClusterLabs] Trying to prevent instantaneous Failover / Failback at standby reconnect
O'Donovan, Garret
Garret.ODonovan at arris.com
Tue Jul 24 13:59:48 EDT 2018
Hello, and thank you for adding me to the list.
We are using Pacemaker in a two-node hot-warm redundancy configuration. Both nodes run ocf:pacemaker:ping (cloned) to monitor a ping group of devices. The nodes share a virtual IP using ocf:heartbeat:IPAddr2. Our applications run in either Primary mode (does all the work and sends status updates to its peer), or Standby mode (receives status updates and is ready to take over if current primary fails). We have constraints set up so that IPAddr2 fails over on any failure Pacemaker detects (including ping group failure), and the applications follow it.
This works great for most use cases, but we have issues in one test case where we disconnect the node in standby (by yanking the eth cable) for about 30 seconds to a minute, and then reconnect it. The problem is that Pacemaker seems to put the Primary into standby for a very short time while reconnecting the two nodes, and then making it primary again.
Is there any way to prevent Pacemaker from doing this? Detailed config info and log file snippet are below.
- Regards
- Garret O'Donovan
PLATFORM
This is all running this on CentOS 7 (centos-release-7-4.1708.el7.centos.x86_64) on VM’s (VMware ESXi 5.5). The two nodes are hosted on physically different servers.
VERSION INFO
corosync-2.4.3-2.el7_5.1.x86_64.rpm
pacemaker-1.1.18-11.el7_5.2.x86_64.rpm
pcs-0.9.162-5.el7.centos.1.x86_64.rpm
resource-agents-3.9.5-124.el7.x86_64.rpm
PACEMAKER CONFIGURATION
[root at DVTVM0302 ~]# pcs config show
Cluster Name: vendor1
Corosync Nodes:
dvtvm0302 dvtvm0303
Pacemaker Nodes:
dvtvm0302 dvtvm0303
Resources:
Resource: ClusterIP (class=ocf provider=heartbeat type=IPaddr2)
Attributes: cidr_netmask=24 ip=10.144.101.210 nic=eth0
Operations: monitor interval=1s (ClusterIP-monitor-interval-1s)
start interval=0s timeout=20s (ClusterIP-start-interval-0s)
stop interval=0s timeout=20s (ClusterIP-stop-interval-0s)
Resource: application (class=ocf provider=vendor type=application)
Operations: migrate_from interval=0s timeout=20 (application-migrate_from-interval-0s)
migrate_to interval=0s timeout=20 (application-migrate_to-interval-0s)
monitor interval=1s (application-monitor-interval-1s)
reload interval=0s timeout=20 (application-reload-interval-0s)
start interval=0s timeout=20 (application-start-interval-0s)
stop interval=0s timeout=20 (application-stop-interval-0s)
Clone: Connected-clone
Meta Attrs: interleave=true
Resource: Connected (class=ocf provider=pacemaker type=ping)
Attributes: attempts=2 dampen=1s debug=true host_list="10.10.24.5 10.10.24.18" multiplier=1000
Operations: monitor interval=3s timeout=10 (Connected-monitor-interval-3s)
start interval=0 timeout=3 (Connected-start-interval-0)
stop interval=0s timeout=20 (Connected-stop-interval-0s)
Stonith Devices:
Fencing Levels:
Location Constraints:
Resource: ClusterIP
Constraint: location-ClusterIP
Rule: boolean-op=or score=-INFINITY (id:location-ClusterIP-rule)
Expression: pingd lt 500 (id:location-ClusterIP-rule-expr)
Expression: not_defined pingd (id:location-ClusterIP-rule-expr-1)
Ordering Constraints:
start ClusterIP then start application (kind:Mandatory)
Colocation Constraints:
ClusterIP with application (score:INFINITY)
Ticket Constraints:
Alerts:
No alerts defined
Resources Defaults:
migration-threshold: 1
failure-timeout: 5s
cluster-recheck-interval: 5s
resource-stickiness: INFINITY
Operations Defaults:
No defaults set
Cluster Properties:
cluster-infrastructure: corosync
cluster-name: vendor1
dc-version: 1.1.18-11.el7_5.2-2b07d5c5a9
have-watchdog: false
no-quorum-policy: ignore
stonith-enabled: false
Quorum:
Options:
COROSYNC CONFIG FILE:
[root at DVTVM0302 corosync]# cat corosync.conf
totem {
version: 2
cluster_name: vendor1
secauth: off
transport: udpu
}
nodelist {
node {
ring0_addr: dvtvm0302
nodeid: 1
}
node {
ring0_addr: dvtvm0303
nodeid: 2
}
}
quorum {
provider: corosync_votequorum
two_node: 1
}
logging {
to_logfile: yes
logfile: /var/log/cluster/corosync.log
to_syslog: yes
}
LOGFILE:
/var/log/cluster/corosync.log
Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com crmd: info: pcmk_cpg_membership: Node 2 joined group crmd (counter=5.0)
Jul 20 07:46:49 [1564] DVTVM0302.mse.am.mot.com stonith-ng: info: pcmk_cpg_membership: Node 2 joined group stonith-ng (counter=5.0)
Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com crmd: info: pcmk_cpg_membership: Node 1 still member of group crmd (peer=dvtvm0302, counter=5.0)
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: pcmk_cpg_membership: Node 2 joined group cib (counter=5.0)
Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com crmd: info: pcmk_cpg_membership: Node 2 still member of group crmd (peer=dvtvm0303, counter=5.1)
Jul 20 07:46:49 [1564] DVTVM0302.mse.am.mot.com stonith-ng: info: pcmk_cpg_membership: Node 1 still member of group stonith-ng (peer=dvtvm0302, counter=5.0)
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: pcmk_cpg_membership: Node 1 still member of group cib (peer=dvtvm0302, counter=5.0)
Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com crmd: info: crm_update_peer_proc: pcmk_cpg_membership: Node dvtvm0303[2] - corosync-cpg is now online
Jul 20 07:46:49 [1385] DVTVM0302.mse.am.mot.com pacemakerd: info: pcmk_cpg_membership: Node 2 joined group pacemakerd (counter=5.0)
Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com crmd: info: peer_update_callback: Client dvtvm0303/peer now has status [online] (DC=true, changed=4000000)
Jul 20 07:46:49 [1385] DVTVM0302.mse.am.mot.com pacemakerd: info: pcmk_cpg_membership: Node 1 still member of group pacemakerd (peer=dvtvm0302, counter=5.0)
Jul 20 07:46:49 [1385] DVTVM0302.mse.am.mot.com pacemakerd: info: pcmk_cpg_membership: Node 2 still member of group pacemakerd (peer=dvtvm0303, counter=5.1)
Jul 20 07:46:49 [1385] DVTVM0302.mse.am.mot.com pacemakerd: info: crm_update_peer_proc: pcmk_cpg_membership: Node dvtvm0303[2] - corosync-cpg is now online
Jul 20 07:46:49 [1385] DVTVM0302.mse.am.mot.com pacemakerd: error: pcmk_cpg_membership: Node dvtvm0303[2] appears to be online even though we think it is dead
Jul 20 07:46:49 [1385] DVTVM0302.mse.am.mot.com pacemakerd: notice: crm_update_peer_state_iter: Node dvtvm0303 state is now member | nodeid=2 previous=lost source=pcmk_cpg_membership
Jul 20 07:46:49 [1385] DVTVM0302.mse.am.mot.com pacemakerd: info: crm_cs_flush: Sent 0 CPG messages (1 remaining, last=11): Try again (6)
Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com crmd: error: pcmk_cpg_membership: Node dvtvm0303[2] appears to be online even though we think it is dead
Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com crmd: notice: crm_update_peer_state_iter: Node dvtvm0303 state is now member | nodeid=2 previous=lost source=pcmk_cpg_membership
Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com crmd: info: peer_update_callback: Cluster node dvtvm0303 is now member (was lost)
Jul 20 07:46:49 [1566] DVTVM0302.mse.am.mot.com attrd: info: pcmk_cpg_membership: Node 2 joined group attrd (counter=5.0)
Jul 20 07:46:49 [1566] DVTVM0302.mse.am.mot.com attrd: info: pcmk_cpg_membership: Node 1 still member of group attrd (peer=dvtvm0302, counter=5.0)
Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com crmd: info: pcmk_quorum_notification: Quorum retained | membership=144 members=2
Jul 20 07:46:49 [1385] DVTVM0302.mse.am.mot.com pacemakerd: info: pcmk_quorum_notification: Quorum retained | membership=144 members=2
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: crm_get_peer: Created entry 603f53c1-5c06-4423-b6f0-420c6194303d/0x56163cadf350 for node dvtvm0303/2 (2 total)
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: crm_get_peer: Node 2 is now known as dvtvm0303
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: crm_get_peer: Node 2 has uuid 2
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: pcmk_cpg_membership: Node 2 still member of group cib (peer=dvtvm0303, counter=5.1)
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: crm_update_peer_proc: pcmk_cpg_membership: Node dvtvm0303[2] - corosync-cpg is now online
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: notice: crm_update_peer_state_iter: Node dvtvm0303 state is now member | nodeid=2 previous=unknown source=crm_update_peer_proc
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.29.54 2
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.29.55 (null)
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=55
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']: @crm-debug-origin=peer_update_callback
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0303/crmd/108, version=0.29.55)
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0303/crmd/109, version=0.29.55)
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=dvtvm0303/crmd/112, version=0.29.55)
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.29.55 2
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.29.56 (null)
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=56
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='2']: @in_ccm=true, @crmd=online, @crm-debug-origin=post_cache_update
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']: @crm-debug-origin=post_cache_update
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0303/crmd/113, version=0.29.56)
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/82)
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/83)
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.29.56 2
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.29.57 (null)
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=57
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='2']: @crm-debug-origin=peer_update_callback
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0302/crmd/82, version=0.29.57)
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0302/crmd/83, version=0.29.57)
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/86)
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/87)
Jul 20 07:46:49 [1564] DVTVM0302.mse.am.mot.com stonith-ng: info: crm_get_peer: Created entry 1fd3c98b-2a62-4f07-9343-0850d9a68e83/0x55d0123ef950 for node dvtvm0303/2 (2 total)
Jul 20 07:46:49 [1564] DVTVM0302.mse.am.mot.com stonith-ng: info: crm_get_peer: Node 2 is now known as dvtvm0303
Jul 20 07:46:49 [1564] DVTVM0302.mse.am.mot.com stonith-ng: info: crm_get_peer: Node 2 has uuid 2
Jul 20 07:46:49 [1564] DVTVM0302.mse.am.mot.com stonith-ng: info: pcmk_cpg_membership: Node 2 still member of group stonith-ng (peer=dvtvm0303, counter=5.1)
Jul 20 07:46:49 [1564] DVTVM0302.mse.am.mot.com stonith-ng: info: crm_update_peer_proc: pcmk_cpg_membership: Node dvtvm0303[2] - corosync-cpg is now online
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=dvtvm0302/crmd/86, version=0.29.57)
Jul 20 07:46:49 [1566] DVTVM0302.mse.am.mot.com attrd: info: crm_get_peer: Created entry 523ee210-80c3-4ba0-a130-989aefa5d05f/0x55b7f9daa8e0 for node dvtvm0303/2 (2 total)
Jul 20 07:46:49 [1566] DVTVM0302.mse.am.mot.com attrd: info: crm_get_peer: Node 2 is now known as dvtvm0303
Jul 20 07:46:49 [1566] DVTVM0302.mse.am.mot.com attrd: info: crm_get_peer: Node 2 has uuid 2
Jul 20 07:46:49 [1566] DVTVM0302.mse.am.mot.com attrd: info: pcmk_cpg_membership: Node 2 still member of group attrd (peer=dvtvm0303, counter=5.1)
Jul 20 07:46:49 [1566] DVTVM0302.mse.am.mot.com attrd: info: crm_update_peer_proc: pcmk_cpg_membership: Node dvtvm0303[2] - corosync-cpg is now online
Jul 20 07:46:49 [1566] DVTVM0302.mse.am.mot.com attrd: notice: crm_update_peer_state_iter: Node dvtvm0303 state is now member | nodeid=2 previous=unknown source=crm_update_peer_proc
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.29.57 2
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.29.58 (null)
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=58
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='2']: @crm-debug-origin=post_cache_update
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0302/crmd/87, version=0.29.58)
Jul 20 07:46:49 [1566] DVTVM0302.mse.am.mot.com attrd: notice: attrd_peer_message: Detected another attribute writer: dvtvm0303
Jul 20 07:46:49 [1566] DVTVM0302.mse.am.mot.com attrd: info: attrd_peer_message: Processing sync-response from dvtvm0303
Jul 20 07:46:49 [1566] DVTVM0302.mse.am.mot.com attrd: info: attrd_peer_update: Setting #attrd-protocol[dvtvm0303]: (null) -> 2 from dvtvm0303
Jul 20 07:46:49 [1566] DVTVM0302.mse.am.mot.com attrd: info: attrd_peer_update: Setting pingd[dvtvm0303]: (null) -> 0 from dvtvm0303
Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com crmd: info: register_fsa_error_adv: Resetting the current action list
Jul 20 07:46:49 [1564] DVTVM0302.mse.am.mot.com stonith-ng: notice: crm_update_peer_state_iter: Node dvtvm0303 state is now member | nodeid=2 previous=unknown source=crm_update_peer_proc
Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com crmd: warning: crmd_ha_msg_filter: Another DC detected: dvtvm0303 (op=noop)
Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com crmd: notice: do_state_transition: State transition S_IDLE -> S_ELECTION | input=I_ELECTION cause=C_FSA_INTERNAL origin=crmd_ha_msg_filter
Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com crmd: info: update_dc: Unset DC. Was dvtvm0302
Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com crmd: info: election_count_vote: Election 6 (owner: 2) lost: vote from dvtvm0303 (Uptime)
Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com crmd: notice: do_state_transition: State transition S_ELECTION -> S_RELEASE_DC | input=I_RELEASE_DC cause=C_FSA_INTERNAL origin=do_election_count_vote
Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com crmd: info: do_dc_release: DC role released
Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com crmd: info: pe_ipc_destroy: Connection to the Policy Engine released
Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com crmd: info: do_te_control: Transitioner is now inactive
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_readwrite: We are now in R/O mode
Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com crmd: info: do_log: Input I_RELEASE_SUCCESS received in state S_RELEASE_DC from do_dc_release
Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com crmd: info: do_state_transition: State transition S_RELEASE_DC -> S_PENDING | input=I_RELEASE_SUCCESS cause=C_FSA_INTERNAL origin=do_dc_release
Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com crmd: info: do_log: Input I_PENDING received in state S_PENDING from do_election_count_vote
Jul 20 07:46:49 [1566] DVTVM0302.mse.am.mot.com attrd: info: election_count_vote: Election 4 (owner: 2) lost: vote from dvtvm0303 (Uptime)
Jul 20 07:46:49 [1566] DVTVM0302.mse.am.mot.com attrd: info: election_count_vote: Election 5 (owner: 2) lost: vote from dvtvm0303 (Uptime)
Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com crmd: info: election_count_vote: Election 7 (owner: 2) lost: vote from dvtvm0303 (Uptime)
Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com crmd: info: do_log: Input I_PENDING received in state S_PENDING from do_election_count_vote
Jul 20 07:46:49 [1566] DVTVM0302.mse.am.mot.com attrd: info: attrd_peer_message: Processing sync-response from dvtvm0303
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section cib: OK (rc=0, origin=dvtvm0303/crmd/115, version=0.29.58)
Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com crmd: info: update_dc: Set DC to dvtvm0303 (3.0.14)
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section crm_config: OK (rc=0, origin=dvtvm0303/crmd/117, version=0.29.58)
Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com crmd: info: election_count_vote: Election 8 (owner: 2) lost: vote from dvtvm0303 (Uptime)
Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com crmd: info: update_dc: Unset DC. Was dvtvm0303
Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com crmd: info: do_log: Input I_PENDING received in state S_PENDING from do_election_count_vote
Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com crmd: info: update_dc: Set DC to dvtvm0303 (3.0.14)
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section crm_config: OK (rc=0, origin=dvtvm0303/crmd/119, version=0.29.58)
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section crm_config: OK (rc=0, origin=dvtvm0303/crmd/121, version=0.29.58)
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_replace: Digest matched on replace from dvtvm0303: 741acf29cc9726f86772f735a16a723a
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_replace: Replaced 0.29.58 with 0.29.64 from dvtvm0303
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.29.58 2
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.29.64 (null)
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: -- /cib/status/node_state[@id='1']/transient_attributes[@id='1']
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=64, @dc-uuid=2
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='2']: @join=member
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='ClusterIP']/lrm_rsc_op[@id='ClusterIP_last_0']: @operation_key=ClusterIP_stop_0, @operation=stop, @crm-debug-origin=do_update_resource, @transition-key=3:7:0:4ad59ad3-3a70-4282-a52c-1f14cce6b076, @transition-magic=0:0;3:7:0:4ad59ad3-3a70-4282-a52c-1f14cce6b076, @call-id=18, @rc-code=0, @last-run=1532087185, @last-rc-change=1532087185, @exec-t
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-pingd']: @value=0
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']: @join=down
Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com crmd: info: action_synced_wait: Managed application_meta-data_0 process 9340 exited with rc=4
Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com crmd: error: lrmd_api_get_metadata: Failed to receive meta-data for ocf:vendor:application
Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com crmd: warning: build_operation_update: Failed to get metadata for application (ocf:vendor:application)
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_replace_notify: Replaced: 0.29.58 -> 0.29.64 from dvtvm0303
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Completed cib_replace operation for section 'all': OK (rc=0, origin=dvtvm0303/crmd/125, version=0.29.64)
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=dvtvm0303/crmd/126, version=0.29.64)
Jul 20 07:46:49 [1566] DVTVM0302.mse.am.mot.com attrd: notice: attrd_cib_replaced_cb: Updating all attributes after cib_refresh_notify event
Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com crmd: info: action_synced_wait: Managed application_meta-data_0 process 9341 exited with rc=4
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=dvtvm0303/crmd/127, version=0.29.64)
Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com crmd: error: lrmd_api_get_metadata: Failed to receive meta-data for ocf:vendor:application
Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com crmd: warning: build_operation_update: Failed to get metadata for application (ocf:vendor:application)
Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com crmd: info: do_log: Input I_NOT_DC received in state S_PENDING from do_cl_join_finalize_respond
Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com crmd: notice: do_state_transition: State transition S_PENDING -> S_NOT_DC | input=I_NOT_DC cause=C_HA_MESSAGE origin=do_cl_join_finalize_respond
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.29.64 2
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.29.65 (null)
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: -- /cib/status/node_state[@id='2']/lrm[@id='2']
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=65
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Completed cib_delete operation for section //node_state[@uname='dvtvm0303']/lrm: OK (rc=0, origin=dvtvm0303/crmd/128, version=0.29.65)
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.29.65 2
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.29.66 (null)
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=66
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_lrm_query_internal
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: ++ /cib/status/node_state[@id='2']: <lrm id="2"/>
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_resources>
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_resource id="Connected" type="ping" class="ocf" provider="pacemaker">
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_rsc_op id="Connected_last_0" operation_key="Connected_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="8:0:0:4ad59ad3-3a70-4282-a52c-1f14cce6b076" transition-magic="0:0;8:0:0:4ad59ad3-3a70-4282-a52c-1f14cce6b076" exit-reason="" on_node="dvtvm0303" call-id="15" rc-code="0" op-status="0" interval="0" last-run="1532086605"
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_rsc_op id="Connected_monitor_3000" operation_key="Connected_monitor_3000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="9:0:0:4ad59ad3-3a70-4282-a52c-1f14cce6b076" transition-magic="0:0;9:0:0:4ad59ad3-3a70-4282-a52c-1f14cce6b076" exit-reason="" on_node="dvtvm0303" call-id="16" rc-code="0" op-status="0" interval="3000" last-r
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: ++ </lrm_resource>
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_resource id="ClusterIP" type="IPaddr2" class="ocf" provider="heartbeat">
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_rsc_op id="ClusterIP_last_0" operation_key="ClusterIP_stop_0" operation="stop" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="3:7:0:4ad59ad3-3a70-4282-a52c-1f14cce6b076" transition-magic="0:0;3:7:0:4ad59ad3-3a70-4282-a52c-1f14cce6b076" exit-reason="" on_node="dvtvm0303" call-id="18" rc-code="0" op-status="0" interval="0" last-run="1532087185" la
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: ++ </lrm_resource>
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_resource id="application" type="application" class="ocf" provider="vendor">
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_rsc_op id="application_last_0" operation_key="application_monitor_0" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="7:0:7:4ad59ad3-3a70-4282-a52c-1f14cce6b076" transition-magic="0:7;7:0:7:4ad59ad3-3a70-4282-a52c-1f14cce6b076" exit-reason="" on_node="dvtvm0303" call-id="14" rc-code="7" op-status="0" interval="0" last-run="1532
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: ++ </lrm_resource>
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: ++ </lrm_resources>
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: ++ </lrm>
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0303/crmd/129, version=0.29.66)
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.29.66 2
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.29.67 (null)
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: -- /cib/status/node_state[@id='1']/lrm[@id='1']
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=67
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Completed cib_delete operation for section //node_state[@uname='dvtvm0302']/lrm: OK (rc=0, origin=dvtvm0303/crmd/130, version=0.29.67)
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.29.67 2
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.29.68 (null)
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=68
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_lrm_query_internal, @join=member
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: ++ /cib/status/node_state[@id='1']: <lrm id="1"/>
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_resources>
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_resource id="Connected" type="ping" class="ocf" provider="pacemaker">
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_rsc_op id="Connected_last_0" operation_key="Connected_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="10:0:0:4ad59ad3-3a70-4282-a52c-1f14cce6b076" transition-magic="0:0;10:0:0:4ad59ad3-3a70-4282-a52c-1f14cce6b076" exit-reason="" on_node="dvtvm0302" call-id="15" rc-code="0" op-status="0" interval="0" last-run="1532086605
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_rsc_op id="Connected_monitor_3000" operation_key="Connected_monitor_3000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="11:0:0:4ad59ad3-3a70-4282-a52c-1f14cce6b076" transition-magic="0:0;11:0:0:4ad59ad3-3a70-4282-a52c-1f14cce6b076" exit-reason="" on_node="dvtvm0302" call-id="16" rc-code="0" op-status="0" interval="3000" last
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: ++ </lrm_resource>
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_resource id="ClusterIP" type="IPaddr2" class="ocf" provider="heartbeat">
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_rsc_op id="ClusterIP_last_0" operation_key="ClusterIP_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="4:1:0:4ad59ad3-3a70-4282-a52c-1f14cce6b076" transition-magic="0:0;4:1:0:4ad59ad3-3a70-4282-a52c-1f14cce6b076" exit-reason="" on_node="dvtvm0302" call-id="17" rc-code="0" op-status="0" interval="0" last-run="1532086609"
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_rsc_op id="ClusterIP_monitor_1000" operation_key="ClusterIP_monitor_1000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="5:1:0:4ad59ad3-3a70-4282-a52c-1f14cce6b076" transition-magic="0:0;5:1:0:4ad59ad3-3a70-4282-a52c-1f14cce6b076" exit-reason="" on_node="dvtvm0302" call-id="18" rc-code="0" op-status="0" interval="1000" last-r
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: ++ </lrm_resource>
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_resource id="application" type="application" class="ocf" provider="vendor">
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_rsc_op id="application_last_0" operation_key="application_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="14:1:0:4ad59ad3-3a70-4282-a52c-1f14cce6b076" transition-magic="0:0;14:1:0:4ad59ad3-3a70-4282-a52c-1f14cce6b076" exit-reason="" on_node="dvtvm0302" call-id="19" rc-code="0" op-status="0" interval="0" last-run="153208
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: ++ <lrm_rsc_op id="application_monitor_1000" operation_key="application_monitor_1000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="15:1:0:4ad59ad3-3a70-4282-a52c-1f14cce6b076" transition-magic="0:0;15:1:0:4ad59ad3-3a70-4282-a52c-1f14cce6b076" exit-reason="" on_node="dvtvm0302" call-id="20" rc-code="0" op-status="0" interval="1000"
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: ++ </lrm_resource>
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: ++ </lrm_resources>
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: ++ </lrm>
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0303/crmd/131, version=0.29.68)
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=dvtvm0303/crmd/134, version=0.29.68)
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.29.68 2
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.29.69 (null)
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=69
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0303/crmd/135, version=0.29.69)
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section cib: OK (rc=0, origin=dvtvm0303/crmd/136, version=0.29.69)
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_file_backup: Archived previous version as /var/lib/pacemaker/cib/cib-49.raw
Jul 20 07:46:49 [1565] DVTVM0302.mse.am.mot.com lrmd: info: cancel_recurring_action: Cancelling ocf operation application_monitor_1000
Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com crmd: info: do_lrm_rsc_op: Performing key=15:8:0:4ad59ad3-3a70-4282-a52c-1f14cce6b076 op=application_stop_0
Jul 20 07:46:49 [1565] DVTVM0302.mse.am.mot.com lrmd: info: log_execute: executing - rsc:application action:stop call_id:22
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/92)
Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com crmd: info: process_lrm_event: Result of monitor operation for application on dvtvm0302: Cancelled | call=20 key=application_monitor_1000 confirmed=true
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.29.69 2
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.29.70 (null)
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=70
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_update_resource
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='application']/lrm_rsc_op[@id='application_last_0']: @operation_key=application_stop_0, @operation=stop, @crm-debug-origin=do_update_resource, @transition-key=15:8:0:4ad59ad3-3a70-4282-a52c-1f14cce6b076, @transition-magic=-1:193;15:8:0:4ad59ad3-3a70-4282-a52c-1f14cce6b076, @call-id=-1, @rc-code=193, @op-status=-1, @last-run=1532087209, @last-r
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0302/crmd/92, version=0.29.70)
Jul 20 07:46:49 application(application)[9364]: INFO: Stopping APP - setting to Standby
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_file_write_with_digest: Wrote version 0.29.0 of the CIB to disk (digest: 67b46b2a11db4ce9d1a5fb649ae91bd3)
Jul 20 07:46:49 [1565] DVTVM0302.mse.am.mot.com lrmd: notice: operation_finished: application_stop_0:9364:stderr [ /usr/local/vendor/bin/PrimaryMode.sh: line 27: $Filename: ambiguous redirect ]
Jul 20 07:46:49 [1565] DVTVM0302.mse.am.mot.com lrmd: info: log_finished: finished - rsc:application action:stop call_id:22 pid:9364 exit-code:0 exec-time:50ms queue-time:0ms
Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com crmd: notice: process_lrm_event: Result of stop operation for application on dvtvm0302: 0 (ok) | call=22 key=application_stop_0 confirmed=true cib-update=93
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/93)
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.29.70 2
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.29.71 (null)
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=71
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='application']/lrm_rsc_op[@id='application_last_0']: @transition-magic=0:0;15:8:0:4ad59ad3-3a70-4282-a52c-1f14cce6b076, @call-id=22, @rc-code=0, @op-status=0, @exec-time=50
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0302/crmd/93, version=0.29.71)
Jul 20 07:46:49 [1385] DVTVM0302.mse.am.mot.com pacemakerd: info: mcp_cpg_deliver: Ignoring process list sent by peer for local node
Jul 20 07:46:49 [1565] DVTVM0302.mse.am.mot.com lrmd: info: cancel_recurring_action: Cancelling ocf operation ClusterIP_monitor_1000
Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com crmd: info: do_lrm_rsc_op: Performing key=6:8:0:4ad59ad3-3a70-4282-a52c-1f14cce6b076 op=ClusterIP_stop_0
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/94)
Jul 20 07:46:49 [1565] DVTVM0302.mse.am.mot.com lrmd: info: log_execute: executing - rsc:ClusterIP action:stop call_id:24
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.29.71 2
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.29.72 (null)
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=72
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='ClusterIP']/lrm_rsc_op[@id='ClusterIP_last_0']: @operation_key=ClusterIP_stop_0, @operation=stop, @crm-debug-origin=do_update_resource, @transition-key=6:8:0:4ad59ad3-3a70-4282-a52c-1f14cce6b076, @transition-magic=-1:193;6:8:0:4ad59ad3-3a70-4282-a52c-1f14cce6b076, @call-id=-1, @rc-code=193, @op-status=-1, @last-run=1532087209, @last-rc-change
Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com crmd: info: process_lrm_event: Result of monitor operation for ClusterIP on dvtvm0302: Cancelled | call=18 key=ClusterIP_monitor_1000 confirmed=true
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0302/crmd/94, version=0.29.72)
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_file_write_with_digest: Reading cluster configuration file /var/lib/pacemaker/cib/cib.mJOEZ4 (digest: /var/lib/pacemaker/cib/cib.Tr4rcL)
Jul 20 07:46:49 IPaddr2(ClusterIP)[9375]: INFO: IP status = ok, IP_CIP=
Jul 20 07:46:49 [1565] DVTVM0302.mse.am.mot.com lrmd: info: log_finished: finished - rsc:ClusterIP action:stop call_id:24 pid:9375 exit-code:0 exec-time:108ms queue-time:0ms
Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com crmd: notice: process_lrm_event: Result of stop operation for ClusterIP on dvtvm0302: 0 (ok) | call=24 key=ClusterIP_stop_0 confirmed=true cib-update=95
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/95)
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.29.72 2
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.29.73 (null)
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=73
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='ClusterIP']/lrm_rsc_op[@id='ClusterIP_last_0']: @transition-magic=0:0;6:8:0:4ad59ad3-3a70-4282-a52c-1f14cce6b076, @call-id=24, @rc-code=0, @op-status=0, @exec-time=108
Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0302/crmd/95, version=0.29.73)
Jul 20 07:46:50 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.29.73 2
Jul 20 07:46:50 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.29.74 (null)
Jul 20 07:46:50 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=74
Jul 20 07:46:50 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: ++ /cib/status/node_state[@id='1']: <transient_attributes id="1"/>
Jul 20 07:46:50 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: ++ <instance_attributes id="status-1">
Jul 20 07:46:50 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: ++ <nvpair id="status-1-pingd" name="pingd" value="2000"/>
Jul 20 07:46:50 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: ++ </instance_attributes>
Jul 20 07:46:50 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: ++ </transient_attributes>
Jul 20 07:46:50 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0303/attrd/8, version=0.29.74)
Jul 20 07:46:50 [1569] DVTVM0302.mse.am.mot.com crmd: info: do_lrm_rsc_op: Performing key=4:9:0:4ad59ad3-3a70-4282-a52c-1f14cce6b076 op=ClusterIP_start_0
Jul 20 07:46:50 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/96)
Jul 20 07:46:50 [1565] DVTVM0302.mse.am.mot.com lrmd: info: log_execute: executing - rsc:ClusterIP action:start call_id:25
Jul 20 07:46:50 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.29.74 2
Jul 20 07:46:50 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.29.75 (null)
Jul 20 07:46:50 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=75
Jul 20 07:46:50 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='ClusterIP']/lrm_rsc_op[@id='ClusterIP_last_0']: @operation_key=ClusterIP_start_0, @operation=start, @transition-key=4:9:0:4ad59ad3-3a70-4282-a52c-1f14cce6b076, @transition-magic=-1:193;4:9:0:4ad59ad3-3a70-4282-a52c-1f14cce6b076, @call-id=-1, @rc-code=193, @op-status=-1, @last-run=1532087210, @last-rc-change=1532087210, @exec-time=0
Jul 20 07:46:50 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0302/crmd/96, version=0.29.75)
Jul 20 07:46:50 IPaddr2(ClusterIP)[9442]: INFO: Adding inet address 10.144.101.210/24 with broadcast address 10.144.101.255 to device eth0
Jul 20 07:46:50 IPaddr2(ClusterIP)[9442]: INFO: Bringing device eth0 up
Jul 20 07:46:50 IPaddr2(ClusterIP)[9442]: INFO: /usr/libexec/heartbeat/send_arp -i 200 -c 5 -p /var/run/resource-agents/send_arp-10.144.101.210 -I eth0 -m auto 10.144.101.210
Jul 20 07:46:52 [1566] DVTVM0302.mse.am.mot.com attrd: info: attrd_peer_update: Setting pingd[dvtvm0303]: 0 -> 2000 from dvtvm0303
Jul 20 07:46:53 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.29.75 2
Jul 20 07:46:53 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.29.76 (null)
Jul 20 07:46:53 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=76
Jul 20 07:46:53 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-pingd']: @value=2000
Jul 20 07:46:53 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0303/attrd/9, version=0.29.76)
Jul 20 07:46:54 [1565] DVTVM0302.mse.am.mot.com lrmd: info: log_finished: finished - rsc:ClusterIP action:start call_id:25 pid:9442 exit-code:0 exec-time:4131ms queue-time:0ms
Jul 20 07:46:54 [1569] DVTVM0302.mse.am.mot.com crmd: notice: process_lrm_event: Result of start operation for ClusterIP on dvtvm0302: 0 (ok) | call=25 key=ClusterIP_start_0 confirmed=true cib-update=97
Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/97)
Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.29.76 2
Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.29.77 (null)
Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=77
Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='ClusterIP']/lrm_rsc_op[@id='ClusterIP_last_0']: @transition-magic=0:0;4:9:0:4ad59ad3-3a70-4282-a52c-1f14cce6b076, @call-id=25, @rc-code=0, @op-status=0, @exec-time=4131
Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0302/crmd/97, version=0.29.77)
Jul 20 07:46:54 [1569] DVTVM0302.mse.am.mot.com crmd: info: do_lrm_rsc_op: Performing key=6:10:0:4ad59ad3-3a70-4282-a52c-1f14cce6b076 op=ClusterIP_monitor_1000
Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/98)
Jul 20 07:46:54 [1569] DVTVM0302.mse.am.mot.com crmd: info: do_lrm_rsc_op: Performing key=15:10:0:4ad59ad3-3a70-4282-a52c-1f14cce6b076 op=application_start_0
Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.29.77 2
Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.29.78 (null)
Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=78
Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='ClusterIP']/lrm_rsc_op[@id='ClusterIP_monitor_1000']: @crm-debug-origin=do_update_resource, @transition-key=6:10:0:4ad59ad3-3a70-4282-a52c-1f14cce6b076, @transition-magic=-1:193;6:10:0:4ad59ad3-3a70-4282-a52c-1f14cce6b076, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1532087214, @exec-time=0
Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0302/crmd/98, version=0.29.78)
Jul 20 07:46:54 [1569] DVTVM0302.mse.am.mot.com crmd: info: action_synced_wait: Managed application_meta-data_0 process 9587 exited with rc=4
Jul 20 07:46:54 [1569] DVTVM0302.mse.am.mot.com crmd: error: lrmd_api_get_metadata: Failed to receive meta-data for ocf:vendor:application
Jul 20 07:46:54 [1569] DVTVM0302.mse.am.mot.com crmd: warning: build_operation_update: Failed to get metadata for application (ocf:vendor:application)
Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/99)
Jul 20 07:46:54 [1565] DVTVM0302.mse.am.mot.com lrmd: info: log_execute: executing - rsc:application action:start call_id:27
Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.29.78 2
Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.29.79 (null)
Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=79
Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='application']/lrm_rsc_op[@id='application_last_0']: @operation_key=application_start_0, @operation=start, @transition-key=15:10:0:4ad59ad3-3a70-4282-a52c-1f14cce6b076, @transition-magic=-1:193;15:10:0:4ad59ad3-3a70-4282-a52c-1f14cce6b076, @call-id=-1, @rc-code=193, @op-status=-1, @last-run=1532087214, @last-rc-change=1532087214, @exec-time=0
Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0302/crmd/99, version=0.29.79)
Jul 20 07:46:54 application(application)[9588]: INFO: Starting APP - setting to Primary
Jul 20 07:46:54 [1565] DVTVM0302.mse.am.mot.com lrmd: info: log_finished: finished - rsc:application action:start call_id:27 pid:9588 exit-code:0 exec-time:50ms queue-time:0ms
Jul 20 07:46:54 [1569] DVTVM0302.mse.am.mot.com crmd: info: action_synced_wait: Managed application_meta-data_0 process 9624 exited with rc=4
Jul 20 07:46:54 [1569] DVTVM0302.mse.am.mot.com crmd: error: lrmd_api_get_metadata: Failed to receive meta-data for ocf:vendor:application
Jul 20 07:46:54 [1569] DVTVM0302.mse.am.mot.com crmd: warning: build_operation_update: Failed to get metadata for application (ocf:vendor:application)
Jul 20 07:46:54 [1569] DVTVM0302.mse.am.mot.com crmd: notice: process_lrm_event: Result of start operation for application on dvtvm0302: 0 (ok) | call=27 key=application_start_0 confirmed=true cib-update=100
Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/100)
Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.29.79 2
Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.29.80 (null)
Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=80
Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='application']/lrm_rsc_op[@id='application_last_0']: @transition-magic=0:0;15:10:0:4ad59ad3-3a70-4282-a52c-1f14cce6b076, @call-id=27, @rc-code=0, @op-status=0, @exec-time=50
Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0302/crmd/100, version=0.29.80)
Jul 20 07:46:54 [1569] DVTVM0302.mse.am.mot.com crmd: info: do_lrm_rsc_op: Performing key=16:10:0:4ad59ad3-3a70-4282-a52c-1f14cce6b076 op=application_monitor_1000
Jul 20 07:46:54 [1569] DVTVM0302.mse.am.mot.com crmd: info: action_synced_wait: Managed application_meta-data_0 process 9629 exited with rc=4
Jul 20 07:46:54 [1569] DVTVM0302.mse.am.mot.com crmd: error: lrmd_api_get_metadata: Failed to receive meta-data for ocf:vendor:application
Jul 20 07:46:54 [1569] DVTVM0302.mse.am.mot.com crmd: warning: build_operation_update: Failed to get metadata for application (ocf:vendor:application)
Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/101)
Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.29.80 2
Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.29.81 (null)
Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=81
Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='application']/lrm_rsc_op[@id='application_monitor_1000']: @crm-debug-origin=do_update_resource, @transition-key=16:10:0:4ad59ad3-3a70-4282-a52c-1f14cce6b076, @transition-magic=-1:193;16:10:0:4ad59ad3-3a70-4282-a52c-1f14cce6b076, @call-id=-1, @rc-code=193, @op-status=-1, @last-rc-change=1532087214, @exec-time=0
Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0302/crmd/101, version=0.29.81)
Jul 20 07:46:54 [1569] DVTVM0302.mse.am.mot.com crmd: info: process_lrm_event: Result of monitor operation for ClusterIP on dvtvm0302: 0 (ok) | call=26 key=ClusterIP_monitor_1000 confirmed=false cib-update=102
Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/102)
Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.29.81 2
Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.29.82 (null)
Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=82
Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='ClusterIP']/lrm_rsc_op[@id='ClusterIP_monitor_1000']: @transition-magic=0:0;6:10:0:4ad59ad3-3a70-4282-a52c-1f14cce6b076, @call-id=26, @rc-code=0, @op-status=0, @exec-time=88
Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0302/crmd/102, version=0.29.82)
Jul 20 07:46:54 [1569] DVTVM0302.mse.am.mot.com crmd: info: action_synced_wait: Managed application_meta-data_0 process 9654 exited with rc=4
Jul 20 07:46:54 [1569] DVTVM0302.mse.am.mot.com crmd: error: lrmd_api_get_metadata: Failed to receive meta-data for ocf:vendor:application
Jul 20 07:46:54 [1569] DVTVM0302.mse.am.mot.com crmd: warning: build_operation_update: Failed to get metadata for application (ocf:vendor:application)
Jul 20 07:46:54 [1569] DVTVM0302.mse.am.mot.com crmd: info: process_lrm_event: Result of monitor operation for application on dvtvm0302: 0 (ok) | call=28 key=application_monitor_1000 confirmed=false cib-update=103
Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/103)
Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: --- 0.29.82 2
Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: Diff: +++ 0.29.83 (null)
Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib: @num_updates=83
Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='application']/lrm_rsc_op[@id='application_monitor_1000']: @transition-magic=0:0;16:10:0:4ad59ad3-3a70-4282-a52c-1f14cce6b076, @call-id=28, @rc-code=0, @op-status=0, @exec-time=28
Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=dvtvm0302/crmd/103, version=0.29.83)
Jul 20 07:46:59 [1563] DVTVM0302.mse.am.mot.com cib: info: cib_process_ping: Reporting our current digest to dvtvm0303: ed20687114e2def7cb9106c09ff45a07 for 0.29.83 (0x56163caf1220 0)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20180724/a46698e9/attachment-0001.html>
More information about the Users
mailing list