[ClusterLabs] Antw: Trying to prevent instantaneous Failover / Failback at standby reconnect

Ulrich Windl Ulrich.Windl at rz.uni-regensburg.de
Wed Jul 25 01:54:20 EDT 2018


Hi!

You seem to have no STONITH device configured. Any you create a classical
split brain scenario.

Another problem is this: "Another DC detected"

Regards,
Ulrich

>>> "O'Donovan, Garret" <Garret.ODonovan at arris.com> schrieb am 24.07.2018 um
19:59
in Nachricht
<46760769342FDA45B4A87E8C5DECFB5702DD4880A4 at HPAEXMBX1.ARRS.ARRISI.com>:
> Hello, and thank you for adding me to the list.
> 
> We are using Pacemaker in a two‑node hot‑warm redundancy configuration. 
Both 
> nodes run ocf:pacemaker:ping (cloned) to monitor a ping group of devices.  
> The nodes share a virtual IP using ocf:heartbeat:IPAddr2.  Our applications

> run in either Primary mode (does all the work and sends status updates to
its 
> peer), or Standby mode (receives status updates and is ready to take over if

> current primary fails).  We have constraints set up so that IPAddr2 fails 
> over on any failure Pacemaker detects (including ping group failure), and
the 
> applications follow it.
> 
> This works great for most use cases, but we have issues in one test case 
> where we disconnect the node in standby (by yanking the eth cable) for about

> 30 seconds to a minute, and then reconnect it.  The problem is that
Pacemaker 
> seems to put the Primary into standby for a very short time while 
> reconnecting the two nodes, and then making it primary again.
> 
> Is there any way to prevent Pacemaker from doing this? Detailed config info

> and log file snippet are below.
> 
> ‑ Regards
> ‑ Garret O'Donovan
> 
> 
> PLATFORM
> This is all running this on CentOS 7 
> (centos‑release‑7‑4.1708.el7.centos.x86_64) on VM’s (VMware ESXi 5.5).  
> The two nodes are hosted on physically different servers.
> 
> VERSION INFO
> corosync‑2.4.3‑2.el7_5.1.x86_64.rpm
> pacemaker‑1.1.18‑11.el7_5.2.x86_64.rpm
> pcs‑0.9.162‑5.el7.centos.1.x86_64.rpm
> resource‑agents‑3.9.5‑124.el7.x86_64.rpm
> 
> PACEMAKER CONFIGURATION
> [root at DVTVM0302 ~]# pcs config show
> Cluster Name: vendor1
> Corosync Nodes:
>  dvtvm0302 dvtvm0303
> Pacemaker Nodes:
>  dvtvm0302 dvtvm0303
> 
> Resources:
>  Resource: ClusterIP (class=ocf provider=heartbeat type=IPaddr2)
>   Attributes: cidr_netmask=24 ip=10.144.101.210 nic=eth0
>   Operations: monitor interval=1s (ClusterIP‑monitor‑interval‑1s)
>               start interval=0s timeout=20s (ClusterIP‑start‑interval‑0s)
>               stop interval=0s timeout=20s (ClusterIP‑stop‑interval‑0s)
>  Resource: application (class=ocf provider=vendor type=application)
>   Operations: migrate_from interval=0s timeout=20 
> (application‑migrate_from‑interval‑0s)
>               migrate_to interval=0s timeout=20 
> (application‑migrate_to‑interval‑0s)
>               monitor interval=1s (application‑monitor‑interval‑1s)
>               reload interval=0s timeout=20
(application‑reload‑interval‑0s)
>               start interval=0s timeout=20 (application‑start‑interval‑0s)
>               stop interval=0s timeout=20 (application‑stop‑interval‑0s)
>  Clone: Connected‑clone
>   Meta Attrs: interleave=true
>   Resource: Connected (class=ocf provider=pacemaker type=ping)
>    Attributes: attempts=2 dampen=1s debug=true host_list="10.10.24.5 
> 10.10.24.18" multiplier=1000
>    Operations: monitor interval=3s timeout=10
(Connected‑monitor‑interval‑3s)
>                start interval=0 timeout=3 (Connected‑start‑interval‑0)
>                stop interval=0s timeout=20 (Connected‑stop‑interval‑0s)
> 
> Stonith Devices:
> Fencing Levels:
> 
> Location Constraints:
>   Resource: ClusterIP
>     Constraint: location‑ClusterIP
>       Rule: boolean‑op=or score=‑INFINITY  (id:location‑ClusterIP‑rule)
>         Expression: pingd lt 500  (id:location‑ClusterIP‑rule‑expr)
>         Expression: not_defined pingd  (id:location‑ClusterIP‑rule‑expr‑1)
> Ordering Constraints:
>   start ClusterIP then start application (kind:Mandatory)
> Colocation Constraints:
>   ClusterIP with application (score:INFINITY)
> Ticket Constraints:
> 
> Alerts:
>  No alerts defined
> 
> Resources Defaults:
>  migration‑threshold: 1
>  failure‑timeout: 5s
>  cluster‑recheck‑interval: 5s
>  resource‑stickiness: INFINITY
> Operations Defaults:
>  No defaults set
> 
> Cluster Properties:
>  cluster‑infrastructure: corosync
>  cluster‑name: vendor1
>  dc‑version: 1.1.18‑11.el7_5.2‑2b07d5c5a9
>  have‑watchdog: false
>  no‑quorum‑policy: ignore
>  stonith‑enabled: false
> 
> Quorum:
>   Options:
> 
> COROSYNC CONFIG FILE:
> 
> [root at DVTVM0302 corosync]# cat corosync.conf
> totem {
>     version: 2
>     cluster_name: vendor1
>     secauth: off
>     transport: udpu
> }
> 
> nodelist {
>     node {
>         ring0_addr: dvtvm0302
>         nodeid: 1
>     }
> 
>     node {
>         ring0_addr: dvtvm0303
>         nodeid: 2
>     }
> }
> 
> quorum {
>     provider: corosync_votequorum
>     two_node: 1
> }
> 
> logging {
>     to_logfile: yes
>     logfile: /var/log/cluster/corosync.log
>     to_syslog: yes
> }
> 
> 
> LOGFILE:
> /var/log/cluster/corosync.log
> 
> Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com       crmd:     info: 
> pcmk_cpg_membership:    Node 2 joined group crmd (counter=5.0)
> Jul 20 07:46:49 [1564] DVTVM0302.mse.am.mot.com stonith‑ng:     info: 
> pcmk_cpg_membership:    Node 2 joined group stonith‑ng (counter=5.0)
> Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com       crmd:     info: 
> pcmk_cpg_membership:    Node 1 still member of group crmd (peer=dvtvm0302, 
> counter=5.0)
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> pcmk_cpg_membership:    Node 2 joined group cib (counter=5.0)
> Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com       crmd:     info: 
> pcmk_cpg_membership:    Node 2 still member of group crmd (peer=dvtvm0303, 
> counter=5.1)
> Jul 20 07:46:49 [1564] DVTVM0302.mse.am.mot.com stonith‑ng:     info: 
> pcmk_cpg_membership:    Node 1 still member of group stonith‑ng 
> (peer=dvtvm0302, counter=5.0)
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> pcmk_cpg_membership:    Node 1 still member of group cib (peer=dvtvm0302, 
> counter=5.0)
> Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com       crmd:     info: 
> crm_update_peer_proc:    pcmk_cpg_membership: Node dvtvm0303[2] ‑
corosync‑cpg 
> is now online
> Jul 20 07:46:49 [1385] DVTVM0302.mse.am.mot.com pacemakerd:     info: 
> pcmk_cpg_membership:    Node 2 joined group pacemakerd (counter=5.0)
> Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com       crmd:     info: 
> peer_update_callback:    Client dvtvm0303/peer now has status [online] 
> (DC=true, changed=4000000)
> Jul 20 07:46:49 [1385] DVTVM0302.mse.am.mot.com pacemakerd:     info: 
> pcmk_cpg_membership:    Node 1 still member of group pacemakerd 
> (peer=dvtvm0302, counter=5.0)
> Jul 20 07:46:49 [1385] DVTVM0302.mse.am.mot.com pacemakerd:     info: 
> pcmk_cpg_membership:    Node 2 still member of group pacemakerd 
> (peer=dvtvm0303, counter=5.1)
> Jul 20 07:46:49 [1385] DVTVM0302.mse.am.mot.com pacemakerd:     info: 
> crm_update_peer_proc:    pcmk_cpg_membership: Node dvtvm0303[2] ‑
corosync‑cpg 
> is now online
> Jul 20 07:46:49 [1385] DVTVM0302.mse.am.mot.com pacemakerd:    error: 
> pcmk_cpg_membership:    Node dvtvm0303[2] appears to be online even though
we 
> think it is dead
> Jul 20 07:46:49 [1385] DVTVM0302.mse.am.mot.com pacemakerd:   notice: 
> crm_update_peer_state_iter:    Node dvtvm0303 state is now member | nodeid=2

> previous=lost source=pcmk_cpg_membership
> Jul 20 07:46:49 [1385] DVTVM0302.mse.am.mot.com pacemakerd:     info: 
> crm_cs_flush:    Sent 0 CPG messages  (1 remaining, last=11): Try again (6)
> Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com       crmd:    error: 
> pcmk_cpg_membership:    Node dvtvm0303[2] appears to be online even though
we 
> think it is dead
> Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com       crmd:   notice: 
> crm_update_peer_state_iter:    Node dvtvm0303 state is now member | nodeid=2

> previous=lost source=pcmk_cpg_membership
> Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com       crmd:     info: 
> peer_update_callback:    Cluster node dvtvm0303 is now member (was lost)
> Jul 20 07:46:49 [1566] DVTVM0302.mse.am.mot.com      attrd:     info: 
> pcmk_cpg_membership:    Node 2 joined group attrd (counter=5.0)
> Jul 20 07:46:49 [1566] DVTVM0302.mse.am.mot.com      attrd:     info: 
> pcmk_cpg_membership:    Node 1 still member of group attrd (peer=dvtvm0302,

> counter=5.0)
> Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com       crmd:     info: 
> pcmk_quorum_notification:    Quorum retained | membership=144 members=2
> Jul 20 07:46:49 [1385] DVTVM0302.mse.am.mot.com pacemakerd:     info: 
> pcmk_quorum_notification:    Quorum retained | membership=144 members=2
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> crm_get_peer:    Created entry 
> 603f53c1‑5c06‑4423‑b6f0‑420c6194303d/0x56163cadf350 for node dvtvm0303/2 
> (2 total)
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> crm_get_peer:    Node 2 is now known as dvtvm0303
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> crm_get_peer:    Node 2 has uuid 2
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> pcmk_cpg_membership:    Node 2 still member of group cib (peer=dvtvm0303, 
> counter=5.1)
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> crm_update_peer_proc:    pcmk_cpg_membership: Node dvtvm0303[2] ‑
corosync‑cpg 
> is now online
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:   notice: 
> crm_update_peer_state_iter:    Node dvtvm0303 state is now member | nodeid=2

> previous=unknown source=crm_update_peer_proc
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: ‑‑‑ 0.29.54 2
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: +++ 0.29.55 (null)
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  /cib:  @num_updates=55
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  /cib/status/node_state[@id='1']:  
> @crm‑debug‑origin=peer_update_callback
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Completed cib_modify operation for section status:
OK 
> (rc=0, origin=dvtvm0303/crmd/108, version=0.29.55)
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Completed cib_modify operation for section status:
OK 
> (rc=0, origin=dvtvm0303/crmd/109, version=0.29.55)
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Completed cib_modify operation for section nodes: OK

> (rc=0, origin=dvtvm0303/crmd/112, version=0.29.55)
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: ‑‑‑ 0.29.55 2
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: +++ 0.29.56 (null)
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  /cib:  @num_updates=56
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  /cib/status/node_state[@id='2']:  @in_ccm=true, 
> @crmd=online, @crm‑debug‑origin=post_cache_update
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  /cib/status/node_state[@id='1']:  
> @crm‑debug‑origin=post_cache_update
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Completed cib_modify operation for section status:
OK 
> (rc=0, origin=dvtvm0303/crmd/113, version=0.29.56)
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Forwarding cib_modify operation for section status
to 
> all (origin=local/crmd/82)
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Forwarding cib_modify operation for section status
to 
> all (origin=local/crmd/83)
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: ‑‑‑ 0.29.56 2
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: +++ 0.29.57 (null)
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  /cib:  @num_updates=57
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  /cib/status/node_state[@id='2']:  
> @crm‑debug‑origin=peer_update_callback
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Completed cib_modify operation for section status:
OK 
> (rc=0, origin=dvtvm0302/crmd/82, version=0.29.57)
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Completed cib_modify operation for section status:
OK 
> (rc=0, origin=dvtvm0302/crmd/83, version=0.29.57)
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Forwarding cib_modify operation for section nodes to

> all (origin=local/crmd/86)
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Forwarding cib_modify operation for section status
to 
> all (origin=local/crmd/87)
> Jul 20 07:46:49 [1564] DVTVM0302.mse.am.mot.com stonith‑ng:     info: 
> crm_get_peer:    Created entry 
> 1fd3c98b‑2a62‑4f07‑9343‑0850d9a68e83/0x55d0123ef950 for node dvtvm0303/2 
> (2 total)
> Jul 20 07:46:49 [1564] DVTVM0302.mse.am.mot.com stonith‑ng:     info: 
> crm_get_peer:    Node 2 is now known as dvtvm0303
> Jul 20 07:46:49 [1564] DVTVM0302.mse.am.mot.com stonith‑ng:     info: 
> crm_get_peer:    Node 2 has uuid 2
> Jul 20 07:46:49 [1564] DVTVM0302.mse.am.mot.com stonith‑ng:     info: 
> pcmk_cpg_membership:    Node 2 still member of group stonith‑ng 
> (peer=dvtvm0303, counter=5.1)
> Jul 20 07:46:49 [1564] DVTVM0302.mse.am.mot.com stonith‑ng:     info: 
> crm_update_peer_proc:    pcmk_cpg_membership: Node dvtvm0303[2] ‑
corosync‑cpg 
> is now online
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Completed cib_modify operation for section nodes: OK

> (rc=0, origin=dvtvm0302/crmd/86, version=0.29.57)
> Jul 20 07:46:49 [1566] DVTVM0302.mse.am.mot.com      attrd:     info: 
> crm_get_peer:    Created entry 
> 523ee210‑80c3‑4ba0‑a130‑989aefa5d05f/0x55b7f9daa8e0 for node dvtvm0303/2 
> (2 total)
> Jul 20 07:46:49 [1566] DVTVM0302.mse.am.mot.com      attrd:     info: 
> crm_get_peer:    Node 2 is now known as dvtvm0303
> Jul 20 07:46:49 [1566] DVTVM0302.mse.am.mot.com      attrd:     info: 
> crm_get_peer:    Node 2 has uuid 2
> Jul 20 07:46:49 [1566] DVTVM0302.mse.am.mot.com      attrd:     info: 
> pcmk_cpg_membership:    Node 2 still member of group attrd (peer=dvtvm0303,

> counter=5.1)
> Jul 20 07:46:49 [1566] DVTVM0302.mse.am.mot.com      attrd:     info: 
> crm_update_peer_proc:    pcmk_cpg_membership: Node dvtvm0303[2] ‑
corosync‑cpg 
> is now online
> Jul 20 07:46:49 [1566] DVTVM0302.mse.am.mot.com      attrd:   notice: 
> crm_update_peer_state_iter:    Node dvtvm0303 state is now member | nodeid=2

> previous=unknown source=crm_update_peer_proc
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: ‑‑‑ 0.29.57 2
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: +++ 0.29.58 (null)
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  /cib:  @num_updates=58
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  /cib/status/node_state[@id='2']:  
> @crm‑debug‑origin=post_cache_update
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Completed cib_modify operation for section status:
OK 
> (rc=0, origin=dvtvm0302/crmd/87, version=0.29.58)
> Jul 20 07:46:49 [1566] DVTVM0302.mse.am.mot.com      attrd:   notice: 
> attrd_peer_message:    Detected another attribute writer: dvtvm0303
> Jul 20 07:46:49 [1566] DVTVM0302.mse.am.mot.com      attrd:     info: 
> attrd_peer_message:    Processing sync‑response from dvtvm0303
> Jul 20 07:46:49 [1566] DVTVM0302.mse.am.mot.com      attrd:     info: 
> attrd_peer_update:    Setting #attrd‑protocol[dvtvm0303]: (null) ‑> 2 from 
> dvtvm0303
> Jul 20 07:46:49 [1566] DVTVM0302.mse.am.mot.com      attrd:     info: 
> attrd_peer_update:    Setting pingd[dvtvm0303]: (null) ‑> 0 from dvtvm0303
> Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com       crmd:     info: 
> register_fsa_error_adv:    Resetting the current action list
> Jul 20 07:46:49 [1564] DVTVM0302.mse.am.mot.com stonith‑ng:   notice: 
> crm_update_peer_state_iter:    Node dvtvm0303 state is now member | nodeid=2

> previous=unknown source=crm_update_peer_proc
> Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com       crmd:  warning: 
> crmd_ha_msg_filter:    Another DC detected: dvtvm0303 (op=noop)
> Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com       crmd:   notice: 
> do_state_transition:    State transition S_IDLE ‑> S_ELECTION | 
> input=I_ELECTION cause=C_FSA_INTERNAL origin=crmd_ha_msg_filter
> Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com       crmd:     info: 
> update_dc:    Unset DC. Was dvtvm0302
> Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com       crmd:     info: 
> election_count_vote:    Election 6 (owner: 2) lost: vote from dvtvm0303 
> (Uptime)
> Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com       crmd:   notice: 
> do_state_transition:    State transition S_ELECTION ‑> S_RELEASE_DC | 
> input=I_RELEASE_DC cause=C_FSA_INTERNAL origin=do_election_count_vote
> Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com       crmd:     info: 
> do_dc_release:    DC role released
> Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com       crmd:     info: 
> pe_ipc_destroy:    Connection to the Policy Engine released
> Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com       crmd:     info: 
> do_te_control:    Transitioner is now inactive
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_readwrite:    We are now in R/O mode
> Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com       crmd:     info: 
> do_log:    Input I_RELEASE_SUCCESS received in state S_RELEASE_DC from 
> do_dc_release
> Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com       crmd:     info: 
> do_state_transition:    State transition S_RELEASE_DC ‑> S_PENDING | 
> input=I_RELEASE_SUCCESS cause=C_FSA_INTERNAL origin=do_dc_release
> Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com       crmd:     info: 
> do_log:    Input I_PENDING received in state S_PENDING from 
> do_election_count_vote
> Jul 20 07:46:49 [1566] DVTVM0302.mse.am.mot.com      attrd:     info: 
> election_count_vote:    Election 4 (owner: 2) lost: vote from dvtvm0303 
> (Uptime)
> Jul 20 07:46:49 [1566] DVTVM0302.mse.am.mot.com      attrd:     info: 
> election_count_vote:    Election 5 (owner: 2) lost: vote from dvtvm0303 
> (Uptime)
> Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com       crmd:     info: 
> election_count_vote:    Election 7 (owner: 2) lost: vote from dvtvm0303 
> (Uptime)
> Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com       crmd:     info: 
> do_log:    Input I_PENDING received in state S_PENDING from 
> do_election_count_vote
> Jul 20 07:46:49 [1566] DVTVM0302.mse.am.mot.com      attrd:     info: 
> attrd_peer_message:    Processing sync‑response from dvtvm0303
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Completed cib_modify operation for section cib: OK 
> (rc=0, origin=dvtvm0303/crmd/115, version=0.29.58)
> Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com       crmd:     info: 
> update_dc:    Set DC to dvtvm0303 (3.0.14)
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Completed cib_modify operation for section 
> crm_config: OK (rc=0, origin=dvtvm0303/crmd/117, version=0.29.58)
> Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com       crmd:     info: 
> election_count_vote:    Election 8 (owner: 2) lost: vote from dvtvm0303 
> (Uptime)
> Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com       crmd:     info: 
> update_dc:    Unset DC. Was dvtvm0303
> Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com       crmd:     info: 
> do_log:    Input I_PENDING received in state S_PENDING from 
> do_election_count_vote
> Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com       crmd:     info: 
> update_dc:    Set DC to dvtvm0303 (3.0.14)
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Completed cib_modify operation for section 
> crm_config: OK (rc=0, origin=dvtvm0303/crmd/119, version=0.29.58)
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Completed cib_modify operation for section 
> crm_config: OK (rc=0, origin=dvtvm0303/crmd/121, version=0.29.58)
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_replace:    Digest matched on replace from dvtvm0303: 
> 741acf29cc9726f86772f735a16a723a
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_replace:    Replaced 0.29.58 with 0.29.64 from dvtvm0303
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: ‑‑‑ 0.29.58 2
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: +++ 0.29.64 (null)
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    ‑‑ 
> /cib/status/node_state[@id='1']/transient_attributes[@id='1']
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  /cib:  @num_updates=64, @dc‑uuid=2
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  /cib/status/node_state[@id='2']:  @join=member
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  
>
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='
> ClusterIP']/lrm_rsc_op[@id='ClusterIP_last_0']:  
> @operation_key=ClusterIP_stop_0, @operation=stop, 
> @crm‑debug‑origin=do_update_resource, 
> @transition‑key=3:7:0:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076, 
> @transition‑magic=0:0;3:7:0:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076,
@call‑id=18, 
> @rc‑code=0, @last‑run=1532087185, @last‑rc‑change=1532087185, @exec‑t
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  
>
/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attrib
> utes[@id='status‑2']/nvpair[@id='status‑2‑pingd']:  @value=0
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  /cib/status/node_state[@id='1']:  @join=down
> Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com       crmd:     info: 
> action_synced_wait:    Managed application_meta‑data_0 process 9340 exited 
> with rc=4
> Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com       crmd:    error: 
> lrmd_api_get_metadata:    Failed to receive meta‑data for 
> ocf:vendor:application
> Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com       crmd:  warning: 
> build_operation_update:    Failed to get metadata for application 
> (ocf:vendor:application)
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_replace_notify:    Replaced: 0.29.58 ‑> 0.29.64 from dvtvm0303
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Completed cib_replace operation for section 'all':
OK 
> (rc=0, origin=dvtvm0303/crmd/125, version=0.29.64)
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Completed cib_modify operation for section nodes: OK

> (rc=0, origin=dvtvm0303/crmd/126, version=0.29.64)
> Jul 20 07:46:49 [1566] DVTVM0302.mse.am.mot.com      attrd:   notice: 
> attrd_cib_replaced_cb:    Updating all attributes after cib_refresh_notify 
> event
> Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com       crmd:     info: 
> action_synced_wait:    Managed application_meta‑data_0 process 9341 exited 
> with rc=4
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Completed cib_modify operation for section nodes: OK

> (rc=0, origin=dvtvm0303/crmd/127, version=0.29.64)
> Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com       crmd:    error: 
> lrmd_api_get_metadata:    Failed to receive meta‑data for 
> ocf:vendor:application
> Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com       crmd:  warning: 
> build_operation_update:    Failed to get metadata for application 
> (ocf:vendor:application)
> Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com       crmd:     info: 
> do_log:    Input I_NOT_DC received in state S_PENDING from 
> do_cl_join_finalize_respond
> Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com       crmd:   notice: 
> do_state_transition:    State transition S_PENDING ‑> S_NOT_DC |
input=I_NOT_DC 
> cause=C_HA_MESSAGE origin=do_cl_join_finalize_respond
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: ‑‑‑ 0.29.64 2
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: +++ 0.29.65 (null)
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    ‑‑ /cib/status/node_state[@id='2']/lrm[@id='2']
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  /cib:  @num_updates=65
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Completed cib_delete operation for section 
> //node_state[@uname='dvtvm0303']/lrm: OK (rc=0, origin=dvtvm0303/crmd/128, 
> version=0.29.65)
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: ‑‑‑ 0.29.65 2
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: +++ 0.29.66 (null)
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  /cib:  @num_updates=66
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  /cib/status/node_state[@id='2']:  
> @crm‑debug‑origin=do_lrm_query_internal
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    ++ /cib/status/node_state[@id='2']:  <lrm id="2"/>
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    ++                                     <lrm_resources>
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    ++                                       <lrm_resource 
> id="Connected" type="ping" class="ocf" provider="pacemaker">
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    ++                                         <lrm_rsc_op 
> id="Connected_last_0" operation_key="Connected_start_0" operation="start" 
> crm‑debug‑origin="build_active_RAs" crm_feature_set="3.0.14" 
> transition‑key="8:0:0:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076" 
> transition‑magic="0:0;8:0:0:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076"
exit‑reason="" 
> on_node="dvtvm0303" call‑id="15" rc‑code="0" op‑status="0" interval="0" 
> last‑run="1532086605"
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    ++                                         <lrm_rsc_op 
> id="Connected_monitor_3000" operation_key="Connected_monitor_3000" 
> operation="monitor" crm‑debug‑origin="build_active_RAs" 
> crm_feature_set="3.0.14" 
> transition‑key="9:0:0:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076" 
> transition‑magic="0:0;9:0:0:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076"
exit‑reason="" 
> on_node="dvtvm0303" call‑id="16" rc‑code="0" op‑status="0" interval="3000"
last‑r
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    ++                                       </lrm_resource>
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    ++                                       <lrm_resource 
> id="ClusterIP" type="IPaddr2" class="ocf" provider="heartbeat">
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    ++                                         <lrm_rsc_op 
> id="ClusterIP_last_0" operation_key="ClusterIP_stop_0" operation="stop" 
> crm‑debug‑origin="build_active_RAs" crm_feature_set="3.0.14" 
> transition‑key="3:7:0:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076" 
> transition‑magic="0:0;3:7:0:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076"
exit‑reason="" 
> on_node="dvtvm0303" call‑id="18" rc‑code="0" op‑status="0" interval="0" 
> last‑run="1532087185" la
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    ++                                       </lrm_resource>
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    ++                                       <lrm_resource 
> id="application" type="application" class="ocf" provider="vendor">
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    ++                                         <lrm_rsc_op 
> id="application_last_0" operation_key="application_monitor_0" 
> operation="monitor" crm‑debug‑origin="build_active_RAs" 
> crm_feature_set="3.0.14" 
> transition‑key="7:0:7:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076" 
> transition‑magic="0:7;7:0:7:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076"
exit‑reason="" 
> on_node="dvtvm0303" call‑id="14" rc‑code="7" op‑status="0" interval="0" 
> last‑run="1532
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    ++                                       </lrm_resource>
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    ++                                     </lrm_resources>
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    ++                                   </lrm>
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Completed cib_modify operation for section status:
OK 
> (rc=0, origin=dvtvm0303/crmd/129, version=0.29.66)
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: ‑‑‑ 0.29.66 2
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: +++ 0.29.67 (null)
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    ‑‑ /cib/status/node_state[@id='1']/lrm[@id='1']
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  /cib:  @num_updates=67
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Completed cib_delete operation for section 
> //node_state[@uname='dvtvm0302']/lrm: OK (rc=0, origin=dvtvm0303/crmd/130, 
> version=0.29.67)
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: ‑‑‑ 0.29.67 2
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: +++ 0.29.68 (null)
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  /cib:  @num_updates=68
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  /cib/status/node_state[@id='1']:  
> @crm‑debug‑origin=do_lrm_query_internal, @join=member
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    ++ /cib/status/node_state[@id='1']:  <lrm id="1"/>
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    ++                                     <lrm_resources>
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    ++                                       <lrm_resource 
> id="Connected" type="ping" class="ocf" provider="pacemaker">
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    ++                                         <lrm_rsc_op 
> id="Connected_last_0" operation_key="Connected_start_0" operation="start" 
> crm‑debug‑origin="build_active_RAs" crm_feature_set="3.0.14" 
> transition‑key="10:0:0:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076" 
> transition‑magic="0:0;10:0:0:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076"
exit‑reason="" 
> on_node="dvtvm0302" call‑id="15" rc‑code="0" op‑status="0" interval="0" 
> last‑run="1532086605
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    ++                                         <lrm_rsc_op 
> id="Connected_monitor_3000" operation_key="Connected_monitor_3000" 
> operation="monitor" crm‑debug‑origin="build_active_RAs" 
> crm_feature_set="3.0.14" 
> transition‑key="11:0:0:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076" 
> transition‑magic="0:0;11:0:0:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076"
exit‑reason="" 
> on_node="dvtvm0302" call‑id="16" rc‑code="0" op‑status="0" interval="3000"
last
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    ++                                       </lrm_resource>
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    ++                                       <lrm_resource 
> id="ClusterIP" type="IPaddr2" class="ocf" provider="heartbeat">
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    ++                                         <lrm_rsc_op 
> id="ClusterIP_last_0" operation_key="ClusterIP_start_0" operation="start" 
> crm‑debug‑origin="build_active_RAs" crm_feature_set="3.0.14" 
> transition‑key="4:1:0:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076" 
> transition‑magic="0:0;4:1:0:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076"
exit‑reason="" 
> on_node="dvtvm0302" call‑id="17" rc‑code="0" op‑status="0" interval="0" 
> last‑run="1532086609"
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    ++                                         <lrm_rsc_op 
> id="ClusterIP_monitor_1000" operation_key="ClusterIP_monitor_1000" 
> operation="monitor" crm‑debug‑origin="build_active_RAs" 
> crm_feature_set="3.0.14" 
> transition‑key="5:1:0:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076" 
> transition‑magic="0:0;5:1:0:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076"
exit‑reason="" 
> on_node="dvtvm0302" call‑id="18" rc‑code="0" op‑status="0" interval="1000"
last‑r
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    ++                                       </lrm_resource>
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    ++                                       <lrm_resource 
> id="application" type="application" class="ocf" provider="vendor">
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    ++                                         <lrm_rsc_op 
> id="application_last_0" operation_key="application_start_0"
operation="start" 
> crm‑debug‑origin="build_active_RAs" crm_feature_set="3.0.14" 
> transition‑key="14:1:0:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076" 
> transition‑magic="0:0;14:1:0:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076"
exit‑reason="" 
> on_node="dvtvm0302" call‑id="19" rc‑code="0" op‑status="0" interval="0" 
> last‑run="153208
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    ++                                         <lrm_rsc_op 
> id="application_monitor_1000" operation_key="application_monitor_1000" 
> operation="monitor" crm‑debug‑origin="build_active_RAs" 
> crm_feature_set="3.0.14" 
> transition‑key="15:1:0:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076" 
> transition‑magic="0:0;15:1:0:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076"
exit‑reason="" 
> on_node="dvtvm0302" call‑id="20" rc‑code="0" op‑status="0" interval="1000"
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    ++                                       </lrm_resource>
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    ++                                     </lrm_resources>
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    ++                                   </lrm>
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Completed cib_modify operation for section status:
OK 
> (rc=0, origin=dvtvm0303/crmd/131, version=0.29.68)
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Completed cib_modify operation for section nodes: OK

> (rc=0, origin=dvtvm0303/crmd/134, version=0.29.68)
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: ‑‑‑ 0.29.68 2
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: +++ 0.29.69 (null)
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  /cib:  @num_updates=69
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  /cib/status/node_state[@id='2']:  
> @crm‑debug‑origin=do_state_transition
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  /cib/status/node_state[@id='1']:  
> @crm‑debug‑origin=do_state_transition
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Completed cib_modify operation for section status:
OK 
> (rc=0, origin=dvtvm0303/crmd/135, version=0.29.69)
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Completed cib_modify operation for section cib: OK 
> (rc=0, origin=dvtvm0303/crmd/136, version=0.29.69)
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_file_backup:    Archived previous version as 
> /var/lib/pacemaker/cib/cib‑49.raw
> Jul 20 07:46:49 [1565] DVTVM0302.mse.am.mot.com       lrmd:     info: 
> cancel_recurring_action:    Cancelling ocf operation
application_monitor_1000
> Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com       crmd:     info: 
> do_lrm_rsc_op:    Performing key=15:8:0:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076

> op=application_stop_0
> Jul 20 07:46:49 [1565] DVTVM0302.mse.am.mot.com       lrmd:     info: 
> log_execute:    executing ‑ rsc:application action:stop call_id:22
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Forwarding cib_modify operation for section status
to 
> all (origin=local/crmd/92)
> Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com       crmd:     info: 
> process_lrm_event:    Result of monitor operation for application on 
> dvtvm0302: Cancelled | call=20 key=application_monitor_1000 confirmed=true
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: ‑‑‑ 0.29.69 2
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: +++ 0.29.70 (null)
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  /cib:  @num_updates=70
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  /cib/status/node_state[@id='1']:  
> @crm‑debug‑origin=do_update_resource
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  
>
/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='
> application']/lrm_rsc_op[@id='application_last_0']:  
> @operation_key=application_stop_0, @operation=stop, 
> @crm‑debug‑origin=do_update_resource, 
> @transition‑key=15:8:0:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076, 
> @transition‑magic=‑1:193;15:8:0:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076,
@call‑id=‑1, 
> @rc‑code=193, @op‑status=‑1, @last‑run=1532087209, @last‑r
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Completed cib_modify operation for section status:
OK 
> (rc=0, origin=dvtvm0302/crmd/92, version=0.29.70)
> Jul 20 07:46:49  application(application)[9364]:    INFO: Stopping APP ‑ 
> setting to Standby
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_file_write_with_digest:    Wrote version 0.29.0 of the CIB to disk 
> (digest: 67b46b2a11db4ce9d1a5fb649ae91bd3)
> Jul 20 07:46:49 [1565] DVTVM0302.mse.am.mot.com       lrmd:   notice: 
> operation_finished:    application_stop_0:9364:stderr [ 
> /usr/local/vendor/bin/PrimaryMode.sh: line 27: $Filename: ambiguous redirect

> ]
> Jul 20 07:46:49 [1565] DVTVM0302.mse.am.mot.com       lrmd:     info: 
> log_finished:    finished ‑ rsc:application action:stop call_id:22 pid:9364

> exit‑code:0 exec‑time:50ms queue‑time:0ms
> Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com       crmd:   notice: 
> process_lrm_event:    Result of stop operation for application on dvtvm0302:

> 0 (ok) | call=22 key=application_stop_0 confirmed=true cib‑update=93
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Forwarding cib_modify operation for section status
to 
> all (origin=local/crmd/93)
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: ‑‑‑ 0.29.70 2
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: +++ 0.29.71 (null)
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  /cib:  @num_updates=71
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  
>
/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='
> application']/lrm_rsc_op[@id='application_last_0']:  
> @transition‑magic=0:0;15:8:0:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076,
@call‑id=22, 
> @rc‑code=0, @op‑status=0, @exec‑time=50
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Completed cib_modify operation for section status:
OK 
> (rc=0, origin=dvtvm0302/crmd/93, version=0.29.71)
> Jul 20 07:46:49 [1385] DVTVM0302.mse.am.mot.com pacemakerd:     info: 
> mcp_cpg_deliver:    Ignoring process list sent by peer for local node
> Jul 20 07:46:49 [1565] DVTVM0302.mse.am.mot.com       lrmd:     info: 
> cancel_recurring_action:    Cancelling ocf operation ClusterIP_monitor_1000
> Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com       crmd:     info: 
> do_lrm_rsc_op:    Performing key=6:8:0:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076

> op=ClusterIP_stop_0
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Forwarding cib_modify operation for section status
to 
> all (origin=local/crmd/94)
> Jul 20 07:46:49 [1565] DVTVM0302.mse.am.mot.com       lrmd:     info: 
> log_execute:    executing ‑ rsc:ClusterIP action:stop call_id:24
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: ‑‑‑ 0.29.71 2
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: +++ 0.29.72 (null)
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  /cib:  @num_updates=72
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  
>
/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='
> ClusterIP']/lrm_rsc_op[@id='ClusterIP_last_0']:  
> @operation_key=ClusterIP_stop_0, @operation=stop, 
> @crm‑debug‑origin=do_update_resource, 
> @transition‑key=6:8:0:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076, 
> @transition‑magic=‑1:193;6:8:0:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076,
@call‑id=‑1, 
> @rc‑code=193, @op‑status=‑1, @last‑run=1532087209, @last‑rc‑change
> Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com       crmd:     info: 
> process_lrm_event:    Result of monitor operation for ClusterIP on
dvtvm0302: 
> Cancelled | call=18 key=ClusterIP_monitor_1000 confirmed=true
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Completed cib_modify operation for section status:
OK 
> (rc=0, origin=dvtvm0302/crmd/94, version=0.29.72)
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_file_write_with_digest:    Reading cluster configuration file 
> /var/lib/pacemaker/cib/cib.mJOEZ4 (digest:
/var/lib/pacemaker/cib/cib.Tr4rcL)
> Jul 20 07:46:49  IPaddr2(ClusterIP)[9375]:    INFO: IP status = ok, IP_CIP=
> Jul 20 07:46:49 [1565] DVTVM0302.mse.am.mot.com       lrmd:     info: 
> log_finished:    finished ‑ rsc:ClusterIP action:stop call_id:24 pid:9375 
> exit‑code:0 exec‑time:108ms queue‑time:0ms
> Jul 20 07:46:49 [1569] DVTVM0302.mse.am.mot.com       crmd:   notice: 
> process_lrm_event:    Result of stop operation for ClusterIP on dvtvm0302: 0

> (ok) | call=24 key=ClusterIP_stop_0 confirmed=true cib‑update=95
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Forwarding cib_modify operation for section status
to 
> all (origin=local/crmd/95)
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: ‑‑‑ 0.29.72 2
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: +++ 0.29.73 (null)
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  /cib:  @num_updates=73
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  
>
/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='
> ClusterIP']/lrm_rsc_op[@id='ClusterIP_last_0']:  
> @transition‑magic=0:0;6:8:0:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076,
@call‑id=24, 
> @rc‑code=0, @op‑status=0, @exec‑time=108
> Jul 20 07:46:49 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Completed cib_modify operation for section status:
OK 
> (rc=0, origin=dvtvm0302/crmd/95, version=0.29.73)
> Jul 20 07:46:50 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: ‑‑‑ 0.29.73 2
> Jul 20 07:46:50 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: +++ 0.29.74 (null)
> Jul 20 07:46:50 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  /cib:  @num_updates=74
> Jul 20 07:46:50 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    ++ /cib/status/node_state[@id='1']: 
<transient_attributes 
> id="1"/>
> Jul 20 07:46:50 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    ++                                    
<instance_attributes 
> id="status‑1">
> Jul 20 07:46:50 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    ++                                       <nvpair 
> id="status‑1‑pingd" name="pingd" value="2000"/>
> Jul 20 07:46:50 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    ++                                     
> </instance_attributes>
> Jul 20 07:46:50 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    ++                                  
</transient_attributes>
> Jul 20 07:46:50 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Completed cib_modify operation for section status:
OK 
> (rc=0, origin=dvtvm0303/attrd/8, version=0.29.74)
> Jul 20 07:46:50 [1569] DVTVM0302.mse.am.mot.com       crmd:     info: 
> do_lrm_rsc_op:    Performing key=4:9:0:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076

> op=ClusterIP_start_0
> Jul 20 07:46:50 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Forwarding cib_modify operation for section status
to 
> all (origin=local/crmd/96)
> Jul 20 07:46:50 [1565] DVTVM0302.mse.am.mot.com       lrmd:     info: 
> log_execute:    executing ‑ rsc:ClusterIP action:start call_id:25
> Jul 20 07:46:50 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: ‑‑‑ 0.29.74 2
> Jul 20 07:46:50 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: +++ 0.29.75 (null)
> Jul 20 07:46:50 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  /cib:  @num_updates=75
> Jul 20 07:46:50 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  
>
/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='
> ClusterIP']/lrm_rsc_op[@id='ClusterIP_last_0']:  
> @operation_key=ClusterIP_start_0, @operation=start, 
> @transition‑key=4:9:0:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076, 
> @transition‑magic=‑1:193;4:9:0:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076,
@call‑id=‑1, 
> @rc‑code=193, @op‑status=‑1, @last‑run=1532087210,
@last‑rc‑change=1532087210, 
> @exec‑time=0
> Jul 20 07:46:50 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Completed cib_modify operation for section status:
OK 
> (rc=0, origin=dvtvm0302/crmd/96, version=0.29.75)
> Jul 20 07:46:50  IPaddr2(ClusterIP)[9442]:    INFO: Adding inet address 
> 10.144.101.210/24 with broadcast address 10.144.101.255 to device eth0
> Jul 20 07:46:50  IPaddr2(ClusterIP)[9442]:    INFO: Bringing device eth0 up
> Jul 20 07:46:50  IPaddr2(ClusterIP)[9442]:    INFO: 
> /usr/libexec/heartbeat/send_arp ‑i 200 ‑c 5 ‑p 
> /var/run/resource‑agents/send_arp‑10.144.101.210 ‑I eth0 ‑m auto 
> 10.144.101.210
> Jul 20 07:46:52 [1566] DVTVM0302.mse.am.mot.com      attrd:     info: 
> attrd_peer_update:    Setting pingd[dvtvm0303]: 0 ‑> 2000 from dvtvm0303
> Jul 20 07:46:53 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: ‑‑‑ 0.29.75 2
> Jul 20 07:46:53 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: +++ 0.29.76 (null)
> Jul 20 07:46:53 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  /cib:  @num_updates=76
> Jul 20 07:46:53 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  
>
/cib/status/node_state[@id='2']/transient_attributes[@id='2']/instance_attrib
> utes[@id='status‑2']/nvpair[@id='status‑2‑pingd']:  @value=2000
> Jul 20 07:46:53 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Completed cib_modify operation for section status:
OK 
> (rc=0, origin=dvtvm0303/attrd/9, version=0.29.76)
> Jul 20 07:46:54 [1565] DVTVM0302.mse.am.mot.com       lrmd:     info: 
> log_finished:    finished ‑ rsc:ClusterIP action:start call_id:25 pid:9442 
> exit‑code:0 exec‑time:4131ms queue‑time:0ms
> Jul 20 07:46:54 [1569] DVTVM0302.mse.am.mot.com       crmd:   notice: 
> process_lrm_event:    Result of start operation for ClusterIP on dvtvm0302:
0 
> (ok) | call=25 key=ClusterIP_start_0 confirmed=true cib‑update=97
> Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Forwarding cib_modify operation for section status
to 
> all (origin=local/crmd/97)
> Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: ‑‑‑ 0.29.76 2
> Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: +++ 0.29.77 (null)
> Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  /cib:  @num_updates=77
> Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  
>
/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='
> ClusterIP']/lrm_rsc_op[@id='ClusterIP_last_0']:  
> @transition‑magic=0:0;4:9:0:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076,
@call‑id=25, 
> @rc‑code=0, @op‑status=0, @exec‑time=4131
> Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Completed cib_modify operation for section status:
OK 
> (rc=0, origin=dvtvm0302/crmd/97, version=0.29.77)
> Jul 20 07:46:54 [1569] DVTVM0302.mse.am.mot.com       crmd:     info: 
> do_lrm_rsc_op:    Performing key=6:10:0:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076

> op=ClusterIP_monitor_1000
> Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Forwarding cib_modify operation for section status
to 
> all (origin=local/crmd/98)
> Jul 20 07:46:54 [1569] DVTVM0302.mse.am.mot.com       crmd:     info: 
> do_lrm_rsc_op:    Performing
key=15:10:0:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076 
> op=application_start_0
> Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: ‑‑‑ 0.29.77 2
> Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: +++ 0.29.78 (null)
> Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  /cib:  @num_updates=78
> Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  
>
/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='
> ClusterIP']/lrm_rsc_op[@id='ClusterIP_monitor_1000']:  
> @crm‑debug‑origin=do_update_resource, 
> @transition‑key=6:10:0:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076, 
> @transition‑magic=‑1:193;6:10:0:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076,
@call‑id=‑1, 
> @rc‑code=193, @op‑status=‑1, @last‑rc‑change=1532087214, @exec‑time=0
> Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Completed cib_modify operation for section status:
OK 
> (rc=0, origin=dvtvm0302/crmd/98, version=0.29.78)
> Jul 20 07:46:54 [1569] DVTVM0302.mse.am.mot.com       crmd:     info: 
> action_synced_wait:    Managed application_meta‑data_0 process 9587 exited 
> with rc=4
> Jul 20 07:46:54 [1569] DVTVM0302.mse.am.mot.com       crmd:    error: 
> lrmd_api_get_metadata:    Failed to receive meta‑data for 
> ocf:vendor:application
> Jul 20 07:46:54 [1569] DVTVM0302.mse.am.mot.com       crmd:  warning: 
> build_operation_update:    Failed to get metadata for application 
> (ocf:vendor:application)
> Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Forwarding cib_modify operation for section status
to 
> all (origin=local/crmd/99)
> Jul 20 07:46:54 [1565] DVTVM0302.mse.am.mot.com       lrmd:     info: 
> log_execute:    executing ‑ rsc:application action:start call_id:27
> Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: ‑‑‑ 0.29.78 2
> Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: +++ 0.29.79 (null)
> Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  /cib:  @num_updates=79
> Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  
>
/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='
> application']/lrm_rsc_op[@id='application_last_0']:  
> @operation_key=application_start_0, @operation=start, 
> @transition‑key=15:10:0:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076, 
> @transition‑magic=‑1:193;15:10:0:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076, 
> @call‑id=‑1, @rc‑code=193, @op‑status=‑1, @last‑run=1532087214, 
> @last‑rc‑change=1532087214, @exec‑time=0
> Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Completed cib_modify operation for section status:
OK 
> (rc=0, origin=dvtvm0302/crmd/99, version=0.29.79)
> Jul 20 07:46:54  application(application)[9588]:    INFO: Starting APP ‑ 
> setting to Primary
> Jul 20 07:46:54 [1565] DVTVM0302.mse.am.mot.com       lrmd:     info: 
> log_finished:    finished ‑ rsc:application action:start call_id:27 pid:9588

> exit‑code:0 exec‑time:50ms queue‑time:0ms
> Jul 20 07:46:54 [1569] DVTVM0302.mse.am.mot.com       crmd:     info: 
> action_synced_wait:    Managed application_meta‑data_0 process 9624 exited 
> with rc=4
> Jul 20 07:46:54 [1569] DVTVM0302.mse.am.mot.com       crmd:    error: 
> lrmd_api_get_metadata:    Failed to receive meta‑data for 
> ocf:vendor:application
> Jul 20 07:46:54 [1569] DVTVM0302.mse.am.mot.com       crmd:  warning: 
> build_operation_update:    Failed to get metadata for application 
> (ocf:vendor:application)
> Jul 20 07:46:54 [1569] DVTVM0302.mse.am.mot.com       crmd:   notice: 
> process_lrm_event:    Result of start operation for application on
dvtvm0302: 
> 0 (ok) | call=27 key=application_start_0 confirmed=true cib‑update=100
> Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Forwarding cib_modify operation for section status
to 
> all (origin=local/crmd/100)
> Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: ‑‑‑ 0.29.79 2
> Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: +++ 0.29.80 (null)
> Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  /cib:  @num_updates=80
> Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  
>
/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='
> application']/lrm_rsc_op[@id='application_last_0']:  
> @transition‑magic=0:0;15:10:0:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076,
@call‑id=27, 
> @rc‑code=0, @op‑status=0, @exec‑time=50
> Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Completed cib_modify operation for section status:
OK 
> (rc=0, origin=dvtvm0302/crmd/100, version=0.29.80)
> Jul 20 07:46:54 [1569] DVTVM0302.mse.am.mot.com       crmd:     info: 
> do_lrm_rsc_op:    Performing
key=16:10:0:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076 
> op=application_monitor_1000
> Jul 20 07:46:54 [1569] DVTVM0302.mse.am.mot.com       crmd:     info: 
> action_synced_wait:    Managed application_meta‑data_0 process 9629 exited 
> with rc=4
> Jul 20 07:46:54 [1569] DVTVM0302.mse.am.mot.com       crmd:    error: 
> lrmd_api_get_metadata:    Failed to receive meta‑data for 
> ocf:vendor:application
> Jul 20 07:46:54 [1569] DVTVM0302.mse.am.mot.com       crmd:  warning: 
> build_operation_update:    Failed to get metadata for application 
> (ocf:vendor:application)
> Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Forwarding cib_modify operation for section status
to 
> all (origin=local/crmd/101)
> Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: ‑‑‑ 0.29.80 2
> Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: +++ 0.29.81 (null)
> Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  /cib:  @num_updates=81
> Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  
>
/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='
> application']/lrm_rsc_op[@id='application_monitor_1000']:  
> @crm‑debug‑origin=do_update_resource, 
> @transition‑key=16:10:0:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076, 
> @transition‑magic=‑1:193;16:10:0:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076,
@call‑id=‑1, 
> @rc‑code=193, @op‑status=‑1, @last‑rc‑change=1532087214, @exec‑time=0
> Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Completed cib_modify operation for section status:
OK 
> (rc=0, origin=dvtvm0302/crmd/101, version=0.29.81)
> Jul 20 07:46:54 [1569] DVTVM0302.mse.am.mot.com       crmd:     info: 
> process_lrm_event:    Result of monitor operation for ClusterIP on
dvtvm0302: 
> 0 (ok) | call=26 key=ClusterIP_monitor_1000 confirmed=false cib‑update=102
> Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Forwarding cib_modify operation for section status
to 
> all (origin=local/crmd/102)
> Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: ‑‑‑ 0.29.81 2
> Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: +++ 0.29.82 (null)
> Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  /cib:  @num_updates=82
> Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  
>
/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='
> ClusterIP']/lrm_rsc_op[@id='ClusterIP_monitor_1000']:  
> @transition‑magic=0:0;6:10:0:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076,
@call‑id=26, 
> @rc‑code=0, @op‑status=0, @exec‑time=88
> Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Completed cib_modify operation for section status:
OK 
> (rc=0, origin=dvtvm0302/crmd/102, version=0.29.82)
> Jul 20 07:46:54 [1569] DVTVM0302.mse.am.mot.com       crmd:     info: 
> action_synced_wait:    Managed application_meta‑data_0 process 9654 exited 
> with rc=4
> Jul 20 07:46:54 [1569] DVTVM0302.mse.am.mot.com       crmd:    error: 
> lrmd_api_get_metadata:    Failed to receive meta‑data for 
> ocf:vendor:application
> Jul 20 07:46:54 [1569] DVTVM0302.mse.am.mot.com       crmd:  warning: 
> build_operation_update:    Failed to get metadata for application 
> (ocf:vendor:application)
> Jul 20 07:46:54 [1569] DVTVM0302.mse.am.mot.com       crmd:     info: 
> process_lrm_event:    Result of monitor operation for application on 
> dvtvm0302: 0 (ok) | call=28 key=application_monitor_1000 confirmed=false 
> cib‑update=103
> Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Forwarding cib_modify operation for section status
to 
> all (origin=local/crmd/103)
> Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: ‑‑‑ 0.29.82 2
> Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    Diff: +++ 0.29.83 (null)
> Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  /cib:  @num_updates=83
> Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_perform_op:    +  
>
/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='
> application']/lrm_rsc_op[@id='application_monitor_1000']:  
> @transition‑magic=0:0;16:10:0:4ad59ad3‑3a70‑4282‑a52c‑1f14cce6b076,
@call‑id=28, 
> @rc‑code=0, @op‑status=0, @exec‑time=28
> Jul 20 07:46:54 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_request:    Completed cib_modify operation for section status:
OK 
> (rc=0, origin=dvtvm0302/crmd/103, version=0.29.83)
> Jul 20 07:46:59 [1563] DVTVM0302.mse.am.mot.com        cib:     info: 
> cib_process_ping:    Reporting our current digest to dvtvm0303: 
> ed20687114e2def7cb9106c09ff45a07 for 0.29.83 (0x56163caf1220 0)






More information about the Users mailing list