[ClusterLabs] Resource switchover taking more time upon shutting off one of the node in a 2 node cluster
avinash sharma
avi.dssd at gmail.com
Fri Feb 23 05:45:11 EST 2018
Subject: Switchover of resource(MS) 'RoutingManager' and resource group
'floatingips', which have 'colocation' and 'after' constraints on each
other, are taking around 5 minutes to get promoted when node running master
instance goes down.
The floatingips group consist of IPADDR2 resources: natomsgwip, omsip,
nataaagwip, aaaip, nataccessgwip, accessip, natcpcoregwip and cpcoreip.
Setup details: There are 2 nodes in the pacemaker cluster, IVM-0 and IVM-1
Resource: RoutingManager is a MS resource and can run on both nodes, and
initially it is running as master on IVM-0.
Pacemaker version : 1.1.16
# corosync -v
Corosync Cluster Engine, version '2.4.2'
Copyright (c) 2006-2009 Red Hat, Inc.
# pacemakerd --version
Pacemaker 1.1.16
Written by Andrew Beekhof
Test steps:
1.Shut off IVM-0 at 21:37:00
expected behaviour:
All the resources running on IVM-0 should switch-over to IVM-1 immediately
including RoutingManager and floatingips.
Actual behaviour:
Switchover of all resources was immediate except following ones:
RoutingManager (MS) and aaaip, nataccessgwip, accessip, natcpcoregwip,
cpcoreip from floatingips resource group.
These resources were started after 5 minutes at 21:42:26.
>From corosync logs, I can see that pengine listed all the required actions
which are exactly as needed, but crmd has just ignored above mentioned 6
resources at 21:37:27 and these were taken up later at 21:42:26.
Could you please help me understand why it took more time to promote the
resources ?
resource configuration:
node 1: IVM-0
node 2: IVM-1
primitive ClusterMon ocf:pacemaker:ClusterMon \
params user=root update=30 extra_options="-E
/opt/vplat/modules/pacemaker/clusterMon.sh" \
op monitor on-fail=restart interval=60
*primitive RoutingManager ocf:pacemaker:RoutingManagerRA \*
* op monitor role=Master interval=20 timeout=20 \*
* op monitor role=Slave interval=30 timeout=20*
primitive aaaip IPaddr2 \
params ip=10.53.201.176 nic=aaa cidr_netmask=27 broadcast="+" \
op monitor timeout=20s interval=10s
primitive accessip IPaddr2 \
params ip=10.53.201.144 nic=access cidr_netmask=27 broadcast="+" \
op monitor timeout=20s interval=10s
primitive alarmSnmpd ocf:pacemaker:alarmSnmpdRA \
op monitor role=Master interval=20 timeout=20 \
op monitor role=Slave interval=30 timeout=20
primitive cpcoreip IPaddr2 \
params ip=10.53.201.240 nic=cpcore cidr_netmask=27 broadcast="+" \
op monitor timeout=20s interval=10s
primitive haproxy ocf:pacemaker:haproxyRA \
op monitor role=Master interval=20 timeout=20 \
op monitor role=Slave interval=30 timeout=20
primitive nataaagwip IPaddr2 \
params ip=192.168.2.1 nic=nataaa cidr_netmask=24 broadcast="+" \
op monitor timeout=20s interval=10s
primitive nataaaip IPaddr2 \
params ip=192.168.2.2 nic=nataaa cidr_netmask=24 broadcast="+" \
op monitor timeout=20s interval=10s
primitive nataccessgwip IPaddr2 \
params ip=192.168.3.1 nic=nataccess cidr_netmask=24 broadcast="+" \
op monitor timeout=20s interval=10s
primitive nataccessip IPaddr2 \
params ip=192.168.3.2 nic=nataccess cidr_netmask=24 broadcast="+" \
op monitor timeout=20s interval=10s
primitive natcpcoregwip IPaddr2 \
params ip=192.168.4.1 nic=natcpcore cidr_netmask=24 broadcast="+" \
op monitor timeout=20s interval=10s
primitive natcpcoreip IPaddr2 \
params ip=192.168.4.2 nic=natcpcore cidr_netmask=24 broadcast="+" \
op monitor timeout=20s interval=10s
primitive natomsgwip IPaddr2 \
params ip=192.168.1.1 nic=natoms cidr_netmask=24 broadcast="+" \
op monitor timeout=20s interval=10s
primitive natomsip IPaddr2 \
params ip=192.168.1.2 nic=natoms cidr_netmask=24 broadcast="+" \
op monitor timeout=20s interval=10s
primitive natomsip_for_netact IPaddr2 \
params ip=192.168.1.4 nic=natoms cidr_netmask=24 broadcast="+" \
op monitor timeout=20s interval=10s
primitive natomsip_for_snmpd IPaddr2 \
params ip=192.168.1.3 nic=natoms cidr_netmask=24 broadcast="+" \
op monitor timeout=20s interval=10s
primitive netactagent ocf:pacemaker:netactAgentRA \
op monitor role=Master interval=20 timeout=20 \
op monitor role=Slave interval=30 timeout=20
primitive omsip IPaddr2 \
params ip=10.53.201.209 nic=oms cidr_netmask=27 broadcast="+" \
op monitor timeout=20s interval=10s
primitive stateful_apache2 ocf:pacemaker:apache2 \
op start timeout=30s interval=0 \
op promote timeout=30s interval=0 \
op monitor interval=30s role=Master timeout=30s \
op monitor interval=20s role=Slave timeout=20s \
meta resource-stickiness=100 migration-threshold=1
failure-timeout=15s
primitive stateful_consul ocf:pacemaker:ConsulRA \
op start timeout=300s interval=0 \
op promote timeout=300s interval=0 \
op monitor interval=5s role=Master timeout=10s \
op monitor interval=7s role=Slave timeout=10s \
meta resource-stickiness=100 migration-threshold=1
failure-timeout=15s
primitive stateful_wildfly ocf:pacemaker:wildfly \
op start timeout=200s interval=0 \
op promote timeout=300s interval=0 \
op monitor interval=90s role=Master timeout=90s \
op monitor interval=80s role=Slave timeout=100s \
meta resource-stickiness=100 migration-threshold=3
failure-timeout=240s
group floatingips natomsgwip omsip nataaagwip aaaip nataccessgwip accessip
natcpcoregwip cpcoreip
ms RoutingManager_MS RoutingManager \
meta resource-stickiness=100 notify=true master-max=1
interleave=true migration-threshold=4 failure-timeout=60
ms alarmSnmpd_MS alarmSnmpd \
meta resource-stickiness=100 notify=true master-max=1
interleave=true migration-threshold=4 failure-timeout=60
ms apache2_MS stateful_apache2 \
meta resource-stickiness=100 notify=true master-max=1
interleave=true
ms haproxy_MS haproxy \
meta resource-stickiness=100 notify=true master-max=1
interleave=true migration-threshold=4 failure-timeout=60
ms netactagent_MS netactagent \
meta resource-stickiness=100 notify=true master-max=1
interleave=true migration-threshold=4 failure-timeout=60
ms wildfly_MS stateful_wildfly \
meta resource-stickiness=100 notify=true master-max=1
interleave=true target-role=Started
clone ClusterMon-clone ClusterMon \
meta target-role=Started
*order RoutingManager_after_ips Mandatory: floatingips
RoutingManager_MS:promote*
*colocation RoutingManager_with_floatingips inf: floatingips
RoutingManager_MS:Master*
order alarmSnmpd_after_ips Mandatory: natomsip_for_snmpd
alarmSnmpd_MS:promote
colocation alarmSnmpd_with_floatingips inf: natomsip_for_snmpd
alarmSnmpd_MS:Master
order apache2_after_ips Mandatory: natomsip natcpcoreip apache2_MS:promote
colocation apache2_with_floatingips inf: natomsip natcpcoreip
apache2_MS:Master
order haproxy_after_ips Mandatory: nataaaip nataccessip haproxy_MS:promote
colocation haproxy_with_floatingips inf: nataaaip nataccessip
haproxy_MS:Master
order netactagent_after_ips Mandatory: natomsip_for_netact
netactagent_MS:promote
location netactagent_rule_0 netactagent_MS \
rule -inf: #uname eq IVM-0
location netactagent_rule_1 netactagent_MS \
rule -inf: #uname eq IVM-1
colocation netactagent_with_floatingips inf: natomsip_for_netact
netactagent_MS:Master
property cib-bootstrap-options: \
stonith-enabled=false \
no-quorum-policy=ignore \
cluster-recheck-interval=30s \
start-failure-is-fatal=false \
stop-all-resources=false \
have-watchdog=false \
dc-version=1.1.16-94ff4df51a \
cluster-infrastructure=corosync \
cluster-name=hacluster-0
corosync logs:
Feb 21 21:37:25 [24024] IVM-1 crmd: notice: peer_update_callback: Our
peer on the DC (IVM-0) is dead
Feb 21 21:37:25 [24024] IVM-1 crmd: info: erase_status_tag: Deleting
xpath: //node_state[@uname='IVM-0']/transient_attributes
Feb 21 21:37:25 [24020] IVM-1 stonith-ng: notice:
crm_update_peer_state_iter: Node IVM-0 state is now lost | nodeid=1
previous=member source=crm_update_peer_proc
Feb 21 21:37:25 [24020] IVM-1 stonith-ng: info:
crm_reap_dead_member: Removing
node with name IVM-0 and id 1 from membership cache
Feb 21 21:37:25 [24024] IVM-1 crmd: info: pcmk_cpg_membership: Node
2 still member of group crmd (peer=IVM-1, counter=1.0)
Feb 21 21:37:25 [24020] IVM-1 stonith-ng: notice: reap_crm_member: Purged
1 peers with id=1 and/or uname=IVM-0 from the membership cache
Feb 21 21:37:25 [24020] IVM-1 stonith-ng: info: pcmk_cpg_membership: Node
2 still member of group stonith-ng (peer=IVM-1, counter=1.0)
Feb 21 21:37:25 [24024] IVM-1 crmd: notice: do_state_transition: State
transition S_NOT_DC -> S_ELECTION | input=I_ELECTION
cause=C_CRMD_STATUS_CALLBACK origin=peer_update_callback
Feb 21 21:37:25 [24024] IVM-1 crmd: info: update_dc: Unset DC.
Was IVM-0
Feb 21 21:37:25 [24022] IVM-1 attrd: info: pcmk_cpg_membership: Node
1 left group attrd (peer=IVM-0, counter=1.0)
Feb 21 21:37:25 [24022] IVM-1 attrd: info:
crm_update_peer_proc: pcmk_cpg_membership:
Node IVM-0[1] - corosync-cpg is now offline
Feb 21 21:37:25 [24022] IVM-1 attrd: notice:
crm_update_peer_state_iter: Node IVM-0 state is now lost | nodeid=1
previous=member source=crm_update_peer_proc
Feb 21 21:37:25 [24022] IVM-1 attrd: notice: attrd_peer_remove: Removing
all IVM-0 attributes for peer loss
Feb 21 21:37:25 [24019] IVM-1 cib: info: pcmk_cpg_membership: Node
1 left group cib (peer=IVM-0, counter=1.0)
Feb 21 21:37:25 [24022] IVM-1 attrd: notice: attrd_peer_change_cb: Lost
attribute writer IVM-0
Feb 21 21:37:25 [24022] IVM-1 attrd: info:
crm_reap_dead_member: Removing
node with name IVM-0 and id 1 from membership cache
Feb 21 21:37:25 [24019] IVM-1 cib: info:
crm_update_peer_proc: pcmk_cpg_membership:
Node IVM-0[1] - corosync-cpg is now offline
Feb 21 21:37:25 [24022] IVM-1 attrd: notice: reap_crm_member: Purged
1 peers with id=1 and/or uname=IVM-0 from the membership cache
Feb 21 21:37:25 [24022] IVM-1 attrd: info: pcmk_cpg_membership: Node
2 still member of group attrd (peer=IVM-1, counter=1.0)
Feb 21 21:37:25 [24019] IVM-1 cib: notice:
crm_update_peer_state_iter: Node IVM-0 state is now lost | nodeid=1
previous=member source=crm_update_peer_proc
Feb 21 21:37:25 [24019] IVM-1 cib: info:
crm_reap_dead_member: Removing
node with name IVM-0 and id 1 from membership cache
Feb 21 21:37:25 [24019] IVM-1 cib: notice: reap_crm_member: Purged
1 peers with id=1 and/or uname=IVM-0 from the membership cache
Feb 21 21:37:25 [24019] IVM-1 cib: info: pcmk_cpg_membership: Node
2 still member of group cib (peer=IVM-1, counter=1.0)
Feb 21 21:37:25 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_delete operation for section
//node_state[@uname='IVM-0']/transient_attributes
to all (origin=local/crmd/49)
Feb 21 21:37:25 [24001] IVM-1 corosync notice [TOTEM ] A new membership (
169.254.0.5:92) was formed. Members left: 1
Feb 21 21:37:25 [24019] IVM-1 cib: info: crm_cs_flush: Sent 0
CPG messages (1 remaining, last=60): Try again (6)
Feb 21 21:37:25 [23966] IVM-1 pacemakerd: info: pcmk_cpg_membership: Node
1 left group pacemakerd (peer=IVM-0, counter=1.0)
Feb 21 21:37:25 [23966] IVM-1 pacemakerd: info:
crm_update_peer_proc: pcmk_cpg_membership:
Node IVM-0[1] - corosync-cpg is now offline
Feb 21 21:37:25 [23966] IVM-1 pacemakerd: info: pcmk_cpg_membership: Node
2 still member of group pacemakerd (peer=IVM-1, counter=1.0)
Feb 21 21:37:25 [23966] IVM-1 pacemakerd: info: crm_cs_flush: Sent 0
CPG messages (1 remaining, last=7): Try again (6)
Feb 21 21:37:25 [24001] IVM-1 corosync notice [QUORUM] This node is within
the non-primary component and will NOT provide any services.
Feb 21 21:37:25 [24001] IVM-1 corosync notice [QUORUM] Members[1]: 2
Feb 21 21:37:25 [23966] IVM-1 pacemakerd: warning:
pcmk_quorum_notification: Quorum lost | membership=92 members=1
Feb 21 21:37:25 [24001] IVM-1 corosync notice [MAIN ] Completed service
synchronization, ready to provide service.
Feb 21 21:37:25 [23966] IVM-1 pacemakerd: notice:
crm_update_peer_state_iter: Node IVM-0 state is now lost | nodeid=1
previous=member source=crm_reap_unseen_nodes
Feb 21 21:37:25 [24024] IVM-1 crmd: warning:
pcmk_quorum_notification: Quorum lost | membership=92 members=1
Feb 21 21:37:25 [24024] IVM-1 crmd: notice:
crm_update_peer_state_iter: Node IVM-0 state is now lost | nodeid=1
previous=member source=crm_reap_unseen_nodes
Feb 21 21:37:25 [24024] IVM-1 crmd: info:
peer_update_callback: Cluster
node IVM-0 is now lost (was member)
Feb 21 21:37:25 [24024] IVM-1 crmd: info: election_complete: Election
election-0 complete
Feb 21 21:37:25 [24024] IVM-1 crmd: info:
election_timeout_popped: Election
failed: Declaring ourselves the winner
Feb 21 21:37:25 [24024] IVM-1 crmd: info: do_log: Input
I_ELECTION_DC received in state S_ELECTION from election_timeout_popped
Feb 21 21:37:25 [24024] IVM-1 crmd: notice: do_state_transition: State
transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC
cause=C_TIMER_POPPED origin=election_timeout_popped
Feb 21 21:37:25 [24024] IVM-1 crmd: info: do_te_control: Registering
TE UUID: e27e11f8-fc4e-493e-a4f2-80a02cb1f079
Feb 21 21:37:25 [24024] IVM-1 crmd: info:
set_graph_functions: Setting
custom graph functions
Feb 21 21:37:25 [24024] IVM-1 crmd: info: do_dc_takeover: Taking
over DC status for this partition
Feb 21 21:37:25 [24019] IVM-1 cib: info: cib_process_readwrite: We
are now in R/W mode
Feb 21 21:37:25 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_master operation for section 'all': OK (rc=0, origin=local/crmd/50,
version=0.108.0)
Feb 21 21:37:25 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section cib to all (origin=local/crmd/51)
Feb 21 21:37:25 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section crm_config to all (origin=local/crmd/53)
Feb 21 21:37:25 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section crm_config to all (origin=local/crmd/55)
Feb 21 21:37:25 [24024] IVM-1 crmd: info: join_make_offer: Not
making an offer to IVM-0: not active (lost)
Feb 21 21:37:25 [24024] IVM-1 crmd: info: join_make_offer: Making
join offers based on membership 92
Feb 21 21:37:25 [24024] IVM-1 crmd: info: join_make_offer: join-1:
Sending offer to IVM-1
Feb 21 21:37:25 [24024] IVM-1 crmd: info:
crm_update_peer_join: join_make_offer:
Node IVM-1[2] - join-1 phase 0 -> 1
Feb 21 21:37:25 [24024] IVM-1 crmd: info:
do_dc_join_offer_all: join-1:
Waiting on 1 outstanding join acks
Feb 21 21:37:25 [24024] IVM-1 crmd: warning: do_log: Input
I_ELECTION_DC received in state S_INTEGRATION from do_election_check
Feb 21 21:37:25 [24024] IVM-1 crmd: info:
crm_update_peer_join: initialize_join:
Node IVM-1[2] - join-2 phase 1 -> 0
Feb 21 21:37:25 [24024] IVM-1 crmd: info: join_make_offer: Not
making an offer to IVM-0: not active (lost)
Feb 21 21:37:25 [24024] IVM-1 crmd: info: join_make_offer: join-2:
Sending offer to IVM-1
Feb 21 21:37:25 [24024] IVM-1 crmd: info:
crm_update_peer_join: join_make_offer:
Node IVM-1[2] - join-2 phase 0 -> 1
Feb 21 21:37:25 [24024] IVM-1 crmd: info:
do_dc_join_offer_all: join-2:
Waiting on 1 outstanding join acks
Feb 21 21:37:25 [24024] IVM-1 crmd: info: update_dc: Set DC to
IVM-1 (3.0.11)
Feb 21 21:37:25 [24024] IVM-1 crmd: info:
crm_update_peer_expected: update_dc: Node IVM-1[2] - expected state is now
member (was (null))
Feb 21 21:37:25 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section crm_config to all (origin=local/crmd/57)
Feb 21 21:37:25 [24024] IVM-1 crmd: info: parse_notifications: We
have an alerts section in the cib
Feb 21 21:37:25 [24024] IVM-1 crmd: info:
crm_update_peer_join: do_dc_join_filter_offer:
Node IVM-1[2] - join-2 phase 1 -> 2
Feb 21 21:37:25 [24024] IVM-1 crmd: info: do_state_transition: State
transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED
cause=C_FSA_INTERNAL origin=check_join_state
Feb 21 21:37:25 [24024] IVM-1 crmd: info:
crmd_join_phase_log: join-2:
IVM-0=none
Feb 21 21:37:25 [24024] IVM-1 crmd: info:
crmd_join_phase_log: join-2:
IVM-1=integrated
Feb 21 21:37:25 [24024] IVM-1 crmd: info:
do_dc_join_finalize: join-2:
Syncing our CIB to the rest of the cluster
Feb 21 21:37:25 [24024] IVM-1 crmd: info:
crm_update_peer_join: finalize_join_for:
Node IVM-1[2] - join-2 phase 2 -> 3
Feb 21 21:37:25 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section nodes to all (origin=local/crmd/62)
Feb 21 21:37:25 [24024] IVM-1 crmd: info: action_synced_wait: Managed
alarmSnmpdRA_meta-data_0 process 32216 exited with rc=0
Feb 21 21:37:25 [24024] IVM-1 crmd: info: action_synced_wait: Managed
wildfly_meta-data_0 process 32227 exited with rc=0
Feb 21 21:37:25 [24024] IVM-1 crmd: info: action_synced_wait: Managed
ClusterMon_meta-data_0 process 32262 exited with rc=0
Feb 21 21:37:25 [24019] IVM-1 cib: info: crm_cs_flush: Sent 7
CPG messages (0 remaining, last=67): OK (1)
Feb 21 21:37:25 [23966] IVM-1 pacemakerd: info: mcp_cpg_deliver: Ignoring
process list sent by peer for local node
Feb 21 21:37:25 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.0 2
Feb 21 21:37:25 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.1 (null)
Feb 21 21:37:25 [24019] IVM-1 cib: info: cib_perform_op: --
/cib/status/node_state[@id='1']/transient_attributes[@id='1']
Feb 21 21:37:25 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=1
Feb 21 21:37:25 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_delete operation for section
//node_state[@uname='IVM-0']/transient_attributes:
OK (rc=0, origin=IVM-1/crmd/49, version=0.108.1)
Feb 21 21:37:25 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section cib: OK (rc=0, origin=IVM-1/crmd/51,
version=0.108.1)
Feb 21 21:37:25 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section crm_config: OK (rc=0,
origin=IVM-1/crmd/53, version=0.108.1)
Feb 21 21:37:25 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section crm_config: OK (rc=0,
origin=IVM-1/crmd/55, version=0.108.1)
Feb 21 21:37:25 [24024] IVM-1 crmd: info: action_synced_wait: Managed
ConsulRA_meta-data_0 process 32270 exited with rc=0
Feb 21 21:37:25 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section crm_config: OK (rc=0,
origin=IVM-1/crmd/57, version=0.108.1)
Feb 21 21:37:25 [24019] IVM-1 cib: info: cib_process_replace: Digest
matched on replace from IVM-1: 8bb36a1ee9ffcb07b8c775a4711a79d2
Feb 21 21:37:25 [24019] IVM-1 cib: info:
cib_process_replace: Replacement
0.108.0 from IVM-1 not applied to 0.108.1: current num_updates is greater
than the replacement
Feb 21 21:37:25 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section nodes: OK (rc=0, origin=IVM-1/crmd/62,
version=0.108.1)
Feb 21 21:37:25 [24024] IVM-1 crmd: info: action_synced_wait: Managed
apache2_meta-data_0 process 32323 exited with rc=0
Feb 21 21:37:25 [24024] IVM-1 crmd: info: action_synced_wait: Managed
RoutingManagerRA_meta-data_0 process 32355 exited with rc=0
Feb 21 21:37:25 [24024] IVM-1 crmd: info:
crm_update_peer_join: do_dc_join_ack:
Node IVM-1[2] - join-2 phase 3 -> 4
Feb 21 21:37:25 [24024] IVM-1 crmd: info: do_dc_join_ack: join-2:
Updating node state to member for IVM-1
Feb 21 21:37:25 [24024] IVM-1 crmd: info: erase_status_tag: Deleting
xpath: //node_state[@uname='IVM-1']/lrm
Feb 21 21:37:25 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_delete operation for section //node_state[@uname='IVM-1']/lrm to all
(origin=local/crmd/63)
Feb 21 21:37:25 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.1 2
Feb 21 21:37:25 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.2 (null)
Feb 21 21:37:25 [24019] IVM-1 cib: info: cib_perform_op: --
/cib/status/node_state[@id='2']/lrm[@id='2']
Feb 21 21:37:25 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=2
Feb 21 21:37:25 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_delete operation for section //node_state[@uname='IVM-1']/lrm: OK
(rc=0, origin=IVM-1/crmd/63, version=0.108.2)
Feb 21 21:37:25 [24024] IVM-1 crmd: info: action_synced_wait: Managed
alarmSnmpdRA_meta-data_0 process 32362 exited with rc=0
Feb 21 21:37:26 [24024] IVM-1 crmd: info: action_synced_wait: Managed
wildfly_meta-data_0 process 32369 exited with rc=0
Feb 21 21:37:26 [24024] IVM-1 crmd: info: action_synced_wait: Managed
ClusterMon_meta-data_0 process 32405 exited with rc=0
Feb 21 21:37:26 [24024] IVM-1 crmd: info: action_synced_wait: Managed
ConsulRA_meta-data_0 process 32410 exited with rc=0
Feb 21 21:37:26 [24024] IVM-1 crmd: info: action_synced_wait: Managed
apache2_meta-data_0 process 32464 exited with rc=0
Feb 21 21:37:26 [24024] IVM-1 crmd: info: action_synced_wait: Managed
RoutingManagerRA_meta-data_0 process 32495 exited with rc=0
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/crmd/64)
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.2 2
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.3 (null)
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=3
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib/status/node_state[@id='2']: @crm-debug-origin=do_lrm_query_internal
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
/cib/status/node_state[@id='2']: <lrm id="2"/>
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_resources>
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_resource id="alarmSnmpd"
type="alarmSnmpdRA" class="ocf" provider="pacemaker">
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_rsc_op id="alarmSnmpd_last_0"
operation_key="alarmSnmpd_start_0" operation="start"
crm-debug-origin="build_active_RAs" crm_feature_set="3.0.11"
transition-key="150:125:0:469bda8b-40fa-427e-8e66-d8e9ec19374f"
transition-magic="0:0;150:125:0:469bda8b-40fa-427e-8e66-d8e9ec19374f"
on_node="IVM-1" call-id="78" rc-code="0" op-status="0" interval="0"
last-run="1519228538" last-rc-change="1519228538"
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_rsc_op
id="alarmSnmpd_monitor_30000" operation_key="alarmSnmpd_monitor_30000"
operation="monitor" crm-debug-origin="build_active_RAs"
crm_feature_set="3.0.11"
transition-key="131:126:0:469bda8b-40fa-427e-8e66-d8e9ec19374f"
transition-magic="0:0;131:126:0:469bda8b-40fa-427e-8e66-d8e9ec19374f"
on_node="IVM-1" call-id="113" rc-code="0" op-status="0" interval="30000"
last-rc-change="1519228608" e
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
</lrm_resource>
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_resource id="nataccessgwip"
type="IPaddr2" class="ocf" provider="heartbeat">
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_rsc_op id="nataccessgwip_last_0"
operation_key="nataccessgwip_monitor_0" operation="monitor"
crm-debug-origin="build_active_RAs" crm_feature_set="3.0.11"
transition-key="40:125:7:469bda8b-40fa-427e-8e66-d8e9ec19374f"
transition-magic="0:7;40:125:7:469bda8b-40fa-427e-8e66-d8e9ec19374f"
on_node="IVM-1" call-id="88" rc-code="7" op-status="0" interval="0"
last-run="1519228538" last-rc-change="151
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
</lrm_resource>
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_resource id="haproxy"
type="haproxyRA" class="ocf" provider="pacemaker">
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_rsc_op id="haproxy_last_failure_0"
operation_key="haproxy_monitor_0" operation="monitor"
crm-debug-origin="build_active_RAs" crm_feature_set="3.0.11"
transition-key="23:126:7:469bda8b-40fa-427e-8e66-d8e9ec19374f"
transition-magic="0:0;23:126:7:469bda8b-40fa-427e-8e66-d8e9ec19374f"
on_node="IVM-1" call-id="114" rc-code="0" op-status="0" interval="0"
last-run="1519228608" last-rc-change="151922
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_rsc_op id="haproxy_last_0"
operation_key="haproxy_monitor_0" operation="monitor"
crm-debug-origin="build_active_RAs" crm_feature_set="3.0.11"
transition-key="30:125:7:469bda8b-40fa-427e-8e66-d8e9ec19374f"
transition-magic="0:7;30:125:7:469bda8b-40fa-427e-8e66-d8e9ec19374f"
on_node="IVM-1" call-id="43" rc-code="7" op-status="0" interval="0"
last-run="1519228537" last-rc-change="1519228537" exe
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
</lrm_resource>
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_resource id="nataaaip"
type="IPaddr2" class="ocf" provider="heartbeat">
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_rsc_op id="nataaaip_last_0"
operation_key="nataaaip_monitor_0" operation="monitor"
crm-debug-origin="build_active_RAs" crm_feature_set="3.0.11"
transition-key="31:125:7:469bda8b-40fa-427e-8e66-d8e9ec19374f"
transition-magic="0:7;31:125:7:469bda8b-40fa-427e-8e66-d8e9ec19374f"
on_node="IVM-1" call-id="47" rc-code="7" op-status="0" interval="0"
last-run="1519228537" last-rc-change="1519228537" e
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
</lrm_resource>
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_resource id="natomsip"
type="IPaddr2" class="ocf" provider="heartbeat">
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_rsc_op id="natomsip_last_0"
operation_key="natomsip_monitor_0" operation="monitor"
crm-debug-origin="build_active_RAs" crm_feature_set="3.0.11"
transition-key="23:125:7:469bda8b-40fa-427e-8e66-d8e9ec19374f"
transition-magic="0:7;23:125:7:469bda8b-40fa-427e-8e66-d8e9ec19374f"
on_node="IVM-1" call-id="10" rc-code="7" op-status="0" interval="0"
last-run="1519228537" last-rc-change="1519228537" e
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
</lrm_resource>
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_resource id="stateful_wildfly"
type="wildfly" class="ocf" provider="pacemaker">
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_rsc_op
id="stateful_wildfly_last_0" operation_key="stateful_wildfly_start_0"
operation="start" crm-debug-origin="build_active_RAs"
crm_feature_set="3.0.11"
transition-key="120:125:0:469bda8b-40fa-427e-8e66-d8e9ec19374f"
transition-magic="0:0;120:125:0:469bda8b-40fa-427e-8e66-d8e9ec19374f"
on_node="IVM-1" call-id="93" rc-code="0" op-status="0" interval="0"
last-run="1519228538" last-rc-change=
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_rsc_op
id="stateful_wildfly_monitor_80000"
operation_key="stateful_wildfly_monitor_80000"
operation="monitor" crm-debug-origin="build_active_RAs"
crm_feature_set="3.0.11"
transition-key="100:126:0:469bda8b-40fa-427e-8e66-d8e9ec19374f"
transition-magic="0:0;100:126:0:469bda8b-40fa-427e-8e66-d8e9ec19374f"
on_node="IVM-1" call-id="112" rc-code="0" op-status="0" interval="80000"
last-rc-change="1
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
</lrm_resource>
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_resource id="ClusterMon"
type="ClusterMon" class="ocf" provider="pacemaker">
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_rsc_op id="ClusterMon_last_0"
operation_key="ClusterMon_start_0" operation="start"
crm-debug-origin="build_active_RAs" crm_feature_set="3.0.11"
transition-key="61:126:0:469bda8b-40fa-427e-8e66-d8e9ec19374f"
transition-magic="0:0;61:126:0:469bda8b-40fa-427e-8e66-d8e9ec19374f"
on_node="IVM-1" call-id="116" rc-code="0" op-status="0" interval="0"
last-run="1519228608" last-rc-change="1519228608"
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_rsc_op
id="ClusterMon_monitor_60000" operation_key="ClusterMon_monitor_60000"
operation="monitor" crm-debug-origin="build_active_RAs"
crm_feature_set="3.0.11"
transition-key="63:127:0:469bda8b-40fa-427e-8e66-d8e9ec19374f"
transition-magic="0:0;63:127:0:469bda8b-40fa-427e-8e66-d8e9ec19374f"
on_node="IVM-1" call-id="117" rc-code="0" op-status="0" interval="60000"
last-rc-change="1519228609" exe
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
</lrm_resource>
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_resource id="stateful_consul"
type="ConsulRA" class="ocf" provider="pacemaker">
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_rsc_op id="stateful_consul_last_0"
operation_key="stateful_consul_start_0" operation="start"
crm-debug-origin="build_active_RAs" crm_feature_set="3.0.11"
transition-key="90:125:0:469bda8b-40fa-427e-8e66-d8e9ec19374f"
transition-magic="0:0;90:125:0:469bda8b-40fa-427e-8e66-d8e9ec19374f"
on_node="IVM-1" call-id="103" rc-code="0" op-status="0" interval="0"
last-run="1519228538" last-rc-change="15
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_rsc_op
id="stateful_consul_monitor_7000"
operation_key="stateful_consul_monitor_7000" operation="monitor"
crm-debug-origin="build_active_RAs" crm_feature_set="3.0.11"
transition-key="69:126:0:469bda8b-40fa-427e-8e66-d8e9ec19374f"
transition-magic="0:0;69:126:0:469bda8b-40fa-427e-8e66-d8e9ec19374f"
on_node="IVM-1" call-id="111" rc-code="0" op-status="0" interval="7000"
last-rc-change="15192286
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
</lrm_resource>
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_resource id="cpcoreip"
type="IPaddr2" class="ocf" provider="heartbeat">
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_rsc_op id="cpcoreip_last_0"
operation_key="cpcoreip_monitor_0" operation="monitor"
crm-debug-origin="build_active_RAs" crm_feature_set="3.0.11"
transition-key="43:125:7:469bda8b-40fa-427e-8e66-d8e9ec19374f"
transition-magic="0:7;43:125:7:469bda8b-40fa-427e-8e66-d8e9ec19374f"
on_node="IVM-1" call-id="102" rc-code="7" op-status="0" interval="0"
last-run="1519228538" last-rc-change="1519228538"
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
</lrm_resource>
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_resource id="natomsgwip"
type="IPaddr2" class="ocf" provider="heartbeat">
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_rsc_op id="natomsgwip_last_0"
operation_key="natomsgwip_monitor_0" operation="monitor"
crm-debug-origin="build_active_RAs" crm_feature_set="3.0.11"
transition-key="36:125:7:469bda8b-40fa-427e-8e66-d8e9ec19374f"
transition-magic="0:7;36:125:7:469bda8b-40fa-427e-8e66-d8e9ec19374f"
on_node="IVM-1" call-id="69" rc-code="7" op-status="0" interval="0"
last-run="1519228537" last-rc-change="151922853
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
</lrm_resource>
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_resource id="stateful_apache2"
type="apache2" class="ocf" provider="pacemaker">
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_rsc_op
id="stateful_apache2_last_0" operation_key="stateful_apache2_start_0"
operation="start" crm-debug-origin="build_active_RAs"
crm_feature_set="3.0.11"
transition-key="48:125:0:469bda8b-40fa-427e-8e66-d8e9ec19374f"
transition-magic="0:0;48:125:0:469bda8b-40fa-427e-8e66-d8e9ec19374f"
on_node="IVM-1" call-id="98" rc-code="0" op-status="0" interval="0"
last-run="1519228538" last-rc-change="1
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_rsc_op
id="stateful_apache2_monitor_20000"
operation_key="stateful_apache2_monitor_20000"
operation="monitor" crm-debug-origin="build_active_RAs"
crm_feature_set="3.0.11"
transition-key="26:126:0:469bda8b-40fa-427e-8e66-d8e9ec19374f"
transition-magic="0:0;26:126:0:469bda8b-40fa-427e-8e66-d8e9ec19374f"
on_node="IVM-1" call-id="110" rc-code="0" op-status="0" interval="20000"
last-rc-change="151
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
</lrm_resource>
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_resource id="omsip" type="IPaddr2"
class="ocf" provider="heartbeat">
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_rsc_op id="omsip_last_0"
operation_key="omsip_monitor_0" operation="monitor"
crm-debug-origin="build_active_RAs"
crm_feature_set="3.0.11"
transition-key="37:125:7:469bda8b-40fa-427e-8e66-d8e9ec19374f"
transition-magic="0:7;37:125:7:469bda8b-40fa-427e-8e66-d8e9ec19374f"
on_node="IVM-1" call-id="73" rc-code="7" op-status="0" interval="0"
last-run="1519228537" last-rc-change="1519228537" exec-ti
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
</lrm_resource>
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_resource id="natomsip_for_netact"
type="IPaddr2" class="ocf" provider="heartbeat">
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_rsc_op
id="natomsip_for_netact_last_0" operation_key="natomsip_for_netact_monitor_0"
operation="monitor" crm-debug-origin="build_active_RAs"
crm_feature_set="3.0.11"
transition-key="34:125:7:469bda8b-40fa-427e-8e66-d8e9ec19374f"
transition-magic="0:7;34:125:7:469bda8b-40fa-427e-8e66-d8e9ec19374f"
on_node="IVM-1" call-id="60" rc-code="7" op-status="0" interval="0"
last-run="1519228537" last-rc
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
</lrm_resource>
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_resource id="netactagent"
type="netactAgentRA" class="ocf" provider="pacemaker">
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_rsc_op id="netactagent_last_0"
operation_key="netactagent_monitor_0" operation="monitor"
crm-debug-origin="build_active_RAs" crm_feature_set="3.0.11"
transition-key="33:125:7:469bda8b-40fa-427e-8e66-d8e9ec19374f"
transition-magic="0:7;33:125:7:469bda8b-40fa-427e-8e66-d8e9ec19374f"
on_node="IVM-1" call-id="56" rc-code="7" op-status="0" interval="0"
last-run="1519228537" last-rc-change="1519228
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
</lrm_resource>
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_resource id="RoutingManager"
type="RoutingManagerRA" class="ocf" provider="pacemaker">
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_rsc_op id="RoutingManager_last_0"
operation_key="RoutingManager_start_0" operation="start"
crm-debug-origin="build_active_RAs" crm_feature_set="3.0.11"
transition-key="240:125:0:469bda8b-40fa-427e-8e66-d8e9ec19374f"
transition-magic="0:0;240:125:0:469bda8b-40fa-427e-8e66-d8e9ec19374f"
on_node="IVM-1" call-id="79" rc-code="0" op-status="0" interval="0"
last-run="1519228538" last-rc-change="151
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_rsc_op
id="RoutingManager_monitor_30000"
operation_key="RoutingManager_monitor_30000" operation="monitor"
crm-debug-origin="build_active_RAs" crm_feature_set="3.0.11"
transition-key="222:126:0:469bda8b-40fa-427e-8e66-d8e9ec19374f"
transition-magic="0:0;222:126:0:469bda8b-40fa-427e-8e66-d8e9ec19374f"
on_node="IVM-1" call-id="115" rc-code="0" op-status="0" interval="30000"
last-rc-change="15192
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
</lrm_resource>
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_resource id="accessip"
type="IPaddr2" class="ocf" provider="heartbeat">
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_rsc_op id="accessip_last_0"
operation_key="accessip_monitor_0" operation="monitor"
crm-debug-origin="build_active_RAs" crm_feature_set="3.0.11"
transition-key="41:125:7:469bda8b-40fa-427e-8e66-d8e9ec19374f"
transition-magic="0:7;41:125:7:469bda8b-40fa-427e-8e66-d8e9ec19374f"
on_node="IVM-1" call-id="92" rc-code="7" op-status="0" interval="0"
last-run="1519228538" last-rc-change="1519228538" e
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
</lrm_resource>
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_resource id="aaaip" type="IPaddr2"
class="ocf" provider="heartbeat">
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_rsc_op id="aaaip_last_0"
operation_key="aaaip_monitor_0" operation="monitor"
crm-debug-origin="build_active_RAs"
crm_feature_set="3.0.11"
transition-key="39:125:7:469bda8b-40fa-427e-8e66-d8e9ec19374f"
transition-magic="0:7;39:125:7:469bda8b-40fa-427e-8e66-d8e9ec19374f"
on_node="IVM-1" call-id="84" rc-code="7" op-status="0" interval="0"
last-run="1519228538" last-rc-change="1519228538" exec-ti
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
</lrm_resource>
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_resource id="nataccessip"
type="IPaddr2" class="ocf" provider="heartbeat">
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_rsc_op id="nataccessip_last_0"
operation_key="nataccessip_monitor_0" operation="monitor"
crm-debug-origin="build_active_RAs" crm_feature_set="3.0.11"
transition-key="32:125:7:469bda8b-40fa-427e-8e66-d8e9ec19374f"
transition-magic="0:7;32:125:7:469bda8b-40fa-427e-8e66-d8e9ec19374f"
on_node="IVM-1" call-id="51" rc-code="7" op-status="0" interval="0"
last-run="1519228537" last-rc-change="1519228
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
</lrm_resource>
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_resource id="natomsip_for_snmpd"
type="IPaddr2" class="ocf" provider="heartbeat">
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_rsc_op
id="natomsip_for_snmpd_last_0" operation_key="natomsip_for_snmpd_monitor_0"
operation="monitor" crm-debug-origin="build_active_RAs"
crm_feature_set="3.0.11"
transition-key="29:125:7:469bda8b-40fa-427e-8e66-d8e9ec19374f"
transition-magic="0:7;29:125:7:469bda8b-40fa-427e-8e66-d8e9ec19374f"
on_node="IVM-1" call-id="38" rc-code="7" op-status="0" interval="0"
last-run="1519228537" last-rc-c
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
</lrm_resource>
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_resource id="natcpcoreip"
type="IPaddr2" class="ocf" provider="heartbeat">
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_rsc_op id="natcpcoreip_last_0"
operation_key="natcpcoreip_monitor_0" operation="monitor"
crm-debug-origin="build_active_RAs" crm_feature_set="3.0.11"
transition-key="24:125:7:469bda8b-40fa-427e-8e66-d8e9ec19374f"
transition-magic="0:7;24:125:7:469bda8b-40fa-427e-8e66-d8e9ec19374f"
on_node="IVM-1" call-id="14" rc-code="7" op-status="0" interval="0"
last-run="1519228537" last-rc-change="1519228
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
</lrm_resource>
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_resource id="natcpcoregwip"
type="IPaddr2" class="ocf" provider="heartbeat">
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_rsc_op id="natcpcoregwip_last_0"
operation_key="natcpcoregwip_monitor_0" operation="monitor"
crm-debug-origin="build_active_RAs" crm_feature_set="3.0.11"
transition-key="42:125:7:469bda8b-40fa-427e-8e66-d8e9ec19374f"
transition-magic="0:7;42:125:7:469bda8b-40fa-427e-8e66-d8e9ec19374f"
on_node="IVM-1" call-id="97" rc-code="7" op-status="0" interval="0"
last-run="1519228538" last-rc-change="151
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
</lrm_resource>
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_resource id="nataaagwip"
type="IPaddr2" class="ocf" provider="heartbeat">
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
<lrm_rsc_op id="nataaagwip_last_0"
operation_key="nataaagwip_monitor_0" operation="monitor"
crm-debug-origin="build_active_RAs" crm_feature_set="3.0.11"
transition-key="38:125:7:469bda8b-40fa-427e-8e66-d8e9ec19374f"
transition-magic="0:7;38:125:7:469bda8b-40fa-427e-8e66-d8e9ec19374f"
on_node="IVM-1" call-id="77" rc-code="7" op-status="0" interval="0"
last-run="1519228538" last-rc-change="151922853
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
</lrm_resource>
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
</lrm_resources>
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
</lrm>
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/crmd/64,
version=0.108.3)
Feb 21 21:37:26 [24024] IVM-1 crmd: info: do_state_transition: State
transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED
cause=C_FSA_INTERNAL origin=check_join_state
Feb 21 21:37:26 [24024] IVM-1 crmd: info:
abort_transition_graph: Transition
aborted: Peer Cancelled | source=do_te_invoke:161 complete=true
Feb 21 21:37:26 [24022] IVM-1 attrd: info:
attrd_client_refresh: Updating
all attributes
Feb 21 21:37:26 [24022] IVM-1 attrd: info: write_attribute: Sent
update 18 with 1 changes for master-RoutingManager, id=<n/a>, set=(null)
Feb 21 21:37:26 [24022] IVM-1 attrd: info: write_attribute: Sent
update 19 with 1 changes for fail-count-haproxy, id=<n/a>, set=(null)
Feb 21 21:37:26 [24022] IVM-1 attrd: info: write_attribute: Sent
update 20 with 1 changes for master-alarmSnmpd, id=<n/a>, set=(null)
Feb 21 21:37:26 [24022] IVM-1 attrd: info: write_attribute: Sent
update 21 with 1 changes for terminate, id=<n/a>, set=(null)
Feb 21 21:37:26 [24022] IVM-1 attrd: info: write_attribute: Sent
update 22 with 1 changes for master-stateful_wildfly, id=<n/a>, set=(null)
Feb 21 21:37:26 [24022] IVM-1 attrd: info: write_attribute: Sent
update 23 with 1 changes for master-stateful_consul, id=<n/a>, set=(null)
Feb 21 21:37:26 [24022] IVM-1 attrd: info: write_attribute: Sent
update 24 with 1 changes for master-haproxy, id=<n/a>, set=(null)
Feb 21 21:37:26 [24022] IVM-1 attrd: info: write_attribute: Sent
update 25 with 1 changes for shutdown, id=<n/a>, set=(null)
Feb 21 21:37:26 [24022] IVM-1 attrd: info: write_attribute: Sent
update 26 with 1 changes for master-stateful_apache2, id=<n/a>, set=(null)
Feb 21 21:37:26 [24022] IVM-1 attrd: info: write_attribute: Sent
update 27 with 1 changes for last-failure-haproxy, id=<n/a>, set=(null)
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section nodes to all (origin=local/crmd/67)
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/crmd/68)
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section cib to all (origin=local/crmd/69)
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/attrd/18)
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/attrd/19)
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/attrd/20)
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/attrd/21)
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/attrd/22)
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section nodes: OK (rc=0, origin=IVM-1/crmd/67,
version=0.108.3)
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.3 2
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.4 (null)
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=4
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib/status/node_state[@id='1']: @in_ccm=false, @crmd=offline,
@join=down, @expected=down
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/crmd/68,
version=0.108.4)
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.4 2
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.5 (null)
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=5, @have-quorum=0, @dc-uuid=2
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section cib: OK (rc=0, origin=IVM-1/crmd/69,
version=0.108.5)
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/attrd/18,
version=0.108.5)
Feb 21 21:37:26 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
18 for master-RoutingManager: OK (0)
Feb 21 21:37:26 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
18 for master-RoutingManager[IVM-1]=5: OK (0)
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.5 2
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.6 (null)
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=6
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/attrd/19,
version=0.108.6)
Feb 21 21:37:26 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
19 for fail-count-haproxy: OK (0)
Feb 21 21:37:26 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
19 for fail-count-haproxy[IVM-1]=(null): OK (0)
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/attrd/20,
version=0.108.6)
Feb 21 21:37:26 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
20 for master-alarmSnmpd: OK (0)
Feb 21 21:37:26 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
20 for master-alarmSnmpd[IVM-1]=5: OK (0)
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.6 2
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.7 (null)
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=7
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/attrd/21,
version=0.108.7)
Feb 21 21:37:26 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
21 for terminate: OK (0)
Feb 21 21:37:26 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
21 for terminate[IVM-1]=(null): OK (0)
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/attrd/22,
version=0.108.7)
Feb 21 21:37:26 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
22 for master-stateful_wildfly: OK (0)
Feb 21 21:37:26 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
22 for master-stateful_wildfly[IVM-1]=5: OK (0)
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/attrd/23)
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/attrd/24)
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/attrd/25)
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/attrd/26)
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/attrd/27)
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/attrd/23,
version=0.108.7)
Feb 21 21:37:26 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
23 for master-stateful_consul: OK (0)
Feb 21 21:37:26 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
23 for master-stateful_consul[IVM-1]=5: OK (0)
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/attrd/24,
version=0.108.7)
Feb 21 21:37:26 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
24 for master-haproxy: OK (0)
Feb 21 21:37:26 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
24 for master-haproxy[IVM-1]=5: OK (0)
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/attrd/25,
version=0.108.7)
Feb 21 21:37:26 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
25 for shutdown: OK (0)
Feb 21 21:37:26 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
25 for shutdown[IVM-1]=0: OK (0)
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/attrd/26,
version=0.108.7)
Feb 21 21:37:26 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
26 for master-stateful_apache2: OK (0)
Feb 21 21:37:26 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
26 for master-stateful_apache2[IVM-1]=5: OK (0)
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.7 2
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.8 (null)
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=8
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/attrd/27,
version=0.108.8)
Feb 21 21:37:26 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
27 for last-failure-haproxy: OK (0)
Feb 21 21:37:26 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
27 for last-failure-haproxy[IVM-1]=(null): OK (0)
Feb 21 21:37:26 [24023] IVM-1 pengine: notice: unpack_config: On loss
of CCM Quorum: Ignore
Feb 21 21:37:26 [24023] IVM-1 pengine: info:
determine_online_status: Node
IVM-1 is online
Feb 21 21:37:26 [24023] IVM-1 pengine: info:
determine_op_status: Operation
monitor found resource haproxy:0 active on IVM-1
Feb 21 21:37:26 [24023] IVM-1 pengine: info: clone_print: Master/Slave
Set: apache2_MS [stateful_apache2]
Feb 21 21:37:26 [24023] IVM-1 pengine: info: short_print:
Slaves: [ IVM-1 ]
Feb 21 21:37:26 [24023] IVM-1 pengine: info: short_print:
Stopped: [ IVM-0 ]
Feb 21 21:37:26 [24023] IVM-1 pengine: info: native_print: natomsip
(ocf::heartbeat:IPaddr2): Stopped
Feb 21 21:37:26 [24023] IVM-1 pengine: info: native_print:
natcpcoreip (ocf::heartbeat:IPaddr2): Stopped
Feb 21 21:37:26 [24023] IVM-1 pengine: info: clone_print: Clone Set:
ClusterMon-clone [ClusterMon]
Feb 21 21:37:26 [24023] IVM-1 pengine: info: short_print:
Started: [ IVM-1 ]
Feb 21 21:37:26 [24023] IVM-1 pengine: info: short_print:
Stopped: [ IVM-0 ]
Feb 21 21:37:26 [24023] IVM-1 pengine: info: clone_print: Master/Slave
Set: stateful_ms_set [stateful_consul]
Feb 21 21:37:26 [24023] IVM-1 pengine: info: short_print:
Slaves: [ IVM-1 ]
Feb 21 21:37:26 [24023] IVM-1 pengine: info: short_print:
Stopped: [ IVM-0 ]
Feb 21 21:37:26 [24023] IVM-1 pengine: info: clone_print: Master/Slave
Set: wildfly_MS [stateful_wildfly]
Feb 21 21:37:26 [24023] IVM-1 pengine: info: short_print:
Slaves: [ IVM-1 ]
Feb 21 21:37:26 [24023] IVM-1 pengine: info: short_print:
Stopped: [ IVM-0 ]
Feb 21 21:37:26 [24023] IVM-1 pengine: info: clone_print: Master/Slave
Set: alarmSnmpd_MS [alarmSnmpd]
Feb 21 21:37:26 [24023] IVM-1 pengine: info: short_print:
Slaves: [ IVM-1 ]
Feb 21 21:37:26 [24023] IVM-1 pengine: info: short_print:
Stopped: [ IVM-0 ]
Feb 21 21:37:26 [24023] IVM-1 pengine: info: native_print:
natomsip_for_snmpd (ocf::heartbeat:IPaddr2): Stopped
Feb 21 21:37:26 [24023] IVM-1 pengine: info: clone_print: Master/Slave
Set: haproxy_MS [haproxy]
Feb 21 21:37:26 [24023] IVM-1 pengine: info: short_print:
Slaves: [ IVM-1 ]
Feb 21 21:37:26 [24023] IVM-1 pengine: info: short_print:
Stopped: [ IVM-0 ]
Feb 21 21:37:26 [24023] IVM-1 pengine: info: native_print: nataaaip
(ocf::heartbeat:IPaddr2): Stopped
Feb 21 21:37:26 [24023] IVM-1 pengine: info: native_print:
nataccessip (ocf::heartbeat:IPaddr2): Stopped
Feb 21 21:37:26 [24023] IVM-1 pengine: info: clone_print: Master/Slave
Set: netactagent_MS [netactagent]
Feb 21 21:37:26 [24023] IVM-1 pengine: info: short_print:
Stopped: [ IVM-0 IVM-1 ]
Feb 21 21:37:26 [24023] IVM-1 pengine: info: native_print:
natomsip_for_netact (ocf::heartbeat:IPaddr2): Stopped
Feb 21 21:37:26 [24023] IVM-1 pengine: info: clone_print: Master/Slave
Set: RoutingManager_MS [RoutingManager]
Feb 21 21:37:26 [24023] IVM-1 pengine: info: short_print:
Slaves: [ IVM-1 ]
Feb 21 21:37:26 [24023] IVM-1 pengine: info: short_print:
Stopped: [ IVM-0 ]
Feb 21 21:37:26 [24023] IVM-1 pengine: info: group_print: Resource
Group: floatingips
Feb 21 21:37:26 [24023] IVM-1 pengine: info: native_print:
natomsgwip (ocf::heartbeat:IPaddr2): Stopped
Feb 21 21:37:26 [24023] IVM-1 pengine: info: native_print: omsip
(ocf::heartbeat:IPaddr2): Stopped
Feb 21 21:37:26 [24023] IVM-1 pengine: info: native_print:
nataaagwip (ocf::heartbeat:IPaddr2): Stopped
Feb 21 21:37:26 [24023] IVM-1 pengine: info: native_print: aaaip
(ocf::heartbeat:IPaddr2): Stopped
Feb 21 21:37:26 [24023] IVM-1 pengine: info: native_print:
nataccessgwip (ocf::heartbeat:IPaddr2): Stopped
Feb 21 21:37:26 [24023] IVM-1 pengine: info: native_print:
accessip (ocf::heartbeat:IPaddr2): Stopped
Feb 21 21:37:26 [24023] IVM-1 pengine: info: native_print:
natcpcoregwip (ocf::heartbeat:IPaddr2): Stopped
Feb 21 21:37:26 [24023] IVM-1 pengine: info: native_print:
cpcoreip (ocf::heartbeat:IPaddr2): Stopped
Feb 21 21:37:26 [24023] IVM-1 pengine: info: native_color: Resource
stateful_apache2:1 cannot run anywhere
Feb 21 21:37:26 [24023] IVM-1 pengine: info: master_color: Promoting
stateful_apache2:0 (Slave IVM-1)
Feb 21 21:37:26 [24023] IVM-1 pengine: info: master_color: apache2_MS:
Promoted 1 instances of a possible 1 to master
Feb 21 21:37:26 [24023] IVM-1 pengine: info: native_color: Resource
ClusterMon:1 cannot run anywhere
Feb 21 21:37:26 [24023] IVM-1 pengine: info: native_color: Resource
stateful_consul:1 cannot run anywhere
Feb 21 21:37:26 [24023] IVM-1 pengine: info: master_color: Promoting
stateful_consul:0 (Slave IVM-1)
Feb 21 21:37:26 [24023] IVM-1 pengine: info: master_color:
stateful_ms_set:
Promoted 1 instances of a possible 1 to master
Feb 21 21:37:26 [24023] IVM-1 pengine: info: native_color: Resource
stateful_wildfly:1 cannot run anywhere
Feb 21 21:37:26 [24023] IVM-1 pengine: info: master_color: Promoting
stateful_wildfly:0 (Slave IVM-1)
Feb 21 21:37:26 [24023] IVM-1 pengine: info: master_color: wildfly_MS:
Promoted 1 instances of a possible 1 to master
Feb 21 21:37:26 [24023] IVM-1 pengine: info: native_color: Resource
alarmSnmpd:1 cannot run anywhere
Feb 21 21:37:26 [24023] IVM-1 pengine: info: master_color: Promoting
alarmSnmpd:0 (Slave IVM-1)
Feb 21 21:37:26 [24023] IVM-1 pengine: info: master_color:
alarmSnmpd_MS:
Promoted 1 instances of a possible 1 to master
Feb 21 21:37:26 [24023] IVM-1 pengine: info: native_color: Resource
haproxy:1 cannot run anywhere
Feb 21 21:37:26 [24023] IVM-1 pengine: info: master_color: Promoting
haproxy:0 (Slave IVM-1)
Feb 21 21:37:26 [24023] IVM-1 pengine: info: master_color: haproxy_MS:
Promoted 1 instances of a possible 1 to master
Feb 21 21:37:26 [24023] IVM-1 pengine: info: rsc_merge_weights:
netactagent_MS:
Rolling back scores from natomsip_for_netact
Feb 21 21:37:26 [24023] IVM-1 pengine: info: native_color: Resource
netactagent:0 cannot run anywhere
Feb 21 21:37:26 [24023] IVM-1 pengine: info: native_color: Resource
netactagent:1 cannot run anywhere
Feb 21 21:37:26 [24023] IVM-1 pengine: info: rsc_merge_weights:
netactagent_MS:
Rolling back scores from natomsip_for_netact
Feb 21 21:37:26 [24023] IVM-1 pengine: info: master_color:
netactagent_MS:
Promoted 0 instances of a possible 1 to master
Feb 21 21:37:26 [24023] IVM-1 pengine: info: native_color: Resource
natomsip_for_netact cannot run anywhere
Feb 21 21:37:26 [24023] IVM-1 pengine: info: native_color: Resource
RoutingManager:1 cannot run anywhere
Feb 21 21:37:26 [24023] IVM-1 pengine: info: master_color: Promoting
RoutingManager:0 (Slave IVM-1)
Feb 21 21:37:26 [24023] IVM-1 pengine: info: master_color:
RoutingManager_MS:
Promoted 1 instances of a possible 1 to master
Feb 21 21:37:26 [24023] IVM-1 pengine: info: RecurringOp: Start
recurring monitor (30s) for stateful_apache2:0 on IVM-1
Feb 21 21:37:26 [24023] IVM-1 pengine: info: RecurringOp: Cancelling
action stateful_apache2:0_monitor_20000 (Slave vs. Master)
Feb 21 21:37:26 [24023] IVM-1 pengine: info: RecurringOp: Start
recurring monitor (30s) for stateful_apache2:0 on IVM-1
Feb 21 21:37:26 [24023] IVM-1 pengine: info: RecurringOp: Cancelling
action stateful_apache2:0_monitor_20000 (Slave vs. Master)
Feb 21 21:37:26 [24023] IVM-1 pengine: info: RecurringOp: Start
recurring monitor (10s) for natomsip on IVM-1
Feb 21 21:37:26 [24023] IVM-1 pengine: info: RecurringOp: Start
recurring monitor (10s) for natcpcoreip on IVM-1
Feb 21 21:37:26 [24023] IVM-1 pengine: info: RecurringOp: Start
recurring monitor (5s) for stateful_consul:0 on IVM-1
Feb 21 21:37:26 [24023] IVM-1 pengine: info: RecurringOp: Cancelling
action stateful_consul:0_monitor_7000 (Slave vs. Master)
Feb 21 21:37:26 [24023] IVM-1 pengine: info: RecurringOp: Start
recurring monitor (5s) for stateful_consul:0 on IVM-1
Feb 21 21:37:26 [24023] IVM-1 pengine: info: RecurringOp: Cancelling
action stateful_consul:0_monitor_7000 (Slave vs. Master)
Feb 21 21:37:26 [24023] IVM-1 pengine: info: RecurringOp: Start
recurring monitor (90s) for stateful_wildfly:0 on IVM-1
Feb 21 21:37:26 [24023] IVM-1 pengine: info: RecurringOp: Cancelling
action stateful_wildfly:0_monitor_80000 (Slave vs. Master)
Feb 21 21:37:26 [24023] IVM-1 pengine: info: RecurringOp: Start
recurring monitor (90s) for stateful_wildfly:0 on IVM-1
Feb 21 21:37:26 [24023] IVM-1 pengine: info: RecurringOp: Cancelling
action stateful_wildfly:0_monitor_80000 (Slave vs. Master)
Feb 21 21:37:26 [24023] IVM-1 pengine: info: RecurringOp: Start
recurring monitor (20s) for alarmSnmpd:0 on IVM-1
Feb 21 21:37:26 [24023] IVM-1 pengine: info: RecurringOp: Cancelling
action alarmSnmpd:0_monitor_30000 (Slave vs. Master)
Feb 21 21:37:26 [24023] IVM-1 pengine: info: RecurringOp: Start
recurring monitor (20s) for alarmSnmpd:0 on IVM-1
Feb 21 21:37:26 [24023] IVM-1 pengine: info: RecurringOp: Cancelling
action alarmSnmpd:0_monitor_30000 (Slave vs. Master)
Feb 21 21:37:26 [24023] IVM-1 pengine: info: RecurringOp: Start
recurring monitor (10s) for natomsip_for_snmpd on IVM-1
Feb 21 21:37:26 [24023] IVM-1 pengine: info: RecurringOp: Start
recurring monitor (20s) for haproxy:0 on IVM-1
Feb 21 21:37:26 [24023] IVM-1 pengine: info: RecurringOp: Start
recurring monitor (20s) for haproxy:0 on IVM-1
Feb 21 21:37:26 [24023] IVM-1 pengine: info: RecurringOp: Start
recurring monitor (10s) for nataaaip on IVM-1
Feb 21 21:37:26 [24023] IVM-1 pengine: info: RecurringOp: Start
recurring monitor (10s) for nataccessip on IVM-1
Feb 21 21:37:26 [24023] IVM-1 pengine: info: RecurringOp: Start
recurring monitor (20s) for RoutingManager:0 on IVM-1
Feb 21 21:37:26 [24023] IVM-1 pengine: info: RecurringOp: Cancelling
action RoutingManager:0_monitor_30000 (Slave vs. Master)
Feb 21 21:37:26 [24023] IVM-1 pengine: info: RecurringOp: Start
recurring monitor (20s) for RoutingManager:0 on IVM-1
Feb 21 21:37:26 [24023] IVM-1 pengine: info: RecurringOp: Cancelling
action RoutingManager:0_monitor_30000 (Slave vs. Master)
Feb 21 21:37:26 [24023] IVM-1 pengine: info: RecurringOp: Start
recurring monitor (10s) for natomsgwip on IVM-1
Feb 21 21:37:26 [24023] IVM-1 pengine: info: RecurringOp: Start
recurring monitor (10s) for omsip on IVM-1
Feb 21 21:37:26 [24023] IVM-1 pengine: info: RecurringOp: Start
recurring monitor (10s) for nataaagwip on IVM-1
Feb 21 21:37:26 [24023] IVM-1 pengine: info: RecurringOp: Start
recurring monitor (10s) for aaaip on IVM-1
Feb 21 21:37:26 [24023] IVM-1 pengine: info: RecurringOp: Start
recurring monitor (10s) for nataccessgwip on IVM-1
Feb 21 21:37:26 [24023] IVM-1 pengine: info: RecurringOp: Start
recurring monitor (10s) for accessip on IVM-1
Feb 21 21:37:26 [24023] IVM-1 pengine: info: RecurringOp: Start
recurring monitor (10s) for natcpcoregwip on IVM-1
Feb 21 21:37:26 [24023] IVM-1 pengine: info: RecurringOp: Start
recurring monitor (10s) for cpcoreip on IVM-1
Feb 21 21:37:26 [24023] IVM-1 pengine: notice: LogActions: Promote
stateful_apache2:0 (Slave -> Master IVM-1)
Feb 21 21:37:26 [24023] IVM-1 pengine: info: LogActions: Leave
stateful_apache2:1 (Stopped)
Feb 21 21:37:26 [24023] IVM-1 pengine: notice: LogActions: Start
natomsip (IVM-1)
Feb 21 21:37:26 [24023] IVM-1 pengine: notice: LogActions: Start
natcpcoreip (IVM-1)
Feb 21 21:37:26 [24023] IVM-1 pengine: info: LogActions: Leave
ClusterMon:0 (Started IVM-1)
Feb 21 21:37:26 [24023] IVM-1 pengine: info: LogActions: Leave
ClusterMon:1 (Stopped)
Feb 21 21:37:26 [24023] IVM-1 pengine: notice: LogActions: Promote
stateful_consul:0 (Slave -> Master IVM-1)
Feb 21 21:37:26 [24023] IVM-1 pengine: info: LogActions: Leave
stateful_consul:1 (Stopped)
Feb 21 21:37:26 [24023] IVM-1 pengine: notice: LogActions: Promote
stateful_wildfly:0 (Slave -> Master IVM-1)
Feb 21 21:37:26 [24023] IVM-1 pengine: info: LogActions: Leave
stateful_wildfly:1 (Stopped)
Feb 21 21:37:26 [24023] IVM-1 pengine: notice: LogActions: Promote
alarmSnmpd:0 (Slave -> Master IVM-1)
Feb 21 21:37:26 [24023] IVM-1 pengine: info: LogActions: Leave
alarmSnmpd:1 (Stopped)
Feb 21 21:37:26 [24023] IVM-1 pengine: notice: LogActions: Start
natomsip_for_snmpd (IVM-1)
Feb 21 21:37:26 [24023] IVM-1 pengine: notice: LogActions: Promote
haproxy:0 (Slave -> Master IVM-1)
Feb 21 21:37:26 [24023] IVM-1 pengine: info: LogActions: Leave
haproxy:1 (Stopped)
Feb 21 21:37:26 [24023] IVM-1 pengine: notice: LogActions: Start
nataaaip (IVM-1)
Feb 21 21:37:26 [24023] IVM-1 pengine: notice: LogActions: Start
nataccessip (IVM-1)
Feb 21 21:37:26 [24023] IVM-1 pengine: info: LogActions: Leave
netactagent:0 (Stopped)
Feb 21 21:37:26 [24023] IVM-1 pengine: info: LogActions: Leave
netactagent:1 (Stopped)
Feb 21 21:37:26 [24023] IVM-1 pengine: info: LogActions: Leave
natomsip_for_netact (Stopped)
Feb 21 21:37:26 [24023] IVM-1 pengine: notice: LogActions: Promote
RoutingManager:0 (Slave -> Master IVM-1)
Feb 21 21:37:26 [24023] IVM-1 pengine: info: LogActions: Leave
RoutingManager:1 (Stopped)
Feb 21 21:37:26 [24023] IVM-1 pengine: notice: LogActions: Start
natomsgwip (IVM-1)
Feb 21 21:37:26 [24023] IVM-1 pengine: notice: LogActions: Start
omsip (IVM-1)
Feb 21 21:37:26 [24023] IVM-1 pengine: notice: LogActions: Start
nataaagwip (IVM-1)
Feb 21 21:37:26 [24023] IVM-1 pengine: notice: LogActions: Start
aaaip (IVM-1)
Feb 21 21:37:26 [24023] IVM-1 pengine: notice: LogActions: Start
nataccessgwip (IVM-1)
Feb 21 21:37:26 [24023] IVM-1 pengine: notice: LogActions: Start
accessip (IVM-1)
Feb 21 21:37:26 [24023] IVM-1 pengine: notice: LogActions: Start
natcpcoregwip (IVM-1)
Feb 21 21:37:26 [24023] IVM-1 pengine: notice: LogActions: Start
cpcoreip (IVM-1)
Feb 21 21:37:26 [24023] IVM-1 pengine: notice:
process_pe_message: Calculated
transition 0, saving inputs in /usr/var/lib/pacemaker/
pengine/pe-input-106.bz2
Feb 21 21:37:26 [24024] IVM-1 crmd: info: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response
Feb 21 21:37:26 [24024] IVM-1 crmd: info: do_te_invoke: Processing
graph 0 (ref=pe_calc-dc-1519229246-22) derived from /usr/var/lib/pacemaker/
pengine/pe-input-106.bz2
Feb 21 21:37:26 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
cancel operation stateful_apache2_monitor_20000 locally on IVM-1 | action 5
Feb 21 21:37:26 [24021] IVM-1 lrmd: info:
cancel_recurring_action: Cancelling
ocf operation stateful_apache2_monitor_20000
Feb 21 21:37:26 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
start operation natomsip_start_0 locally on IVM-1 | action 36
Feb 21 21:37:26 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=36:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079 op=natomsip_start_0
Feb 21 21:37:26 [24021] IVM-1 lrmd: info: log_execute: executing
- rsc:natomsip action:start call_id:122
Feb 21 21:37:26 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
cancel operation stateful_consul_monitor_7000 locally on IVM-1 | action 4
Feb 21 21:37:26 [24021] IVM-1 lrmd: info:
cancel_recurring_action: Cancelling
ocf operation stateful_consul_monitor_7000
Feb 21 21:37:26 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
cancel operation stateful_wildfly_monitor_80000 locally on IVM-1 | action 2
Feb 21 21:37:26 [24021] IVM-1 lrmd: info:
cancel_recurring_action: Cancelling
ocf operation stateful_wildfly_monitor_80000
Feb 21 21:37:26 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
cancel operation alarmSnmpd_monitor_30000 locally on IVM-1 | action 1
Feb 21 21:37:26 [24021] IVM-1 lrmd: info:
cancel_recurring_action: Cancelling
ocf operation alarmSnmpd_monitor_30000
Feb 21 21:37:26 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
start operation natomsip_for_snmpd_start_0 locally on IVM-1 | action 130
Feb 21 21:37:26 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=130:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079
op=natomsip_for_snmpd_start_0
Feb 21 21:37:26 [24021] IVM-1 lrmd: info: log_execute: executing
- rsc:natomsip_for_snmpd action:start call_id:126
Feb 21 21:37:26 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
start operation nataaaip_start_0 locally on IVM-1 | action 160
Feb 21 21:37:26 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=160:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079 op=nataaaip_start_0
Feb 21 21:37:26 [24021] IVM-1 lrmd: info: log_execute: executing
- rsc:nataaaip action:start call_id:127
Feb 21 21:37:26 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
cancel operation RoutingManager_monitor_30000 locally on IVM-1 | action 6
Feb 21 21:37:26 [24021] IVM-1 lrmd: info:
cancel_recurring_action: Cancelling
ocf operation RoutingManager_monitor_30000
Feb 21 21:37:26 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
start operation natomsgwip_start_0 locally on IVM-1 | action 216
Feb 21 21:37:26 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=216:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079 op=natomsgwip_start_0
Feb 21 21:37:26 [24021] IVM-1 lrmd: info: log_execute: executing
- rsc:natomsgwip action:start call_id:129
Feb 21 21:37:26 [24024] IVM-1 crmd: info: process_lrm_event: Result
of monitor operation for stateful_apache2 on IVM-1: Cancelled | call=110
key=stateful_apache2_monitor_20000 confirmed=true
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_delete operation for section status to all (origin=local/crmd/71)
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.8 2
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.9 (null)
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: --
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/
lrm_resource[@id='stateful_apache2']/lrm_rsc_op[@id='
stateful_apache2_monitor_20000']
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=9
Feb 21 21:37:26 [24024] IVM-1 crmd: info: process_lrm_event: Result
of monitor operation for stateful_consul on IVM-1: Cancelled | call=111
key=stateful_consul_monitor_7000 confirmed=true
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_delete operation for section status: OK (rc=0, origin=IVM-1/crmd/71,
version=0.108.9)
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_delete operation for section status to all (origin=local/crmd/72)
Feb 21 21:37:26 [24024] IVM-1 crmd: info: process_lrm_event: Result
of monitor operation for stateful_wildfly on IVM-1: Cancelled | call=112
key=stateful_wildfly_monitor_80000 confirmed=true
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.9 2
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.10 (null)
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: --
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/
lrm_resource[@id='stateful_consul']/lrm_rsc_op[@id='
stateful_consul_monitor_7000']
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=10
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_delete operation for section status: OK (rc=0, origin=IVM-1/crmd/72,
version=0.108.10)
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_delete operation for section status to all (origin=local/crmd/73)
Feb 21 21:37:26 [24024] IVM-1 crmd: info: process_lrm_event: Result
of monitor operation for alarmSnmpd on IVM-1: Cancelled | call=113
key=alarmSnmpd_monitor_30000 confirmed=true
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_delete operation for section status to all (origin=local/crmd/74)
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.10 2
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.11 (null)
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: --
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/
lrm_resource[@id='stateful_wildfly']/lrm_rsc_op[@id='
stateful_wildfly_monitor_80000']
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=11
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_delete operation for section status: OK (rc=0, origin=IVM-1/crmd/73,
version=0.108.11)
Feb 21 21:37:26 [24024] IVM-1 crmd: info: process_lrm_event: Result
of monitor operation for RoutingManager on IVM-1: Cancelled | call=115
key=RoutingManager_monitor_30000 confirmed=true
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.11 2
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.12 (null)
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: --
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/
lrm_resource[@id='alarmSnmpd']/lrm_rsc_op[@id='alarmSnmpd_monitor_30000']
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=12
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_delete operation for section status: OK (rc=0, origin=IVM-1/crmd/74,
version=0.108.12)
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_delete operation for section status to all (origin=local/crmd/75)
Feb 21 21:37:26 [24024] IVM-1 crmd: info: te_update_diff:
Cancellation
of stateful_apache2_monitor_20000 on 2 confirmed (5)
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.12 2
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.13 (null)
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: --
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/
lrm_resource[@id='RoutingManager']/lrm_rsc_op[@id='RoutingManager_monitor_
30000']
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=13
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_delete operation for section status: OK (rc=0, origin=IVM-1/crmd/75,
version=0.108.13)
Feb 21 21:37:26 [24024] IVM-1 crmd: info: te_update_diff:
Cancellation
of stateful_consul_monitor_7000 on 2 confirmed (4)
Feb 21 21:37:26 [24024] IVM-1 crmd: info: te_update_diff:
Cancellation
of stateful_wildfly_monitor_80000 on 2 confirmed (2)
Feb 21 21:37:26 [24024] IVM-1 crmd: info: te_update_diff:
Cancellation
of alarmSnmpd_monitor_30000 on 2 confirmed (1)
Feb 21 21:37:26 [24024] IVM-1 crmd: info: te_update_diff:
Cancellation
of RoutingManager_monitor_30000 on 2 confirmed (6)
Feb 21 21:37:26 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
notify operation stateful_apache2_pre_notify_promote_0 locally on IVM-1 |
action 267
Feb 21 21:37:26 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=267:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079
op=stateful_apache2_notify_0
Feb 21 21:37:26 [24021] IVM-1 lrmd: info: log_execute: executing
- rsc:stateful_apache2 action:notify call_id:130
Feb 21 21:37:26 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
notify operation stateful_consul_pre_notify_promote_0 locally on IVM-1 |
action 273
Feb 21 21:37:26 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=273:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079 op=stateful_consul_notify_0
Feb 21 21:37:26 [24021] IVM-1 lrmd: info: log_execute: executing
- rsc:stateful_consul action:notify call_id:131
Feb 21 21:37:26 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
notify operation stateful_wildfly_pre_notify_promote_0 locally on IVM-1 |
action 279
Feb 21 21:37:26 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=279:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079
op=stateful_wildfly_notify_0
Feb 21 21:37:26 [24021] IVM-1 lrmd: info: log_execute: executing
- rsc:stateful_wildfly action:notify call_id:132
Feb 21 21:37:26 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
notify operation alarmSnmpd_pre_notify_promote_0 locally on IVM-1 | action
285
Feb 21 21:37:26 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=285:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079 op=alarmSnmpd_notify_0
Feb 21 21:37:26 [24021] IVM-1 lrmd: info: log_execute: executing
- rsc:alarmSnmpd action:notify call_id:133
Feb 21 21:37:26 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
notify operation haproxy_pre_notify_promote_0 locally on IVM-1 | action 291
Feb 21 21:37:26 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=291:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079 op=haproxy_notify_0
Feb 21 21:37:26 [24021] IVM-1 lrmd: info: log_execute: executing
- rsc:haproxy action:notify call_id:134
Feb 21 21:37:26 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
notify operation RoutingManager_pre_notify_promote_0 locally on IVM-1 |
action 297
Feb 21 21:37:26 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=297:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079 op=RoutingManager_notify_0
Feb 21 21:37:26 [24021] IVM-1 lrmd: info: log_execute: executing
- rsc:RoutingManager action:notify call_id:135
Feb 21 21:37:26 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
24 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:37:26 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
26 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:37:26 [24021] IVM-1 lrmd: info: log_finished: finished
- rsc:RoutingManager action:notify call_id:135 pid:510 exit-code:0
exec-time:47ms queue-time:0ms
Feb 21 21:37:26 [24024] IVM-1 crmd: info: match_graph_event: Action
RoutingManager_notify_0 (297) confirmed on IVM-1 (rc=0)
Feb 21 21:37:26 [24024] IVM-1 crmd: notice: process_lrm_event: Result
of notify operation for RoutingManager on IVM-1: 0 (ok) | call=135
key=RoutingManager_notify_0 confirmed=true cib-update=0
Feb 21 21:37:26 [24021] IVM-1 lrmd: info: log_finished: finished
- rsc:alarmSnmpd action:notify call_id:133 pid:495 exit-code:3
exec-time:65ms queue-time:0ms
Feb 21 21:37:26 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
28 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:37:26 [24024] IVM-1 crmd: info: match_graph_event: Action
alarmSnmpd_notify_0 (285) confirmed on IVM-1 (rc=0)
Feb 21 21:37:26 [24024] IVM-1 crmd: notice: process_lrm_event: Result
of notify operation for alarmSnmpd on IVM-1: 0 (ok) | call=133
key=alarmSnmpd_notify_0 confirmed=true cib-update=0
Feb 21 21:37:26 [24021] IVM-1 lrmd: info: log_finished: finished
- rsc:haproxy action:notify call_id:134 pid:507 exit-code:0 exec-time:74ms
queue-time:0ms
Feb 21 21:37:26 [24024] IVM-1 crmd: info: match_graph_event: Action
haproxy_notify_0 (291) confirmed on IVM-1 (rc=0)
Feb 21 21:37:26 [24024] IVM-1 crmd: notice: process_lrm_event: Result
of notify operation for haproxy on IVM-1: 0 (ok) | call=134
key=haproxy_notify_0 confirmed=true cib-update=0
Feb 21 21:37:26 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
25 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:37:26 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
27 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:37:26 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
29 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:37:26 [24021] IVM-1 lrmd: info: log_finished: finished
- rsc:stateful_consul action:notify call_id:131 pid:477 exit-code:0
exec-time:132ms queue-time:0ms
Feb 21 21:37:26 [24021] IVM-1 lrmd: info: log_finished: finished
- rsc:stateful_wildfly action:notify call_id:132 pid:492 exit-code:0
exec-time:116ms queue-time:0ms
Feb 21 21:37:26 [24024] IVM-1 crmd: info: match_graph_event: Action
stateful_consul_notify_0 (273) confirmed on IVM-1 (rc=0)
Feb 21 21:37:26 [24024] IVM-1 crmd: notice: process_lrm_event: Result
of notify operation for stateful_consul on IVM-1: 0 (ok) | call=131
key=stateful_consul_notify_0 confirmed=true cib-update=0
Feb 21 21:37:26 [24024] IVM-1 crmd: info: match_graph_event: Action
stateful_wildfly_notify_0 (279) confirmed on IVM-1 (rc=0)
Feb 21 21:37:26 [24024] IVM-1 crmd: notice: process_lrm_event: Result
of notify operation for stateful_wildfly on IVM-1: 0 (ok) | call=132
key=stateful_wildfly_notify_0 confirmed=true cib-update=0
Feb 21 21:37:26 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
promote operation stateful_consul_promote_0 locally on IVM-1 | action 48
Feb 21 21:37:26 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=48:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079 op=stateful_consul_promote_0
Feb 21 21:37:26 [24021] IVM-1 lrmd: info: log_execute: executing
- rsc:stateful_consul action:promote call_id:136
Feb 21 21:37:26 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
promote operation stateful_wildfly_promote_0 locally on IVM-1 | action 76
Feb 21 21:37:26 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=76:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079
op=stateful_wildfly_promote_0
Feb 21 21:37:26 [24021] IVM-1 lrmd: info: log_execute: executing
- rsc:stateful_wildfly action:promote call_id:137
Feb 21 21:37:26 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
30 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:37:26 [24021] IVM-1 lrmd: info: log_finished: finished
- rsc:stateful_apache2 action:notify call_id:130 pid:473 exit-code:0
exec-time:142ms queue-time:0ms
Feb 21 21:37:26 [24024] IVM-1 crmd: info: match_graph_event: Action
stateful_apache2_notify_0 (267) confirmed on IVM-1 (rc=0)
Feb 21 21:37:26 [24024] IVM-1 crmd: notice: process_lrm_event: Result
of notify operation for stateful_apache2 on IVM-1: 0 (ok) | call=130
key=stateful_apache2_notify_0 confirmed=true cib-update=0
IPaddr2(natomsgwip)[449]: 2018/02/21_21:37:26 INFO: Adding inet address
192.168.1.1/24 with broadcast address + to device natoms
IPaddr2(natomsip)[436]: 2018/02/21_21:37:26 INFO: Adding inet address
192.168.1.2/24 with broadcast address + to device natoms
IPaddr2(natomsip_for_snmpd)[444]: 2018/02/21_21:37:26 INFO: Adding inet
address 192.168.1.3/24 with broadcast address + to device natoms
IPaddr2(nataaaip)[446]: 2018/02/21_21:37:26 INFO: Adding inet address
192.168.2.2/24 with broadcast address + to device nataaa
Feb 21 21:37:26 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
32 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:37:26 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
31 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
IPaddr2(natomsip_for_snmpd)[444]: 2018/02/21_21:37:26 INFO: Bringing device
natoms up
IPaddr2(natomsgwip)[449]: 2018/02/21_21:37:26 INFO: Bringing device natoms
up
IPaddr2(natomsip)[436]: 2018/02/21_21:37:26 INFO: Bringing device natoms up
IPaddr2(nataaaip)[446]: 2018/02/21_21:37:26 INFO: Bringing device nataaa up
IPaddr2(natomsip_for_snmpd)[444]: 2018/02/21_21:37:26 INFO:
/usr/libexec/heartbeat/send_arp -i 200 -r 5 -p
/usr/var/run/resource-agents/send_arp-192.168.1.3
natoms 192.168.1.3 auto not_used not_used
IPaddr2(natomsip)[436]: 2018/02/21_21:37:26 INFO:
/usr/libexec/heartbeat/send_arp -i 200 -r 5 -p
/usr/var/run/resource-agents/send_arp-192.168.1.2
natoms 192.168.1.2 auto not_used not_used
IPaddr2(nataaaip)[446]: 2018/02/21_21:37:26 INFO:
/usr/libexec/heartbeat/send_arp -i 200 -r 5 -p
/usr/var/run/resource-agents/send_arp-192.168.2.2
nataaa 192.168.2.2 auto not_used not_used
Feb 21 21:37:26 [24021] IVM-1 lrmd: info: log_finished: finished
- rsc:natomsip_for_snmpd action:start call_id:126 pid:444 exit-code:0
exec-time:200ms queue-time:0ms
Feb 21 21:37:26 [24021] IVM-1 lrmd: info: log_finished: finished
- rsc:natomsip action:start call_id:122 pid:436 exit-code:0 exec-time:204ms
queue-time:0ms
Feb 21 21:37:26 [24021] IVM-1 lrmd: info: log_finished: finished
- rsc:nataaaip action:start call_id:127 pid:446 exit-code:0 exec-time:200ms
queue-time:0ms
IPaddr2(natomsgwip)[449]: 2018/02/21_21:37:26 INFO:
/usr/libexec/heartbeat/send_arp -i 200 -r 5 -p
/usr/var/run/resource-agents/send_arp-192.168.1.1
natoms 192.168.1.1 auto not_used not_used
Feb 21 21:37:26 [24021] IVM-1 lrmd: info: log_finished: finished
- rsc:natomsgwip action:start call_id:129 pid:449 exit-code:0
exec-time:210ms queue-time:0ms
Feb 21 21:37:26 [24024] IVM-1 crmd: info: action_synced_wait: Managed
IPaddr2_meta-data_0 process 1077 exited with rc=0
Feb 21 21:37:26 [24024] IVM-1 crmd: notice: process_lrm_event: Result
of start operation for natomsip_for_snmpd on IVM-1: 0 (ok) | call=126
key=natomsip_for_snmpd_start_0 confirmed=true cib-update=76
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/crmd/76)
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.13 2
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.14 (null)
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=14
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib/status/node_state[@id='2']: @crm-debug-origin=do_update_resource
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_
resources/lrm_resource[@id='natomsip_for_snmpd']/lrm_rsc_
op[@id='natomsip_for_snmpd_last_0']:
@operation_key=natomsip_for_snmpd_start_0,
@operation=start, @crm-debug-origin=do_update_resource,
@transition-key=130:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079,
@transition-magic=0:0;130:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079,
@call-id=126, @rc-code=0, @last-run=1519229246, @last-rc-change=151
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/crmd/76,
version=0.108.14)
Feb 21 21:37:26 [24024] IVM-1 crmd: info: action_synced_wait: Managed
IPaddr2_meta-data_0 process 1135 exited with rc=0
Feb 21 21:37:26 [24024] IVM-1 crmd: notice: process_lrm_event: Result
of start operation for natomsip on IVM-1: 0 (ok) | call=122
key=natomsip_start_0 confirmed=true cib-update=77
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/crmd/77)
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.14 2
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.15 (null)
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=15
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_
resources/lrm_resource[@id='natomsip']/lrm_rsc_op[@id='natomsip_last_0']:
@operation_key=natomsip_start_0, @operation=start,
@crm-debug-origin=do_update_resource, @transition-key=36:0:0:
e27e11f8-fc4e-493e-a4f2-80a02cb1f079, @transition-magic=0:0;36:0:0:
e27e11f8-fc4e-493e-a4f2-80a02cb1f079, @call-id=122, @rc-code=0,
@last-run=1519229246, @last-rc-change=1519229246, @exec-time=204
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/crmd/77,
version=0.108.15)
Feb 21 21:37:26 [24024] IVM-1 crmd: info: action_synced_wait: Managed
IPaddr2_meta-data_0 process 1189 exited with rc=0
Feb 21 21:37:26 [24024] IVM-1 crmd: notice: process_lrm_event: Result
of start operation for nataaaip on IVM-1: 0 (ok) | call=127
key=nataaaip_start_0 confirmed=true cib-update=78
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/crmd/78)
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.15 2
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.16 (null)
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=16
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_
resources/lrm_resource[@id='nataaaip']/lrm_rsc_op[@id='nataaaip_last_0']:
@operation_key=nataaaip_start_0, @operation=start,
@crm-debug-origin=do_update_resource, @transition-key=160:0:0:
e27e11f8-fc4e-493e-a4f2-80a02cb1f079, @transition-magic=0:0;160:0:0:
e27e11f8-fc4e-493e-a4f2-80a02cb1f079, @call-id=127, @rc-code=0,
@last-run=1519229246, @last-rc-change=1519229246, @exec-time=200
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/crmd/78,
version=0.108.16)
Feb 21 21:37:26 [24024] IVM-1 crmd: info: action_synced_wait: Managed
IPaddr2_meta-data_0 process 1285 exited with rc=0
Feb 21 21:37:26 [24024] IVM-1 crmd: notice: process_lrm_event: Result
of start operation for natomsgwip on IVM-1: 0 (ok) | call=129
key=natomsgwip_start_0 confirmed=true cib-update=79
Feb 21 21:37:26 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
33 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:37:26 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
34 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:37:26 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
35 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:37:26 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
36 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:37:26 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
37 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:37:26 [24024] IVM-1 crmd: info: match_graph_event: Action
natomsip_for_snmpd_start_0 (130) confirmed on IVM-1 (rc=0)
Feb 21 21:37:26 [24024] IVM-1 crmd: info: match_graph_event: Action
natomsip_start_0 (36) confirmed on IVM-1 (rc=0)
Feb 21 21:37:26 [24024] IVM-1 crmd: info: match_graph_event: Action
nataaaip_start_0 (160) confirmed on IVM-1 (rc=0)
Feb 21 21:37:26 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
monitor operation natomsip_monitor_10000 locally on IVM-1 | action 37
Feb 21 21:37:26 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=37:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079 op=natomsip_monitor_10000
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/crmd/79)
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.16 2
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.17 (null)
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=17
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_
resources/lrm_resource[@id='natomsgwip']/lrm_rsc_op[@id='natomsgwip_last_0']:
@operation_key=natomsgwip_start_0, @operation=start,
@crm-debug-origin=do_update_resource, @transition-key=216:0:0:
e27e11f8-fc4e-493e-a4f2-80a02cb1f079, @transition-magic=0:0;216:0:0:
e27e11f8-fc4e-493e-a4f2-80a02cb1f079, @call-id=129, @rc-code=0,
@last-run=1519229246, @last-rc-change=1519229246, @exec-time=210
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/crmd/79,
version=0.108.17)
Feb 21 21:37:26 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
start operation natcpcoreip_start_0 locally on IVM-1 | action 38
Feb 21 21:37:26 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=38:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079 op=natcpcoreip_start_0
Feb 21 21:37:26 [24021] IVM-1 lrmd: info: log_execute: executing
- rsc:natcpcoreip action:start call_id:139
Feb 21 21:37:26 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
monitor operation natomsip_for_snmpd_monitor_10000 locally on IVM-1 |
action 131
Feb 21 21:37:26 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=131:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079
op=natomsip_for_snmpd_monitor_10000
Feb 21 21:37:26 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
monitor operation nataaaip_monitor_10000 locally on IVM-1 | action 161
Feb 21 21:37:26 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=161:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079 op=nataaaip_monitor_10000
Feb 21 21:37:26 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
start operation nataccessip_start_0 locally on IVM-1 | action 162
Feb 21 21:37:26 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=162:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079 op=nataccessip_start_0
Feb 21 21:37:26 [24021] IVM-1 lrmd: info: log_execute: executing
- rsc:nataccessip action:start call_id:142
Feb 21 21:37:26 [24024] IVM-1 crmd: info: match_graph_event: Action
natomsgwip_start_0 (216) confirmed on IVM-1 (rc=0)
Feb 21 21:37:26 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
promote operation alarmSnmpd_promote_0 locally on IVM-1 | action 104
Feb 21 21:37:26 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=104:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079 op=alarmSnmpd_promote_0
Feb 21 21:37:26 [24021] IVM-1 lrmd: info: log_execute: executing
- rsc:alarmSnmpd action:promote call_id:143
Feb 21 21:37:26 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
monitor operation natomsgwip_monitor_10000 locally on IVM-1 | action 217
Feb 21 21:37:26 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=217:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079 op=natomsgwip_monitor_10000
Feb 21 21:37:26 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
start operation omsip_start_0 locally on IVM-1 | action 218
Feb 21 21:37:26 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=218:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079 op=omsip_start_0
Feb 21 21:37:26 [24021] IVM-1 lrmd: info: log_execute: executing
- rsc:omsip action:start call_id:145
Feb 21 21:37:26 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
38 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:37:26 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
39 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:37:26 [24024] IVM-1 crmd: info: process_lrm_event: Result
of monitor operation for nataaaip on IVM-1: 0 (ok) | call=141
key=nataaaip_monitor_10000 confirmed=false cib-update=80
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/crmd/80)
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.17 2
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.18 (null)
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=18
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='nataaaip']:
<lrm_rsc_op id="nataaaip_monitor_10000" operation_key="nataaaip_monitor_10000"
operation="monitor" crm-debug-origin="do_update_resource"
crm_feature_set="3.0.11"
transition-key="161:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079"
transition-magic="0:0;161:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079"
on_node="IVM-1" call-id="141" rc-code="0" op-status="0" in
Feb 21 21:37:26 [24024] IVM-1 crmd: info: match_graph_event: Action
nataaaip_monitor_10000 (161) confirmed on IVM-1 (rc=0)
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/crmd/80,
version=0.108.18)
Feb 21 21:37:26 [24024] IVM-1 crmd: info: process_lrm_event: Result
of monitor operation for natomsip_for_snmpd on IVM-1: 0 (ok) | call=140
key=natomsip_for_snmpd_monitor_10000 confirmed=false cib-update=81
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/crmd/81)
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.18 2
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.19 (null)
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=19
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/
lrm_resource[@id='natomsip_for_snmpd']: <lrm_rsc_op
id="natomsip_for_snmpd_monitor_10000"
operation_key="natomsip_for_snmpd_monitor_10000" operation="monitor"
crm-debug-origin="do_update_resource" crm_feature_set="3.0.11"
transition-key="131:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079"
transition-magic="0:0;131:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079"
on_node="IVM-1" call-id="140
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/crmd/81,
version=0.108.19)
Feb 21 21:37:26 [24024] IVM-1 crmd: info: match_graph_event: Action
natomsip_for_snmpd_monitor_10000 (131) confirmed on IVM-1 (rc=0)
Feb 21 21:37:26 [24024] IVM-1 crmd: info: process_lrm_event: Result
of monitor operation for natomsgwip on IVM-1: 0 (ok) | call=144
key=natomsgwip_monitor_10000 confirmed=false cib-update=82
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/crmd/82)
IPaddr2(natcpcoreip)[1331]: 2018/02/21_21:37:26 INFO: Adding inet address
192.168.4.2/24 with broadcast address + to device natcpcore
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.19 2
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.20 (null)
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=20
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/
lrm_resource[@id='natomsgwip']: <lrm_rsc_op id="natomsgwip_monitor_10000"
operation_key="natomsgwip_monitor_10000" operation="monitor"
crm-debug-origin="do_update_resource" crm_feature_set="3.0.11"
transition-key="217:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079"
transition-magic="0:0;217:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079"
on_node="IVM-1" call-id="144" rc-code="0" op-status=
Feb 21 21:37:26 [24024] IVM-1 crmd: info: match_graph_event: Action
natomsgwip_monitor_10000 (217) confirmed on IVM-1 (rc=0)
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/crmd/82,
version=0.108.20)
Feb 21 21:37:26 [24024] IVM-1 crmd: info: process_lrm_event: Result
of monitor operation for natomsip on IVM-1: 0 (ok) | call=138
key=natomsip_monitor_10000 confirmed=false cib-update=83
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/crmd/83)
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.20 2
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.21 (null)
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=21
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='natomsip']:
<lrm_rsc_op id="natomsip_monitor_10000" operation_key="natomsip_monitor_10000"
operation="monitor" crm-debug-origin="do_update_resource"
crm_feature_set="3.0.11"
transition-key="37:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079"
transition-magic="0:0;37:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079"
on_node="IVM-1" call-id="138" rc-code="0" op-status="0" inte
Feb 21 21:37:26 [24024] IVM-1 crmd: info: match_graph_event: Action
natomsip_monitor_10000 (37) confirmed on IVM-1 (rc=0)
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/crmd/83,
version=0.108.21)
IPaddr2(natcpcoreip)[1331]: 2018/02/21_21:37:26 INFO: Bringing device
natcpcore up
Feb 21 21:37:26 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
40 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
IPaddr2(natcpcoreip)[1331]: 2018/02/21_21:37:26 INFO:
/usr/libexec/heartbeat/send_arp -i 200 -r 5 -p
/usr/var/run/resource-agents/send_arp-192.168.4.2
natcpcore 192.168.4.2 auto not_used not_used
Feb 21 21:37:26 [24021] IVM-1 lrmd: info: log_finished: finished
- rsc:natcpcoreip action:start call_id:139 pid:1331 exit-code:0
exec-time:163ms queue-time:0ms
Feb 21 21:37:26 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
41 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
IPaddr2(omsip)[1355]: 2018/02/21_21:37:26 INFO: Adding inet address
10.53.201.209/27 with broadcast address + to device oms
IPaddr2(nataccessip)[1340]: 2018/02/21_21:37:26 INFO: Adding inet address
192.168.3.2/24 with broadcast address + to device nataccess
Feb 21 21:37:26 [24024] IVM-1 crmd: info: action_synced_wait: Managed
IPaddr2_meta-data_0 process 1952 exited with rc=0
Feb 21 21:37:26 [24024] IVM-1 crmd: notice: process_lrm_event: Result
of start operation for natcpcoreip on IVM-1: 0 (ok) | call=139
key=natcpcoreip_start_0 confirmed=true cib-update=84
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/crmd/84)
Feb 21 21:37:26 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
42 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:37:26 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
43 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.21 2
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.22 (null)
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=22
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_
resources/lrm_resource[@id='natcpcoreip']/lrm_rsc_op[@id='natcpcoreip_last_0']:
@operation_key=natcpcoreip_start_0, @operation=start,
@crm-debug-origin=do_update_resource, @transition-key=38:0:0:
e27e11f8-fc4e-493e-a4f2-80a02cb1f079, @transition-magic=0:0;38:0:0:
e27e11f8-fc4e-493e-a4f2-80a02cb1f079, @call-id=139, @rc-code=0,
@last-run=1519229246, @last-rc-change=1519229246, @exec-time=163
IPaddr2(nataccessip)[1340]: 2018/02/21_21:37:26 INFO: Bringing device
nataccess up
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/crmd/84,
version=0.108.22)
Feb 21 21:37:26 [24024] IVM-1 crmd: info: match_graph_event: Action
natcpcoreip_start_0 (38) confirmed on IVM-1 (rc=0)
Feb 21 21:37:26 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
monitor operation natcpcoreip_monitor_10000 locally on IVM-1 | action 39
Feb 21 21:37:26 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=39:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079 op=natcpcoreip_monitor_10000
Feb 21 21:37:26 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
promote operation stateful_apache2_promote_0 locally on IVM-1 | action 10
Feb 21 21:37:26 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=10:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079
op=stateful_apache2_promote_0
IPaddr2(omsip)[1355]: 2018/02/21_21:37:26 INFO: Bringing device oms up
Feb 21 21:37:26 [24021] IVM-1 lrmd: info: log_execute: executing
- rsc:stateful_apache2 action:promote call_id:147
IPaddr2(omsip)[1355]: 2018/02/21_21:37:26 INFO: /usr/libexec/heartbeat/send_arp
-i 200 -r 5 -p /usr/var/run/resource-agents/send_arp-10.53.201.209 oms
10.53.201.209 auto not_used not_used
Feb 21 21:37:26 [24021] IVM-1 lrmd: info: log_finished: finished
- rsc:omsip action:start call_id:145 pid:1355 exit-code:0 exec-time:205ms
queue-time:0ms
IPaddr2(nataccessip)[1340]: 2018/02/21_21:37:26 INFO:
/usr/libexec/heartbeat/send_arp -i 200 -r 5 -p
/usr/var/run/resource-agents/send_arp-192.168.3.2
nataccess 192.168.3.2 auto not_used not_used
Feb 21 21:37:26 [24021] IVM-1 lrmd: info: log_finished: finished
- rsc:nataccessip action:start call_id:142 pid:1340 exit-code:0
exec-time:222ms queue-time:0ms
Feb 21 21:37:26 [24024] IVM-1 crmd: info: action_synced_wait: Managed
IPaddr2_meta-data_0 process 2089 exited with rc=0
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/crmd/85)
Feb 21 21:37:26 [24024] IVM-1 crmd: notice: process_lrm_event: Result
of start operation for omsip on IVM-1: 0 (ok) | call=145 key=omsip_start_0
confirmed=true cib-update=85
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.22 2
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.23 (null)
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=23
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_
resources/lrm_resource[@id='omsip']/lrm_rsc_op[@id='omsip_last_0']:
@operation_key=omsip_start_0, @operation=start,
@crm-debug-origin=do_update_resource, @transition-key=218:0:0:
e27e11f8-fc4e-493e-a4f2-80a02cb1f079, @transition-magic=0:0;218:0:0:
e27e11f8-fc4e-493e-a4f2-80a02cb1f079, @call-id=145, @rc-code=0,
@last-run=1519229246, @last-rc-change=1519229246, @exec-time=205
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/crmd/85,
version=0.108.23)
Feb 21 21:37:26 [24024] IVM-1 crmd: info: action_synced_wait: Managed
IPaddr2_meta-data_0 process 2170 exited with rc=0
Feb 21 21:37:26 [24024] IVM-1 crmd: notice: process_lrm_event: Result
of start operation for nataccessip on IVM-1: 0 (ok) | call=142
key=nataccessip_start_0 confirmed=true cib-update=86
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/crmd/86)
Feb 21 21:37:26 [24024] IVM-1 crmd: info: match_graph_event: Action
omsip_start_0 (218) confirmed on IVM-1 (rc=0)
Feb 21 21:37:26 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
monitor operation omsip_monitor_10000 locally on IVM-1 | action 219
Feb 21 21:37:26 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=219:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079 op=omsip_monitor_10000
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.23 2
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.24 (null)
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=24
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_
resources/lrm_resource[@id='nataccessip']/lrm_rsc_op[@id='nataccessip_last_0']:
@operation_key=nataccessip_start_0, @operation=start,
@crm-debug-origin=do_update_resource, @transition-key=162:0:0:
e27e11f8-fc4e-493e-a4f2-80a02cb1f079, @transition-magic=0:0;162:0:0:
e27e11f8-fc4e-493e-a4f2-80a02cb1f079, @call-id=142, @rc-code=0,
@last-run=1519229246, @last-rc-change=1519229246, @exec-time=2
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/crmd/86,
version=0.108.24)
Feb 21 21:37:26 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
start operation nataaagwip_start_0 locally on IVM-1 | action 220
Feb 21 21:37:26 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=220:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079 op=nataaagwip_start_0
Feb 21 21:37:26 [24021] IVM-1 lrmd: info: log_execute: executing
- rsc:nataaagwip action:start call_id:149
Feb 21 21:37:26 [24024] IVM-1 crmd: info: match_graph_event: Action
nataccessip_start_0 (162) confirmed on IVM-1 (rc=0)
Feb 21 21:37:26 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
monitor operation nataccessip_monitor_10000 locally on IVM-1 | action 163
Feb 21 21:37:26 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=163:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079
op=nataccessip_monitor_10000
Feb 21 21:37:26 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
promote operation haproxy_promote_0 locally on IVM-1 | action 134
Feb 21 21:37:26 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=134:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079 op=haproxy_promote_0
Feb 21 21:37:26 [24021] IVM-1 lrmd: info: log_execute: executing
- rsc:haproxy action:promote call_id:151
Feb 21 21:37:26 [24024] IVM-1 crmd: info: process_lrm_event: Result
of monitor operation for natcpcoreip on IVM-1: 0 (ok) | call=146
key=natcpcoreip_monitor_10000 confirmed=false cib-update=87
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/crmd/87)
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.24 2
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.25 (null)
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=25
Feb 21 21:37:26 [24019] IVM-1 cib: info: cib_perform_op: ++
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/
lrm_resource[@id='natcpcoreip']: <lrm_rsc_op
id="natcpcoreip_monitor_10000" operation_key="natcpcoreip_monitor_10000"
operation="monitor" crm-debug-origin="do_update_resource"
crm_feature_set="3.0.11"
transition-key="39:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079"
transition-magic="0:0;39:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079"
on_node="IVM-1" call-id="146" rc-code="0" op-status
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/crmd/87,
version=0.108.25)
Feb 21 21:37:26 [24024] IVM-1 crmd: info: match_graph_event: Action
natcpcoreip_monitor_10000 (39) confirmed on IVM-1 (rc=0)
Feb 21 21:37:26 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
44 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:37:26 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
45 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:37:26 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
46 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:37:26 [24022] IVM-1 attrd: info:
attrd_client_update: Starting
an election to determine the writer
Feb 21 21:37:26 [24022] IVM-1 attrd: info: election_complete: Election
election-attrd complete
Feb 21 21:37:26 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
47 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:37:26 [24022] IVM-1 attrd: info: write_attribute: Sent
update 28 with 1 changes for master-RoutingManager, id=<n/a>, set=(null)
Feb 21 21:37:26 [24022] IVM-1 attrd: info: write_attribute: Sent
update 29 with 1 changes for fail-count-haproxy, id=<n/a>, set=(null)
Feb 21 21:37:26 [24022] IVM-1 attrd: info: write_attribute: Sent
update 30 with 1 changes for master-alarmSnmpd, id=<n/a>, set=(null)
Feb 21 21:37:26 [24022] IVM-1 attrd: info: write_attribute: Sent
update 31 with 1 changes for terminate, id=<n/a>, set=(null)
Feb 21 21:37:26 [24022] IVM-1 attrd: info: write_attribute: Sent
update 32 with 1 changes for master-stateful_wildfly, id=<n/a>, set=(null)
Feb 21 21:37:26 [24022] IVM-1 attrd: info: write_attribute: Sent
update 33 with 1 changes for master-stateful_consul, id=<n/a>, set=(null)
Feb 21 21:37:26 [24022] IVM-1 attrd: info: write_attribute: Sent
update 34 with 1 changes for master-haproxy, id=<n/a>, set=(null)
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/attrd/28)
Feb 21 21:37:26 [24022] IVM-1 attrd: info: write_attribute: Sent
update 35 with 1 changes for shutdown, id=<n/a>, set=(null)
Feb 21 21:37:26 [24022] IVM-1 attrd: info: write_attribute: Sent
update 36 with 1 changes for master-stateful_apache2, id=<n/a>, set=(null)
Feb 21 21:37:26 [24022] IVM-1 attrd: info: write_attribute: Sent
update 37 with 1 changes for last-failure-haproxy, id=<n/a>, set=(null)
Feb 21 21:37:26 [24022] IVM-1 attrd: info: attrd_peer_update: Setting
master-alarmSnmpd[IVM-1]: 5 -> 10 from IVM-1
Feb 21 21:37:26 [24022] IVM-1 attrd: info: write_attribute: Write
out of 'master-alarmSnmpd' delayed: update 30 in progress
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/attrd/29)
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/attrd/30)
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/attrd/31)
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/attrd/32)
Feb 21 21:37:26 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/attrd/28,
version=0.108.25)
Feb 21 21:37:26 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
28 for master-RoutingManager: OK (0)
Feb 21 21:37:26 [24021] IVM-1 lrmd: notice: operation_finished:
alarmSnmpd_promote_0:1350:stderr [ error: list of process IDs must follow
-p ]
Feb 21 21:37:26 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
28 for master-RoutingManager[IVM-1]=5: OK (0)
Feb 21 21:37:26 [24021] IVM-1 lrmd: notice: operation_finished:
alarmSnmpd_promote_0:1350:stderr [ ]
Feb 21 21:37:26 [24021] IVM-1 lrmd: notice: operation_finished:
alarmSnmpd_promote_0:1350:stderr [ Usage: ]
Feb 21 21:37:26 [24021] IVM-1 lrmd: notice: operation_finished:
alarmSnmpd_promote_0:1350:stderr [ ps [options] ]
Feb 21 21:37:27 [24021] IVM-1 lrmd: notice: operation_finished:
alarmSnmpd_promote_0:1350:stderr [ ]
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.25 2
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.26 (null)
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=26
Feb 21 21:37:27 [24021] IVM-1 lrmd: notice: operation_finished:
alarmSnmpd_promote_0:1350:stderr [ Try 'ps --help
<simple|list|output|threads|misc|all>' ]
Feb 21 21:37:27 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/attrd/29,
version=0.108.26)
Feb 21 21:37:27 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
29 for fail-count-haproxy: OK (0)
Feb 21 21:37:27 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
29 for fail-count-haproxy[IVM-1]=(null): OK (0)
Feb 21 21:37:27 [24021] IVM-1 lrmd: notice: operation_finished:
alarmSnmpd_promote_0:1350:stderr [ or 'ps --help <s|l|o|t|m|a>' ]
Feb 21 21:37:27 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/attrd/30,
version=0.108.26)
Feb 21 21:37:27 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
30 for master-alarmSnmpd: OK (0)
Feb 21 21:37:27 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
30 for master-alarmSnmpd[IVM-1]=5: OK (0)
Feb 21 21:37:27 [24021] IVM-1 lrmd: notice: operation_finished:
alarmSnmpd_promote_0:1350:stderr [ for additional help text. ]
Feb 21 21:37:27 [24022] IVM-1 attrd: info: write_attribute: Sent
update 38 with 1 changes for master-alarmSnmpd, id=<n/a>, set=(null)
Feb 21 21:37:27 [24021] IVM-1 lrmd: notice: operation_finished:
alarmSnmpd_promote_0:1350:stderr [ ]
Feb 21 21:37:27 [24021] IVM-1 lrmd: notice: operation_finished:
alarmSnmpd_promote_0:1350:stderr [ For more details see ps(1). ]
Feb 21 21:37:27 [24021] IVM-1 lrmd: info: log_finished: finished
- rsc:alarmSnmpd action:promote call_id:143 pid:1350 exit-code:0
exec-time:359ms queue-time:0ms
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.26 2
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.27 (null)
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=27
Feb 21 21:37:27 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/attrd/31,
version=0.108.27)
Feb 21 21:37:27 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
31 for terminate: OK (0)
Feb 21 21:37:27 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
31 for terminate[IVM-1]=(null): OK (0)
Feb 21 21:37:27 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/attrd/32,
version=0.108.27)
Feb 21 21:37:27 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/attrd/33)
Feb 21 21:37:27 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
32 for master-stateful_wildfly: OK (0)
Feb 21 21:37:27 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
32 for master-stateful_wildfly[IVM-1]=5: OK (0)
Feb 21 21:37:27 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/attrd/34)
Feb 21 21:37:27 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/attrd/35)
Feb 21 21:37:27 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/attrd/36)
Feb 21 21:37:27 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/attrd/37)
Feb 21 21:37:27 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/attrd/33,
version=0.108.27)
Feb 21 21:37:27 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
33 for master-stateful_consul: OK (0)
Feb 21 21:37:27 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
33 for master-stateful_consul[IVM-1]=5: OK (0)
Feb 21 21:37:27 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/attrd/34,
version=0.108.27)
Feb 21 21:37:27 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
34 for master-haproxy: OK (0)
Feb 21 21:37:27 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
34 for master-haproxy[IVM-1]=5: OK (0)
Feb 21 21:37:27 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/attrd/35,
version=0.108.27)
Feb 21 21:37:27 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
35 for shutdown: OK (0)
Feb 21 21:37:27 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
35 for shutdown[IVM-1]=0: OK (0)
Feb 21 21:37:27 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/attrd/36,
version=0.108.27)
Feb 21 21:37:27 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
36 for master-stateful_apache2: OK (0)
Feb 21 21:37:27 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
36 for master-stateful_apache2[IVM-1]=5: OK (0)
Feb 21 21:37:27 [24024] IVM-1 crmd: notice: process_lrm_event: Result
of promote operation for alarmSnmpd on IVM-1: 0 (ok) | call=143
key=alarmSnmpd_promote_0 confirmed=true cib-update=88
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.27 2
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.28 (null)
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=28
Feb 21 21:37:27 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/attrd/37,
version=0.108.28)
Feb 21 21:37:27 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/crmd/88)
Feb 21 21:37:27 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
37 for last-failure-haproxy: OK (0)
Feb 21 21:37:27 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
37 for last-failure-haproxy[IVM-1]=(null): OK (0)
Feb 21 21:37:27 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/attrd/38)
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.28 2
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.29 (null)
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=29
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: +
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_
resources/lrm_resource[@id='alarmSnmpd']/lrm_rsc_op[@id='alarmSnmpd_last_0']:
@operation_key=alarmSnmpd_promote_0, @operation=promote,
@crm-debug-origin=do_update_resource, @transition-key=104:0:0:
e27e11f8-fc4e-493e-a4f2-80a02cb1f079, @transition-magic=0:0;104:0:0:
e27e11f8-fc4e-493e-a4f2-80a02cb1f079, @call-id=143, @last-run=1519229246,
@last-rc-change=1519229246, @exec-time=359
Feb 21 21:37:27 [24024] IVM-1 crmd: info: process_lrm_event: Result
of monitor operation for nataccessip on IVM-1: 0 (ok) | call=150
key=nataccessip_monitor_10000 confirmed=false cib-update=89
Feb 21 21:37:27 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/crmd/88,
version=0.108.29)
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.29 2
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.30 (null)
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=30
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: +
/cib/status/node_state[@id='2']/transient_attributes[@id='
2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-master-alarmSnmpd']:
@value=10
Feb 21 21:37:27 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/attrd/38,
version=0.108.30)
Feb 21 21:37:27 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
38 for master-alarmSnmpd: OK (0)
Feb 21 21:37:27 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
38 for master-alarmSnmpd[IVM-1]=10: OK (0)
Feb 21 21:37:27 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/crmd/89)
Feb 21 21:37:27 [24024] IVM-1 crmd: info: process_lrm_event: Result
of monitor operation for omsip on IVM-1: 0 (ok) | call=148
key=omsip_monitor_10000 confirmed=false cib-update=90
Feb 21 21:37:27 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/crmd/90)
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.30 2
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.31 (null)
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=31
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: ++
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/
lrm_resource[@id='nataccessip']: <lrm_rsc_op
id="nataccessip_monitor_10000" operation_key="nataccessip_monitor_10000"
operation="monitor" crm-debug-origin="do_update_resource"
crm_feature_set="3.0.11"
transition-key="163:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079"
transition-magic="0:0;163:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079"
on_node="IVM-1" call-id="150" rc-code="0" op-stat
Feb 21 21:37:27 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/crmd/89,
version=0.108.31)
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.31 2
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.32 (null)
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=32
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: ++
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='omsip']:
<lrm_rsc_op id="omsip_monitor_10000" operation_key="omsip_monitor_10000"
operation="monitor" crm-debug-origin="do_update_resource"
crm_feature_set="3.0.11"
transition-key="219:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079"
transition-magic="0:0;219:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079"
on_node="IVM-1" call-id="148" rc-code="0" op-status="0" interval="1
Feb 21 21:37:27 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/crmd/90,
version=0.108.32)
Feb 21 21:37:27 [24024] IVM-1 crmd: info: match_graph_event: Action
alarmSnmpd_promote_0 (104) confirmed on IVM-1 (rc=0)
Feb 21 21:37:27 [24024] IVM-1 crmd: notice:
abort_transition_graph: Transition
aborted by status-2-master-alarmSnmpd doing modify master-alarmSnmpd=10:
Transient attribute change | cib=0.108.30 source=abort_unless_down:331
path=/cib/status/node_state[@id='2']/transient_attributes[@
id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-master-alarmSnmpd']
complete=false
Feb 21 21:37:27 [24024] IVM-1 crmd: info: match_graph_event: Action
nataccessip_monitor_10000 (163) confirmed on IVM-1 (rc=0)
Feb 21 21:37:27 [24024] IVM-1 crmd: info: match_graph_event: Action
omsip_monitor_10000 (219) confirmed on IVM-1 (rc=0)
Feb 21 21:37:27 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
notify operation alarmSnmpd_post_notify_promote_0 locally on IVM-1 | action
286
Feb 21 21:37:27 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=286:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079 op=alarmSnmpd_notify_0
Feb 21 21:37:27 [24021] IVM-1 lrmd: info: log_execute: executing
- rsc:alarmSnmpd action:notify call_id:152
Feb 21 21:37:27 [24021] IVM-1 lrmd: info: log_finished: finished
- rsc:alarmSnmpd action:notify call_id:152 pid:2498 exit-code:3
exec-time:31ms queue-time:0ms
IPaddr2(nataaagwip)[2214]: 2018/02/21_21:37:27 INFO: Adding inet address
192.168.2.1/24 with broadcast address + to device nataaa
Feb 21 21:37:27 [24024] IVM-1 crmd: info: match_graph_event: Action
alarmSnmpd_notify_0 (286) confirmed on IVM-1 (rc=0)
Feb 21 21:37:27 [24024] IVM-1 crmd: notice: process_lrm_event: Result
of notify operation for alarmSnmpd on IVM-1: 0 (ok) | call=152
key=alarmSnmpd_notify_0 confirmed=true cib-update=0
IPaddr2(nataaagwip)[2214]: 2018/02/21_21:37:27 INFO: Bringing device nataaa
up
Feb 21 21:37:27 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
50 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
IPaddr2(nataaagwip)[2214]: 2018/02/21_21:37:27 INFO:
/usr/libexec/heartbeat/send_arp -i 200 -r 5 -p
/usr/var/run/resource-agents/send_arp-192.168.2.1
nataaa 192.168.2.1 auto not_used not_used
Feb 21 21:37:27 [24021] IVM-1 lrmd: info: log_finished: finished
- rsc:nataaagwip action:start call_id:149 pid:2214 exit-code:0
exec-time:175ms queue-time:0ms
Feb 21 21:37:27 [24022] IVM-1 attrd: info: attrd_peer_update: Setting
master-haproxy[IVM-1]: 5 -> 10 from IVM-1
Feb 21 21:37:27 [24022] IVM-1 attrd: info: write_attribute: Sent
update 39 with 1 changes for master-haproxy, id=<n/a>, set=(null)
Feb 21 21:37:27 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/attrd/39)
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.32 2
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.33 (null)
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=33
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: +
/cib/status/node_state[@id='2']/transient_attributes[@id='
2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-master-haproxy']:
@value=10
Feb 21 21:37:27 [24021] IVM-1 lrmd: info: log_finished: finished
- rsc:haproxy action:promote call_id:151 pid:2216 exit-code:0
exec-time:184ms queue-time:0ms
Feb 21 21:37:27 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/attrd/39,
version=0.108.33)
Feb 21 21:37:27 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
39 for master-haproxy: OK (0)
Feb 21 21:37:27 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
39 for master-haproxy[IVM-1]=10: OK (0)
Feb 21 21:37:27 [24024] IVM-1 crmd: info: action_synced_wait: Managed
IPaddr2_meta-data_0 process 2629 exited with rc=0
Feb 21 21:37:27 [24024] IVM-1 crmd: notice: process_lrm_event: Result
of start operation for nataaagwip on IVM-1: 0 (ok) | call=149
key=nataaagwip_start_0 confirmed=true cib-update=91
Feb 21 21:37:27 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/crmd/91)
Feb 21 21:37:27 [24024] IVM-1 crmd: notice: process_lrm_event: Result
of promote operation for haproxy on IVM-1: 0 (ok) | call=151
key=haproxy_promote_0 confirmed=true cib-update=92
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.33 2
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.34 (null)
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=34
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: +
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_
resources/lrm_resource[@id='nataaagwip']/lrm_rsc_op[@id='nataaagwip_last_0']:
@operation_key=nataaagwip_start_0, @operation=start,
@crm-debug-origin=do_update_resource, @transition-key=220:0:0:
e27e11f8-fc4e-493e-a4f2-80a02cb1f079, @transition-magic=0:0;220:0:0:
e27e11f8-fc4e-493e-a4f2-80a02cb1f079, @call-id=149, @rc-code=0,
@last-run=1519229246, @last-rc-change=1519229246, @exec-time=175
Feb 21 21:37:27 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/crmd/91,
version=0.108.34)
Feb 21 21:37:27 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/crmd/92)
Feb 21 21:37:27 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
48 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:37:27 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
49 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:37:27 [24024] IVM-1 crmd: info:
abort_transition_graph: Transition
aborted by status-2-master-haproxy doing modify master-haproxy=10:
Transient attribute change | cib=0.108.33 source=abort_unless_down:331
path=/cib/status/node_state[@id='2']/transient_attributes[@
id='2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-master-haproxy']
complete=false
Feb 21 21:37:27 [24024] IVM-1 crmd: info: match_graph_event: Action
nataaagwip_start_0 (220) confirmed on IVM-1 (rc=0)
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.34 2
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.35 (null)
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=35
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: +
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_
resources/lrm_resource[@id='haproxy']/lrm_rsc_op[@id='haproxy_last_0']:
@operation_key=haproxy_promote_0, @operation=promote,
@crm-debug-origin=do_update_resource, @transition-key=134:0:0:
e27e11f8-fc4e-493e-a4f2-80a02cb1f079, @transition-magic=0:0;134:0:0:
e27e11f8-fc4e-493e-a4f2-80a02cb1f079, @call-id=151, @rc-code=0,
@last-run=1519229246, @last-rc-change=1519229246, @exec-time=184
Feb 21 21:37:27 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/crmd/92,
version=0.108.35)
Feb 21 21:37:27 [24024] IVM-1 crmd: info: match_graph_event: Action
haproxy_promote_0 (134) confirmed on IVM-1 (rc=0)
Feb 21 21:37:27 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
notify operation haproxy_post_notify_promote_0 locally on IVM-1 | action 292
Feb 21 21:37:27 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=292:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079 op=haproxy_notify_0
Feb 21 21:37:27 [24021] IVM-1 lrmd: info: log_execute: executing
- rsc:haproxy action:notify call_id:153
Feb 21 21:37:27 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
51 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:37:27 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
53 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:37:27 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
52 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:37:27 [24021] IVM-1 lrmd: notice:
operation_finished: haproxy_notify_0:2664:stderr
[ psql: could not connect to server: Connection refused ]
Feb 21 21:37:27 [24021] IVM-1 lrmd: notice:
operation_finished: haproxy_notify_0:2664:stderr
[ Is the server running on host "169.254.0.4" and accepting ]
Feb 21 21:37:27 [24021] IVM-1 lrmd: notice:
operation_finished: haproxy_notify_0:2664:stderr
[ TCP/IP connections on port 5433? ]
Feb 21 21:37:27 [24021] IVM-1 lrmd: info: log_finished: finished
- rsc:haproxy action:notify call_id:153 pid:2664 exit-code:0 exec-time:73ms
queue-time:0ms
Feb 21 21:37:27 [24024] IVM-1 crmd: info: match_graph_event: Action
haproxy_notify_0 (292) confirmed on IVM-1 (rc=0)
Feb 21 21:37:27 [24024] IVM-1 crmd: notice: process_lrm_event: Result
of notify operation for haproxy on IVM-1: 0 (ok) | call=153
key=haproxy_notify_0 confirmed=true cib-update=0
Feb 21 21:37:27 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
54 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:37:27 [24022] IVM-1 attrd: info: attrd_peer_update: Setting
master-stateful_apache2[IVM-1]: 5 -> 10 from IVM-1
Feb 21 21:37:27 [24022] IVM-1 attrd: info: write_attribute: Sent
update 40 with 1 changes for master-stateful_apache2, id=<n/a>, set=(null)
Feb 21 21:37:27 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/attrd/40)
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.35 2
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.36 (null)
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=36
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: +
/cib/status/node_state[@id='2']/transient_attributes[@id='
2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-master-stateful_apache2']:
@value=10
Feb 21 21:37:27 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_apache2_promote_0:2021:stderr [ psql: could not connect to server:
Connection refused ]
Feb 21 21:37:27 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_apache2_promote_0:2021:stderr [ Is the server running on host
"169.254.0.4" and accepting ]
Feb 21 21:37:27 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_apache2_promote_0:2021:stderr [ TCP/IP connections on port 5433? ]
Feb 21 21:37:27 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_apache2_promote_0:2021:stderr [ psql: could not connect to server:
Connection refused ]
Feb 21 21:37:27 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_apache2_promote_0:2021:stderr [ Is the server running on host
"169.254.0.4" and accepting ]
Feb 21 21:37:27 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_apache2_promote_0:2021:stderr [ TCP/IP connections on port 5433? ]
Feb 21 21:37:27 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_apache2_promote_0:2021:stderr [ /opt/wlc/tools/
CertificateValidation/OnStartCertificateActivation.sh: line 141: [: ==:
unary operator expected ]
Feb 21 21:37:27 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_apache2_promote_0:2021:stderr [ /opt/wlc/tools/
CertificateValidation/OnStartCertificateActivation.sh: line 148: [: ==:
unary operator expected ]
Feb 21 21:37:27 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_apache2_promote_0:2021:stderr [ /opt/wlc/tools/
CertificateValidation/OnStartCertificateActivation.sh: line 155: [: ==:
unary operator expected ]
Feb 21 21:37:27 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_apache2_promote_0:2021:stderr [
/usr/lib/ocf/resource.d/pacemaker/apache2:
line 190: [: missing `]' ]
Feb 21 21:37:27 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_apache2_promote_0:2021:stderr [ awk: cmd. line:1: warning: escape
sequence `\/' treated as plain `/' ]
Feb 21 21:37:27 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_apache2_promote_0:2021:stderr [ awk: cmd. line:1: warning: escape
sequence `\/' treated as plain `/' ]
Feb 21 21:37:27 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_apache2_promote_0:2021:stderr [ awk: cmd. line:1: warning: escape
sequence `\/' treated as plain `/' ]
Feb 21 21:37:27 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_apache2_promote_0:2021:stderr [ awk: cmd. line:1: warning: escape
sequence `\/' treated as plain `/' ]
Feb 21 21:37:27 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_apache2_promote_0:2021:stderr [ awk: cmd. line:1: warning: escape
sequence `\/' treated as plain `/' ]
Feb 21 21:37:27 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_apache2_promote_0:2021:stderr [ awk: cmd. line:1: warning: escape
sequence `\/' treated as plain `/' ]
Feb 21 21:37:27 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_apache2_promote_0:2021:stderr [ awk: cmd. line:1: warning: escape
sequence `\/' treated as plain `/' ]
Feb 21 21:37:27 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_apache2_promote_0:2021:stderr [ psql: could not connect to server:
Connection refused ]
Feb 21 21:37:27 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_apache2_promote_0:2021:stderr [ Is the server running on host
"169.254.0.4" and accepting ]
Feb 21 21:37:27 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_apache2_promote_0:2021:stderr [ TCP/IP connections on port 5433? ]
Feb 21 21:37:27 [24021] IVM-1 lrmd: info: log_finished: finished
- rsc:stateful_apache2 action:promote call_id:147 pid:2021 exit-code:0
exec-time:644ms queue-time:1ms
Feb 21 21:37:27 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/attrd/40,
version=0.108.36)
Feb 21 21:37:27 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
40 for master-stateful_apache2: OK (0)
Feb 21 21:37:27 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
40 for master-stateful_apache2[IVM-1]=10: OK (0)
Feb 21 21:37:27 [24024] IVM-1 crmd: info:
abort_transition_graph: Transition
aborted by status-2-master-stateful_apache2 doing modify
master-stateful_apache2=10: Transient attribute change | cib=0.108.36
source=abort_unless_down:331 path=/cib/status/node_state[@
id='2']/transient_attributes[@id='2']/instance_attributes[@
id='status-2']/nvpair[@id='status-2-master-stateful_apache2'] complete=false
Feb 21 21:37:27 [24024] IVM-1 crmd: notice: process_lrm_event: Result
of promote operation for stateful_apache2 on IVM-1: 0 (ok) | call=147
key=stateful_apache2_promote_0 confirmed=true cib-update=93
Feb 21 21:37:27 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/crmd/93)
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.36 2
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.37 (null)
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=37
Feb 21 21:37:27 [24019] IVM-1 cib: info: cib_perform_op: +
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_
resources/lrm_resource[@id='stateful_apache2']/lrm_rsc_op[
@id='stateful_apache2_last_0']: @operation_key=stateful_apache2_promote_0,
@operation=promote, @crm-debug-origin=do_update_resource,
@transition-key=10:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079,
@transition-magic=0:0;10:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079,
@call-id=147, @last-run=1519229246, @last-rc-change=1519229246, @exec-t
Feb 21 21:37:27 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/crmd/93,
version=0.108.37)
Feb 21 21:37:27 [24024] IVM-1 crmd: info: match_graph_event: Action
stateful_apache2_promote_0 (10) confirmed on IVM-1 (rc=0)
Feb 21 21:37:27 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
notify operation stateful_apache2_post_notify_promote_0 locally on IVM-1 |
action 268
Feb 21 21:37:27 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=268:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079
op=stateful_apache2_notify_0
Feb 21 21:37:27 [24021] IVM-1 lrmd: info: log_execute: executing
- rsc:stateful_apache2 action:notify call_id:154
Feb 21 21:37:27 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
55 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:37:27 [24021] IVM-1 lrmd: info: log_finished: finished
- rsc:stateful_apache2 action:notify call_id:154 pid:3107 exit-code:0
exec-time:62ms queue-time:0ms
Feb 21 21:37:27 [24024] IVM-1 crmd: info: match_graph_event: Action
stateful_apache2_notify_0 (268) confirmed on IVM-1 (rc=0)
Feb 21 21:37:27 [24024] IVM-1 crmd: notice: process_lrm_event: Result
of notify operation for stateful_apache2 on IVM-1: 0 (ok) | call=154
key=stateful_apache2_notify_0 confirmed=true cib-update=0
Feb 21 21:37:27 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
56 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:37:32 [24019] IVM-1 cib: info: cib_process_ping: Reporting
our current digest to IVM-1: 80b3d0d414bfbf408cde202cb183f46c for 0.108.37
(0x558e28d66b70 0)
Feb 21 21:37:40 [24022] IVM-1 attrd: info: attrd_peer_update: Setting
master-stateful_consul[IVM-1]: 5 -> 10 from IVM-1
Feb 21 21:37:40 [24022] IVM-1 attrd: info: write_attribute: Sent
update 41 with 1 changes for master-stateful_consul, id=<n/a>, set=(null)
Feb 21 21:37:40 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/attrd/41)
Feb 21 21:37:40 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.37 2
Feb 21 21:37:40 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.38 (null)
Feb 21 21:37:40 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=38
Feb 21 21:37:40 [24019] IVM-1 cib: info: cib_perform_op: +
/cib/status/node_state[@id='2']/transient_attributes[@id='
2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-master-stateful_consul']:
@value=10
Feb 21 21:37:40 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/attrd/41,
version=0.108.38)
Feb 21 21:37:40 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_consul_promote_0:864:stderr [ ssh_exchange_identification:
Connection closed by remote host
]
Feb 21 21:37:40 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_consul_promote_0:864:stderr [ rsync: connection unexpectedly
closed (0 bytes received so far) [sender] ]
Feb 21 21:37:40 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_consul_promote_0:864:stderr [ rsync error: unexplained error (code
255) at io.c(226) [sender=3.1.2] ]
Feb 21 21:37:40 [24021] IVM-1 lrmd: info: log_finished: finished
- rsc:stateful_consul action:promote call_id:136 pid:864 exit-code:0
exec-time:14022ms queue-time:0ms
Feb 21 21:37:40 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
41 for master-stateful_consul: OK (0)
Feb 21 21:37:40 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
41 for master-stateful_consul[IVM-1]=10: OK (0)
Feb 21 21:37:40 [24024] IVM-1 crmd: info:
abort_transition_graph: Transition
aborted by status-2-master-stateful_consul doing modify
master-stateful_consul=10: Transient attribute change | cib=0.108.38
source=abort_unless_down:331 path=/cib/status/node_state[@
id='2']/transient_attributes[@id='2']/instance_attributes[@
id='status-2']/nvpair[@id='status-2-master-stateful_consul'] complete=false
Feb 21 21:37:40 [24024] IVM-1 crmd: notice: process_lrm_event: Result
of promote operation for stateful_consul on IVM-1: 0 (ok) | call=136
key=stateful_consul_promote_0 confirmed=true cib-update=94
Feb 21 21:37:40 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/crmd/94)
Feb 21 21:37:40 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.38 2
Feb 21 21:37:40 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.39 (null)
Feb 21 21:37:40 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=39
Feb 21 21:37:40 [24019] IVM-1 cib: info: cib_perform_op: +
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_
resources/lrm_resource[@id='stateful_consul']/lrm_rsc_op[@id='stateful_consul_last_0']:
@operation_key=stateful_consul_promote_0, @operation=promote,
@crm-debug-origin=do_update_resource, @transition-key=48:0:0:
e27e11f8-fc4e-493e-a4f2-80a02cb1f079, @transition-magic=0:0;48:0:0:
e27e11f8-fc4e-493e-a4f2-80a02cb1f079, @call-id=136, @last-run=1519229246,
@last-rc-change=1519229246, @exec-time
Feb 21 21:37:40 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/crmd/94,
version=0.108.39)
Feb 21 21:37:40 [24024] IVM-1 crmd: info: match_graph_event: Action
stateful_consul_promote_0 (48) confirmed on IVM-1 (rc=0)
Feb 21 21:37:40 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
notify operation stateful_consul_post_notify_promote_0 locally on IVM-1 |
action 274
Feb 21 21:37:40 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=274:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079 op=stateful_consul_notify_0
Feb 21 21:37:40 [24021] IVM-1 lrmd: info: log_execute: executing
- rsc:stateful_consul action:notify call_id:155
Feb 21 21:37:40 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
57 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:37:40 [24021] IVM-1 lrmd: info: log_finished: finished
- rsc:stateful_consul action:notify call_id:155 pid:4822 exit-code:0
exec-time:78ms queue-time:0ms
Feb 21 21:37:40 [24024] IVM-1 crmd: info: match_graph_event: Action
stateful_consul_notify_0 (274) confirmed on IVM-1 (rc=0)
Feb 21 21:37:40 [24024] IVM-1 crmd: notice: process_lrm_event: Result
of notify operation for stateful_consul on IVM-1: 0 (ok) | call=155
key=stateful_consul_notify_0 confirmed=true cib-update=0
Feb 21 21:37:40 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
58 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:37:45 [24019] IVM-1 cib: info: cib_process_ping: Reporting
our current digest to IVM-1: 93e4b267c2810ce33ee8d25db0354364 for 0.108.39
(0x558e28d66b70 0)
Feb 21 21:37:54 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:6347:stderr [
/opt/vplat/modules/pacemaker/clusterMon.sh:
line 127: [: too many arguments ]
Feb 21 21:37:54 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:6347:stderr [ resource stateful_consul is NOT
running ]
Feb 21 21:37:54 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:6347:stderr [ resource RoutingManager is NOT
running ]
Feb 21 21:37:54 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:6347:stderr [ resource haproxy is NOT running ]
Feb 21 21:37:54 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:6347:stderr [
/opt/vplat/modules/pacemaker/clusterMon.sh:
line 127: [: too many arguments ]
Feb 21 21:37:54 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:6347:stderr [ resource stateful_wildfly is NOT
running ]
Feb 21 21:37:54 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:6347:stderr [ resource stateful_apache2 is NOT
running ]
Feb 21 21:38:55 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:15092:stderr [
/opt/vplat/modules/pacemaker/clusterMon.sh:
line 127: [: too many arguments ]
Feb 21 21:38:55 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:15092:stderr [ resource stateful_consul is NOT
running ]
Feb 21 21:38:55 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:15092:stderr [ resource RoutingManager is NOT
running ]
Feb 21 21:38:55 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:15092:stderr [ resource haproxy is NOT running ]
Feb 21 21:38:55 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:15092:stderr [
/opt/vplat/modules/pacemaker/clusterMon.sh:
line 127: [: too many arguments ]
Feb 21 21:38:55 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:15092:stderr [ resource stateful_wildfly is NOT
running ]
Feb 21 21:38:55 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:15092:stderr [ resource stateful_apache2 is NOT
running ]
Feb 21 21:39:56 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:24026:stderr [
/opt/vplat/modules/pacemaker/clusterMon.sh:
line 127: [: too many arguments ]
Feb 21 21:39:56 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:24026:stderr [ resource stateful_consul is NOT
running ]
Feb 21 21:39:56 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:24026:stderr [ resource RoutingManager is NOT
running ]
Feb 21 21:39:56 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:24026:stderr [ resource haproxy is NOT running ]
Feb 21 21:39:56 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:24026:stderr [
/opt/vplat/modules/pacemaker/clusterMon.sh:
line 127: [: too many arguments ]
Feb 21 21:39:56 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:24026:stderr [ resource stateful_wildfly is NOT
running ]
Feb 21 21:39:56 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:24026:stderr [ resource stateful_apache2 is NOT
running ]
Feb 21 21:40:56 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:425:stderr [
/opt/vplat/modules/pacemaker/clusterMon.sh:
line 127: [: too many arguments ]
Feb 21 21:40:56 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:425:stderr [ resource stateful_consul is NOT
running ]
Feb 21 21:40:56 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:425:stderr [ resource RoutingManager is NOT
running ]
Feb 21 21:40:56 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:425:stderr [ resource haproxy is NOT running ]
Feb 21 21:40:56 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:425:stderr [
/opt/vplat/modules/pacemaker/clusterMon.sh:
line 127: [: too many arguments ]
Feb 21 21:40:56 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:425:stderr [ resource stateful_wildfly is NOT
running ]
Feb 21 21:40:56 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:425:stderr [ resource stateful_apache2 is NOT
running ]
Feb 21 21:41:57 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:8834:stderr [
/opt/vplat/modules/pacemaker/clusterMon.sh:
line 127: [: too many arguments ]
Feb 21 21:41:57 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:8834:stderr [ resource stateful_consul is NOT
running ]
Feb 21 21:41:57 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:8834:stderr [ resource RoutingManager is NOT
running ]
Feb 21 21:41:57 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:8834:stderr [ resource haproxy is NOT running ]
Feb 21 21:41:57 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:8834:stderr [
/opt/vplat/modules/pacemaker/clusterMon.sh:
line 127: [: too many arguments ]
Feb 21 21:41:57 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:8834:stderr [ resource stateful_wildfly is NOT
running ]
Feb 21 21:41:57 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:8834:stderr [ resource stateful_apache2 is NOT
running ]
Feb 21 21:42:26 [24021] IVM-1 lrmd: warning:
child_timeout_callback: stateful_wildfly_promote_0
process (PID 869) timed out
Feb 21 21:42:26 [24021] IVM-1 lrmd: warning:
operation_finished: stateful_wildfly_promote_0:869
- timed out after 300000ms
Feb 21 21:42:26 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_promote_0:869:stderr [ ln: failed to create symbolic link
'/usr/java/jre/jre1.8.0_152': File exists ]
Feb 21 21:42:26 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_promote_0:869:stderr [ psql: could not connect to server:
Connection refused ]
Feb 21 21:42:26 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_promote_0:869:stderr [ Is the server running on host
"169.254.0.4" and accepting ]
Feb 21 21:42:26 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_promote_0:869:stderr [ TCP/IP connections on port 5433? ]
Feb 21 21:42:26 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_promote_0:869:stderr [ psql: could not connect to server:
Connection refused ]
Feb 21 21:42:26 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_promote_0:869:stderr [ Is the server running on host
"169.254.0.4" and accepting ]
Feb 21 21:42:26 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_promote_0:869:stderr [ TCP/IP connections on port 5433? ]
Feb 21 21:42:26 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_promote_0:869:stderr [ psql: could not connect to server:
Connection refused ]
Feb 21 21:42:26 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_promote_0:869:stderr [ Is the server running on host
"169.254.0.4" and accepting ]
Feb 21 21:42:26 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_promote_0:869:stderr [ TCP/IP connections on port 5433? ]
Feb 21 21:42:26 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_promote_0:869:stderr [ psql: could not connect to server:
Connection refused ]
Feb 21 21:42:26 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_promote_0:869:stderr [ Is the server running on host
"169.254.0.4" and accepting ]
Feb 21 21:42:26 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_promote_0:869:stderr [ TCP/IP connections on port 5433? ]
Feb 21 21:42:26 [24021] IVM-1 lrmd: info: log_finished: finished
- rsc:stateful_wildfly action:promote call_id:137 pid:869 exit-code:1
exec-time:300003ms queue-time:0ms
Feb 21 21:42:26 [24024] IVM-1 crmd: error: process_lrm_event: Result
of promote operation for stateful_wildfly on IVM-1: Timed Out | call=137
key=stateful_wildfly_promote_0 timeout=300000ms
Feb 21 21:42:26 [24024] IVM-1 crmd: notice: process_lrm_event:
IVM-1-stateful_wildfly_promote_0:137 [ ln: failed to create symbolic link
'/usr/java/jre/jre1.8.0_152': File exists\npsql: could not connect to
server: Connection refused\n Is the server running on host "169.254.0.4"
and accepting\n TCP/IP connections on port 5433?\npsql: could not
connect to server: Connection refused\n Is the server running on host
"169.254.0.4" and accepting\n TCP/IP connections on port 5433?\npsql:
could
Feb 21 21:42:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/crmd/95)
Feb 21 21:42:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.39 2
Feb 21 21:42:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.40 (null)
Feb 21 21:42:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=40
Feb 21 21:42:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_
resources/lrm_resource[@id='stateful_wildfly']/lrm_rsc_op[
@id='stateful_wildfly_last_0']: @operation_key=stateful_wildfly_promote_0,
@operation=promote, @crm-debug-origin=do_update_resource,
@transition-key=76:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079,
@transition-magic=2:1;76:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079,
@call-id=137, @rc-code=1, @op-status=2, @last-run=1519229246, @last-rc-
Feb 21 21:42:26 [24019] IVM-1 cib: info: cib_perform_op: ++
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/
lrm_resource[@id='stateful_wildfly']: <lrm_rsc_op
id="stateful_wildfly_last_failure_0" operation_key="stateful_wildfly_promote_0"
operation="promote" crm-debug-origin="do_update_resource"
crm_feature_set="3.0.11"
transition-key="76:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079"
transition-magic="2:1;76:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079"
on_node="IVM-1" call-id="137" rc-code="
Feb 21 21:42:26 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/crmd/95,
version=0.108.40)
Feb 21 21:42:26 [24024] IVM-1 crmd: warning: status_from_rc: Action
76 (stateful_wildfly_promote_0) on IVM-1 failed (target: 0 vs. rc: 1): Error
Feb 21 21:42:26 [24024] IVM-1 crmd: info:
abort_transition_graph: Transition
aborted by operation stateful_wildfly_promote_0 'modify' on IVM-1: Event
failed | magic=2:1;76:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079 cib=0.108.40
source=match_graph_event:310 complete=false
Feb 21 21:42:26 [24024] IVM-1 crmd: info: match_graph_event: Action
stateful_wildfly_promote_0 (76) confirmed on IVM-1 (rc=1)
Feb 21 21:42:26 [24024] IVM-1 crmd: info: update_failcount: Updating
failcount for stateful_wildfly on IVM-1 after failed promote: rc=1
(update=value++, time=1519229546)
Feb 21 21:42:26 [24024] IVM-1 crmd: info:
process_graph_event: Detected
action (0.76) stateful_wildfly_promote_0.137=unknown error: failed
Feb 21 21:42:26 [24024] IVM-1 crmd: warning: status_from_rc: Action
76 (stateful_wildfly_promote_0) on IVM-1 failed (target: 0 vs. rc: 1): Error
Feb 21 21:42:26 [24024] IVM-1 crmd: info:
abort_transition_graph: Transition
aborted by operation stateful_wildfly_promote_0 'create' on IVM-1: Event
failed | magic=2:1;76:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079 cib=0.108.40
source=match_graph_event:310 complete=false
Feb 21 21:42:26 [24024] IVM-1 crmd: info: match_graph_event: Action
stateful_wildfly_promote_0 (76) confirmed on IVM-1 (rc=1)
Feb 21 21:42:26 [24022] IVM-1 attrd: info:
attrd_client_update: Expanded
fail-count-stateful_wildfly=value++ to 1
Feb 21 21:42:26 [24024] IVM-1 crmd: info: update_failcount: Updating
failcount for stateful_wildfly on IVM-1 after failed promote: rc=1
(update=value++, time=1519229546)
Feb 21 21:42:26 [24024] IVM-1 crmd: info:
process_graph_event: Detected
action (0.76) stateful_wildfly_promote_0.137=unknown error: failed
Feb 21 21:42:26 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
notify operation stateful_wildfly_post_notify_promote_0 locally on IVM-1 |
action 280
Feb 21 21:42:26 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=280:0:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079
op=stateful_wildfly_notify_0
Feb 21 21:42:26 [24022] IVM-1 attrd: info: attrd_peer_update: Setting
fail-count-stateful_wildfly[IVM-1]: (null) -> 1 from IVM-1
Feb 21 21:42:26 [24022] IVM-1 attrd: info: write_attribute: Sent
update 42 with 1 changes for fail-count-stateful_wildfly, id=<n/a>,
set=(null)
Feb 21 21:42:26 [24021] IVM-1 lrmd: info: log_execute: executing
- rsc:stateful_wildfly action:notify call_id:156
Feb 21 21:42:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/attrd/42)
Feb 21 21:42:26 [24022] IVM-1 attrd: info:
attrd_client_update: Expanded
fail-count-stateful_wildfly=value++ to 2
Feb 21 21:42:26 [24022] IVM-1 attrd: info: attrd_peer_update: Setting
last-failure-stateful_wildfly[IVM-1]: (null) -> 1519229546 from IVM-1
Feb 21 21:42:26 [24022] IVM-1 attrd: info: write_attribute: Sent
update 43 with 1 changes for last-failure-stateful_wildfly, id=<n/a>,
set=(null)
Feb 21 21:42:26 [24022] IVM-1 attrd: info: attrd_peer_update: Setting
fail-count-stateful_wildfly[IVM-1]: 1 -> 2 from IVM-1
Feb 21 21:42:26 [24022] IVM-1 attrd: info: write_attribute: Write
out of 'fail-count-stateful_wildfly' delayed: update 42 in progress
Feb 21 21:42:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.40 2
Feb 21 21:42:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.41 (null)
Feb 21 21:42:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=41
Feb 21 21:42:26 [24019] IVM-1 cib: info: cib_perform_op: ++
/cib/status/node_state[@id='2']/transient_attributes[@id='2'
]/instance_attributes[@id='status-2']: <nvpair
id="status-2-fail-count-stateful_wildfly"
name="fail-count-stateful_wildfly" value="1"/>
Feb 21 21:42:26 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/attrd/42,
version=0.108.41)
Feb 21 21:42:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/attrd/43)
Feb 21 21:42:26 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
42 for fail-count-stateful_wildfly: OK (0)
Feb 21 21:42:26 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
42 for fail-count-stateful_wildfly[IVM-1]=1: OK (0)
Feb 21 21:42:26 [24022] IVM-1 attrd: info: write_attribute: Sent
update 44 with 1 changes for fail-count-stateful_wildfly, id=<n/a>,
set=(null)
Feb 21 21:42:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/attrd/44)
Feb 21 21:42:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.41 2
Feb 21 21:42:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.42 (null)
Feb 21 21:42:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=42
Feb 21 21:42:26 [24019] IVM-1 cib: info: cib_perform_op: ++
/cib/status/node_state[@id='2']/transient_attributes[@id='2'
]/instance_attributes[@id='status-2']: <nvpair
id="status-2-last-failure-stateful_wildfly"
name="last-failure-stateful_wildfly" value="1519229546"/>
Feb 21 21:42:26 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/attrd/43,
version=0.108.42)
Feb 21 21:42:26 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
43 for last-failure-stateful_wildfly: OK (0)
Feb 21 21:42:26 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
43 for last-failure-stateful_wildfly[IVM-1]=1519229546: OK (0)
Feb 21 21:42:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.42 2
Feb 21 21:42:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.43 (null)
Feb 21 21:42:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=43
Feb 21 21:42:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib/status/node_state[@id='2']/transient_attributes[@id='
2']/instance_attributes[@id='status-2']/nvpair[@id='status-
2-fail-count-stateful_wildfly']: @value=2
Feb 21 21:42:26 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/attrd/44,
version=0.108.43)
Feb 21 21:42:26 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
44 for fail-count-stateful_wildfly: OK (0)
Feb 21 21:42:26 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
44 for fail-count-stateful_wildfly[IVM-1]=2: OK (0)
Feb 21 21:42:26 [24024] IVM-1 crmd: info:
abort_transition_graph: Transition
aborted by status-2-fail-count-stateful_wildfly doing create
fail-count-stateful_wildfly=1: Transient attribute change | cib=0.108.41
source=abort_unless_down:331 path=/cib/status/node_state[@
id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']
complete=false
Feb 21 21:42:26 [24024] IVM-1 crmd: info:
abort_transition_graph: Transition
aborted by status-2-last-failure-stateful_wildfly doing create
last-failure-stateful_wildfly=1519229546: Transient attribute change |
cib=0.108.42 source=abort_unless_down:331 path=/cib/status/node_state[@
id='2']/transient_attributes[@id='2']/instance_attributes[@id='status-2']
complete=false
Feb 21 21:42:26 [24024] IVM-1 crmd: info:
abort_transition_graph: Transition
aborted by status-2-fail-count-stateful_wildfly doing modify
fail-count-stateful_wildfly=2: Transient attribute change | cib=0.108.43
source=abort_unless_down:331 path=/cib/status/node_state[@
id='2']/transient_attributes[@id='2']/instance_attributes[@
id='status-2']/nvpair[@id='status-2-fail-count-stateful_wildfly']
complete=false
Feb 21 21:42:26 [24021] IVM-1 lrmd: info: log_finished: finished
- rsc:stateful_wildfly action:notify call_id:156 pid:13295 exit-code:0
exec-time:65ms queue-time:0ms
Feb 21 21:42:26 [24024] IVM-1 crmd: info: match_graph_event: Action
stateful_wildfly_notify_0 (280) confirmed on IVM-1 (rc=0)
Feb 21 21:42:26 [24024] IVM-1 crmd: notice: process_lrm_event: Result
of notify operation for stateful_wildfly on IVM-1: 0 (ok) | call=156
key=stateful_wildfly_notify_0 confirmed=true cib-update=0
Feb 21 21:42:26 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
59 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:42:26 [24024] IVM-1 crmd: notice: run_graph: Transition 0
(Complete=69, Pending=0, Fired=0, Skipped=6, Incomplete=24,
Source=/usr/var/lib/pacemaker/pengine/pe-input-106.bz2): Stopped
Feb 21 21:42:26 [24024] IVM-1 crmd: info: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC
cause=C_FSA_INTERNAL origin=notify_crmd
Feb 21 21:42:26 [24023] IVM-1 pengine: notice: unpack_config: On loss
of CCM Quorum: Ignore
Feb 21 21:42:26 [24023] IVM-1 pengine: info:
determine_online_status: Node
IVM-1 is online
Feb 21 21:42:26 [24023] IVM-1 pengine: info:
determine_op_status: Operation
monitor found resource haproxy:0 active on IVM-1
Feb 21 21:42:26 [24023] IVM-1 pengine: info:
get_failcount_full: stateful_wildfly:0
has failed 2 times on IVM-1
Feb 21 21:42:26 [24023] IVM-1 pengine: info:
get_failcount_full: stateful_wildfly:0
has failed 2 times on IVM-1
Feb 21 21:42:26 [24023] IVM-1 pengine: warning:
unpack_rsc_op_failure: Processing
failed op promote for stateful_wildfly:0 on IVM-1: unknown error (1)
Feb 21 21:42:26 [24023] IVM-1 pengine: info:
get_failcount_full: stateful_wildfly:0
has failed 2 times on IVM-1
Feb 21 21:42:26 [24023] IVM-1 pengine: info:
get_failcount_full: stateful_wildfly:0
has failed 2 times on IVM-1
Feb 21 21:42:26 [24023] IVM-1 pengine: warning:
unpack_rsc_op_failure: Processing
failed op promote for stateful_wildfly:0 on IVM-1: unknown error (1)
Feb 21 21:42:26 [24023] IVM-1 pengine: info: clone_print: Master/Slave
Set: apache2_MS [stateful_apache2]
Feb 21 21:42:26 [24023] IVM-1 pengine: info: short_print:
Masters: [ IVM-1 ]
Feb 21 21:42:26 [24023] IVM-1 pengine: info: short_print:
Stopped: [ IVM-0 ]
Feb 21 21:42:26 [24023] IVM-1 pengine: info: native_print: natomsip
(ocf::heartbeat:IPaddr2): Started IVM-1
Feb 21 21:42:26 [24023] IVM-1 pengine: info: native_print:
natcpcoreip (ocf::heartbeat:IPaddr2): Started IVM-1
Feb 21 21:42:26 [24023] IVM-1 pengine: info: clone_print: Clone Set:
ClusterMon-clone [ClusterMon]
Feb 21 21:42:26 [24023] IVM-1 pengine: info: short_print:
Started: [ IVM-1 ]
Feb 21 21:42:26 [24023] IVM-1 pengine: info: short_print:
Stopped: [ IVM-0 ]
Feb 21 21:42:26 [24023] IVM-1 pengine: info: clone_print: Master/Slave
Set: stateful_ms_set [stateful_consul]
Feb 21 21:42:26 [24023] IVM-1 pengine: info: short_print:
Masters: [ IVM-1 ]
Feb 21 21:42:26 [24023] IVM-1 pengine: info: short_print:
Stopped: [ IVM-0 ]
Feb 21 21:42:26 [24023] IVM-1 pengine: info: clone_print: Master/Slave
Set: wildfly_MS [stateful_wildfly]
Feb 21 21:42:26 [24023] IVM-1 pengine: info: native_print:
stateful_wildfly (ocf::pacemaker:wildfly): FAILED Master IVM-1
Feb 21 21:42:26 [24023] IVM-1 pengine: info: short_print:
Stopped: [ IVM-0 ]
Feb 21 21:42:26 [24023] IVM-1 pengine: info: clone_print: Master/Slave
Set: alarmSnmpd_MS [alarmSnmpd]
Feb 21 21:42:26 [24023] IVM-1 pengine: info: short_print:
Masters: [ IVM-1 ]
Feb 21 21:42:26 [24023] IVM-1 pengine: info: short_print:
Stopped: [ IVM-0 ]
Feb 21 21:42:26 [24023] IVM-1 pengine: info: native_print:
natomsip_for_snmpd (ocf::heartbeat:IPaddr2): Started IVM-1
Feb 21 21:42:26 [24023] IVM-1 pengine: info: clone_print: Master/Slave
Set: haproxy_MS [haproxy]
Feb 21 21:42:26 [24023] IVM-1 pengine: info: short_print:
Masters: [ IVM-1 ]
Feb 21 21:42:26 [24023] IVM-1 pengine: info: short_print:
Stopped: [ IVM-0 ]
Feb 21 21:42:26 [24023] IVM-1 pengine: info: native_print: nataaaip
(ocf::heartbeat:IPaddr2): Started IVM-1
Feb 21 21:42:26 [24023] IVM-1 pengine: info: native_print:
nataccessip (ocf::heartbeat:IPaddr2): Started IVM-1
Feb 21 21:42:26 [24023] IVM-1 pengine: info: clone_print: Master/Slave
Set: netactagent_MS [netactagent]
Feb 21 21:42:26 [24023] IVM-1 pengine: info: short_print:
Stopped: [ IVM-0 IVM-1 ]
Feb 21 21:42:26 [24023] IVM-1 pengine: info: native_print:
natomsip_for_netact (ocf::heartbeat:IPaddr2): Stopped
Feb 21 21:42:26 [24023] IVM-1 pengine: info: clone_print: Master/Slave
Set: RoutingManager_MS [RoutingManager]
Feb 21 21:42:26 [24023] IVM-1 pengine: info: short_print:
Slaves: [ IVM-1 ]
Feb 21 21:42:26 [24023] IVM-1 pengine: info: short_print:
Stopped: [ IVM-0 ]
Feb 21 21:42:26 [24023] IVM-1 pengine: info: group_print: Resource
Group: floatingips
Feb 21 21:42:26 [24023] IVM-1 pengine: info: native_print:
natomsgwip (ocf::heartbeat:IPaddr2): Started IVM-1
Feb 21 21:42:26 [24023] IVM-1 pengine: info: native_print: omsip
(ocf::heartbeat:IPaddr2): Started IVM-1
Feb 21 21:42:26 [24023] IVM-1 pengine: info: native_print:
nataaagwip (ocf::heartbeat:IPaddr2): Started IVM-1
Feb 21 21:42:26 [24023] IVM-1 pengine: info: native_print: aaaip
(ocf::heartbeat:IPaddr2): Stopped
Feb 21 21:42:26 [24023] IVM-1 pengine: info: native_print:
nataccessgwip (ocf::heartbeat:IPaddr2): Stopped
Feb 21 21:42:26 [24023] IVM-1 pengine: info: native_print:
accessip (ocf::heartbeat:IPaddr2): Stopped
Feb 21 21:42:26 [24023] IVM-1 pengine: info: native_print:
natcpcoregwip (ocf::heartbeat:IPaddr2): Stopped
Feb 21 21:42:26 [24023] IVM-1 pengine: info: native_print:
cpcoreip (ocf::heartbeat:IPaddr2): Stopped
Feb 21 21:42:26 [24023] IVM-1 pengine: info:
get_failcount_full: stateful_wildfly:0
has failed 2 times on IVM-1
Feb 21 21:42:26 [24023] IVM-1 pengine: info:
check_migration_threshold: wildfly_MS can fail 1 more times on IVM-1 before
being forced off
Feb 21 21:42:26 [24023] IVM-1 pengine: info:
get_failcount_full: wildfly_MS
has failed 2 times on IVM-1
Feb 21 21:42:26 [24023] IVM-1 pengine: info:
check_migration_threshold: wildfly_MS can fail 1 more times on IVM-1 before
being forced off
Feb 21 21:42:26 [24023] IVM-1 pengine: info: native_color: Resource
stateful_apache2:1 cannot run anywhere
Feb 21 21:42:26 [24023] IVM-1 pengine: info: master_color: Promoting
stateful_apache2:0 (Master IVM-1)
Feb 21 21:42:26 [24023] IVM-1 pengine: info: master_color: apache2_MS:
Promoted 1 instances of a possible 1 to master
Feb 21 21:42:26 [24023] IVM-1 pengine: info: native_color: Resource
ClusterMon:1 cannot run anywhere
Feb 21 21:42:26 [24023] IVM-1 pengine: info: native_color: Resource
stateful_consul:1 cannot run anywhere
Feb 21 21:42:26 [24023] IVM-1 pengine: info: master_color: Promoting
stateful_consul:0 (Master IVM-1)
Feb 21 21:42:26 [24023] IVM-1 pengine: info: master_color:
stateful_ms_set:
Promoted 1 instances of a possible 1 to master
Feb 21 21:42:26 [24023] IVM-1 pengine: info: native_color: Resource
stateful_wildfly:1 cannot run anywhere
Feb 21 21:42:26 [24023] IVM-1 pengine: info: master_color: wildfly_MS:
Promoted 0 instances of a possible 1 to master
Feb 21 21:42:26 [24023] IVM-1 pengine: info: native_color: Resource
alarmSnmpd:1 cannot run anywhere
Feb 21 21:42:26 [24023] IVM-1 pengine: info: master_color: Promoting
alarmSnmpd:0 (Master IVM-1)
Feb 21 21:42:26 [24023] IVM-1 pengine: info: master_color:
alarmSnmpd_MS:
Promoted 1 instances of a possible 1 to master
Feb 21 21:42:26 [24023] IVM-1 pengine: info: native_color: Resource
haproxy:1 cannot run anywhere
Feb 21 21:42:26 [24023] IVM-1 pengine: info: master_color: Promoting
haproxy:0 (Master IVM-1)
Feb 21 21:42:26 [24023] IVM-1 pengine: info: master_color: haproxy_MS:
Promoted 1 instances of a possible 1 to master
Feb 21 21:42:26 [24023] IVM-1 pengine: info: rsc_merge_weights:
netactagent_MS:
Rolling back scores from natomsip_for_netact
Feb 21 21:42:26 [24023] IVM-1 pengine: info: native_color: Resource
netactagent:0 cannot run anywhere
Feb 21 21:42:26 [24023] IVM-1 pengine: info: native_color: Resource
netactagent:1 cannot run anywhere
Feb 21 21:42:26 [24023] IVM-1 pengine: info: rsc_merge_weights:
netactagent_MS:
Rolling back scores from natomsip_for_netact
Feb 21 21:42:26 [24023] IVM-1 pengine: info: master_color:
netactagent_MS:
Promoted 0 instances of a possible 1 to master
Feb 21 21:42:26 [24023] IVM-1 pengine: info: native_color: Resource
natomsip_for_netact cannot run anywhere
Feb 21 21:42:26 [24023] IVM-1 pengine: info: native_color: Resource
RoutingManager:1 cannot run anywhere
Feb 21 21:42:26 [24023] IVM-1 pengine: info: master_color: Promoting
RoutingManager:0 (Slave IVM-1)
Feb 21 21:42:26 [24023] IVM-1 pengine: info: master_color:
RoutingManager_MS:
Promoted 1 instances of a possible 1 to master
Feb 21 21:42:26 [24023] IVM-1 pengine: info: RecurringOp: Start
recurring monitor (30s) for stateful_apache2:0 on IVM-1
Feb 21 21:42:26 [24023] IVM-1 pengine: info: RecurringOp: Start
recurring monitor (30s) for stateful_apache2:0 on IVM-1
Feb 21 21:42:26 [24023] IVM-1 pengine: info: RecurringOp: Start
recurring monitor (5s) for stateful_consul:0 on IVM-1
Feb 21 21:42:26 [24023] IVM-1 pengine: info: RecurringOp: Start
recurring monitor (5s) for stateful_consul:0 on IVM-1
Feb 21 21:42:26 [24023] IVM-1 pengine: info: RecurringOp: Start
recurring monitor (80s) for stateful_wildfly:0 on IVM-1
Feb 21 21:42:26 [24023] IVM-1 pengine: info: RecurringOp: Start
recurring monitor (80s) for stateful_wildfly:0 on IVM-1
Feb 21 21:42:26 [24023] IVM-1 pengine: info: RecurringOp: Start
recurring monitor (20s) for alarmSnmpd:0 on IVM-1
Feb 21 21:42:26 [24023] IVM-1 pengine: info: RecurringOp: Start
recurring monitor (20s) for alarmSnmpd:0 on IVM-1
Feb 21 21:42:26 [24023] IVM-1 pengine: info: RecurringOp: Start
recurring monitor (20s) for haproxy:0 on IVM-1
Feb 21 21:42:26 [24023] IVM-1 pengine: info: RecurringOp: Start
recurring monitor (20s) for haproxy:0 on IVM-1
Feb 21 21:42:26 [24023] IVM-1 pengine: info: RecurringOp: Start
recurring monitor (20s) for RoutingManager:0 on IVM-1
Feb 21 21:42:26 [24023] IVM-1 pengine: info: RecurringOp: Start
recurring monitor (20s) for RoutingManager:0 on IVM-1
Feb 21 21:42:26 [24023] IVM-1 pengine: info: RecurringOp: Start
recurring monitor (10s) for nataaagwip on IVM-1
Feb 21 21:42:26 [24023] IVM-1 pengine: info: RecurringOp: Start
recurring monitor (10s) for aaaip on IVM-1
Feb 21 21:42:26 [24023] IVM-1 pengine: info: RecurringOp: Start
recurring monitor (10s) for nataccessgwip on IVM-1
Feb 21 21:42:26 [24023] IVM-1 pengine: info: RecurringOp: Start
recurring monitor (10s) for accessip on IVM-1
Feb 21 21:42:26 [24023] IVM-1 pengine: info: RecurringOp: Start
recurring monitor (10s) for natcpcoregwip on IVM-1
Feb 21 21:42:26 [24023] IVM-1 pengine: info: RecurringOp: Start
recurring monitor (10s) for cpcoreip on IVM-1
Feb 21 21:42:26 [24023] IVM-1 pengine: info: LogActions: Leave
stateful_apache2:0 (Master IVM-1)
Feb 21 21:42:26 [24023] IVM-1 pengine: info: LogActions: Leave
stateful_apache2:1 (Stopped)
Feb 21 21:42:26 [24023] IVM-1 pengine: info: LogActions: Leave
natomsip (Started IVM-1)
Feb 21 21:42:26 [24023] IVM-1 pengine: info: LogActions: Leave
natcpcoreip (Started IVM-1)
Feb 21 21:42:26 [24023] IVM-1 pengine: info: LogActions: Leave
ClusterMon:0 (Started IVM-1)
Feb 21 21:42:26 [24023] IVM-1 pengine: info: LogActions: Leave
ClusterMon:1 (Stopped)
Feb 21 21:42:26 [24023] IVM-1 pengine: info: LogActions: Leave
stateful_consul:0 (Master IVM-1)
Feb 21 21:42:26 [24023] IVM-1 pengine: info: LogActions: Leave
stateful_consul:1 (Stopped)
Feb 21 21:42:26 [24023] IVM-1 pengine: notice: LogActions: Demote
stateful_wildfly:0 (Master -> Slave IVM-1)
Feb 21 21:42:26 [24023] IVM-1 pengine: notice: LogActions: Recover
stateful_wildfly:0 (Master IVM-1)
Feb 21 21:42:26 [24023] IVM-1 pengine: info: LogActions: Leave
stateful_wildfly:1 (Stopped)
Feb 21 21:42:26 [24023] IVM-1 pengine: info: LogActions: Leave
alarmSnmpd:0 (Master IVM-1)
Feb 21 21:42:26 [24023] IVM-1 pengine: info: LogActions: Leave
alarmSnmpd:1 (Stopped)
Feb 21 21:42:26 [24023] IVM-1 pengine: info: LogActions: Leave
natomsip_for_snmpd (Started IVM-1)
Feb 21 21:42:26 [24023] IVM-1 pengine: info: LogActions: Leave
haproxy:0 (Master IVM-1)
Feb 21 21:42:26 [24023] IVM-1 pengine: info: LogActions: Leave
haproxy:1 (Stopped)
Feb 21 21:42:26 [24023] IVM-1 pengine: info: LogActions: Leave
nataaaip (Started IVM-1)
Feb 21 21:42:26 [24023] IVM-1 pengine: info: LogActions: Leave
nataccessip (Started IVM-1)
Feb 21 21:42:26 [24023] IVM-1 pengine: info: LogActions: Leave
netactagent:0 (Stopped)
Feb 21 21:42:26 [24023] IVM-1 pengine: info: LogActions: Leave
netactagent:1 (Stopped)
Feb 21 21:42:26 [24023] IVM-1 pengine: info: LogActions: Leave
natomsip_for_netact (Stopped)
Feb 21 21:42:26 [24023] IVM-1 pengine: notice: LogActions: Promote
RoutingManager:0 (Slave -> Master IVM-1)
Feb 21 21:42:26 [24023] IVM-1 pengine: info: LogActions: Leave
RoutingManager:1 (Stopped)
Feb 21 21:42:26 [24023] IVM-1 pengine: info: LogActions: Leave
natomsgwip (Started IVM-1)
Feb 21 21:42:26 [24023] IVM-1 pengine: info: LogActions: Leave
omsip (Started IVM-1)
Feb 21 21:42:26 [24023] IVM-1 pengine: info: LogActions: Leave
nataaagwip (Started IVM-1)
Feb 21 21:42:26 [24023] IVM-1 pengine: notice: LogActions: Start
aaaip (IVM-1)
Feb 21 21:42:26 [24023] IVM-1 pengine: notice: LogActions: Start
nataccessgwip (IVM-1)
Feb 21 21:42:26 [24023] IVM-1 pengine: notice: LogActions: Start
accessip (IVM-1)
Feb 21 21:42:26 [24023] IVM-1 pengine: notice: LogActions: Start
natcpcoregwip (IVM-1)
Feb 21 21:42:26 [24023] IVM-1 pengine: notice: LogActions: Start
cpcoreip (IVM-1)
Feb 21 21:42:26 [24023] IVM-1 pengine: notice:
process_pe_message: Calculated
transition 1, saving inputs in /usr/var/lib/pacemaker/
pengine/pe-input-107.bz2
Feb 21 21:42:26 [24024] IVM-1 crmd: info: do_state_transition: State
transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS
cause=C_IPC_MESSAGE origin=handle_response
Feb 21 21:42:26 [24024] IVM-1 crmd: info: do_te_invoke: Processing
graph 1 (ref=pe_calc-dc-1519229546-70) derived from /usr/var/lib/pacemaker/
pengine/pe-input-107.bz2
Feb 21 21:42:26 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
monitor operation stateful_apache2_monitor_30000 locally on IVM-1 | action
15
Feb 21 21:42:26 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=15:1:8:e27e11f8-fc4e-493e-a4f2-80a02cb1f079 op=stateful_apache2_monitor_
30000
Feb 21 21:42:26 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
monitor operation stateful_consul_monitor_5000 locally on IVM-1 | action 54
Feb 21 21:42:26 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=54:1:8:e27e11f8-fc4e-493e-a4f2-80a02cb1f079 op=stateful_consul_monitor_
5000
Feb 21 21:42:26 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
monitor operation alarmSnmpd_monitor_20000 locally on IVM-1 | action 110
Feb 21 21:42:26 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=110:1:8:e27e11f8-fc4e-493e-a4f2-80a02cb1f079 op=alarmSnmpd_monitor_20000
Feb 21 21:42:26 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
monitor operation haproxy_monitor_20000 locally on IVM-1 | action 141
Feb 21 21:42:26 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=141:1:8:e27e11f8-fc4e-493e-a4f2-80a02cb1f079 op=haproxy_monitor_20000
Feb 21 21:42:26 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
monitor operation nataaagwip_monitor_10000 locally on IVM-1 | action 228
Feb 21 21:42:26 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=228:1:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079 op=nataaagwip_monitor_10000
Feb 21 21:42:26 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
start operation aaaip_start_0 locally on IVM-1 | action 229
Feb 21 21:42:26 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=229:1:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079 op=aaaip_start_0
Feb 21 21:42:26 [24021] IVM-1 lrmd: info: log_execute: executing
- rsc:aaaip action:start call_id:162
Feb 21 21:42:26 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
60 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:42:26 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
notify operation stateful_wildfly_pre_notify_demote_0 locally on IVM-1 |
action 289
Feb 21 21:42:26 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=289:1:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079
op=stateful_wildfly_notify_0
Feb 21 21:42:26 [24021] IVM-1 lrmd: info: log_execute: executing
- rsc:stateful_wildfly action:notify call_id:163
Feb 21 21:42:26 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
notify operation RoutingManager_pre_notify_promote_0 locally on IVM-1 |
action 311
Feb 21 21:42:26 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=311:1:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079 op=RoutingManager_notify_0
Feb 21 21:42:26 [24021] IVM-1 lrmd: info: log_execute: executing
- rsc:RoutingManager action:notify call_id:164
Feb 21 21:42:26 [24021] IVM-1 lrmd: info: log_finished: finished
- rsc:RoutingManager action:notify call_id:164 pid:13461 exit-code:0
exec-time:21ms queue-time:0ms
Feb 21 21:42:26 [24024] IVM-1 crmd: info: match_graph_event: Action
RoutingManager_notify_0 (311) confirmed on IVM-1 (rc=0)
Feb 21 21:42:26 [24024] IVM-1 crmd: notice: process_lrm_event: Result
of notify operation for RoutingManager on IVM-1: 0 (ok) | call=164
key=RoutingManager_notify_0 confirmed=true cib-update=0
Feb 21 21:42:26 [24024] IVM-1 crmd: info: process_lrm_event: Result
of monitor operation for haproxy on IVM-1: 8 (master) | call=160
key=haproxy_monitor_20000 confirmed=false cib-update=97
Feb 21 21:42:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/crmd/97)
Feb 21 21:42:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.43 2
Feb 21 21:42:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.44 (null)
Feb 21 21:42:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=44
Feb 21 21:42:26 [24019] IVM-1 cib: info: cib_perform_op: ++
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='haproxy']:
<lrm_rsc_op id="haproxy_monitor_20000" operation_key="haproxy_monitor_20000"
operation="monitor" crm-debug-origin="do_update_resource"
crm_feature_set="3.0.11"
transition-key="141:1:8:e27e11f8-fc4e-493e-a4f2-80a02cb1f079"
transition-magic="0:8;141:1:8:e27e11f8-fc4e-493e-a4f2-80a02cb1f079"
on_node="IVM-1" call-id="160" rc-code="8" op-status="0" inter
Feb 21 21:42:26 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/crmd/97,
version=0.108.44)
Feb 21 21:42:26 [24024] IVM-1 crmd: info: match_graph_event: Action
haproxy_monitor_20000 (141) confirmed on IVM-1 (rc=8)
Feb 21 21:42:26 [24021] IVM-1 lrmd: info: log_finished: finished
- rsc:stateful_wildfly action:notify call_id:163 pid:13460 exit-code:0
exec-time:72ms queue-time:0ms
Feb 21 21:42:26 [24024] IVM-1 crmd: info: match_graph_event: Action
stateful_wildfly_notify_0 (289) confirmed on IVM-1 (rc=0)
Feb 21 21:42:26 [24024] IVM-1 crmd: notice: process_lrm_event: Result
of notify operation for stateful_wildfly on IVM-1: 0 (ok) | call=163
key=stateful_wildfly_notify_0 confirmed=true cib-update=0
Feb 21 21:42:26 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
demote operation stateful_wildfly_demote_0 locally on IVM-1 | action 79
Feb 21 21:42:26 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=79:1:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079 op=stateful_wildfly_demote_0
Feb 21 21:42:26 [24021] IVM-1 lrmd: info: log_execute: executing
- rsc:stateful_wildfly action:demote call_id:165
Feb 21 21:42:26 [24024] IVM-1 crmd: info: process_lrm_event: Result
of monitor operation for nataaagwip on IVM-1: 0 (ok) | call=161
key=nataaagwip_monitor_10000 confirmed=false cib-update=98
Feb 21 21:42:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/crmd/98)
Feb 21 21:42:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.44 2
Feb 21 21:42:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.45 (null)
Feb 21 21:42:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=45
Feb 21 21:42:26 [24019] IVM-1 cib: info: cib_perform_op: ++
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/
lrm_resource[@id='nataaagwip']: <lrm_rsc_op id="nataaagwip_monitor_10000"
operation_key="nataaagwip_monitor_10000" operation="monitor"
crm-debug-origin="do_update_resource" crm_feature_set="3.0.11"
transition-key="228:1:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079"
transition-magic="0:0;228:1:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079"
on_node="IVM-1" call-id="161" rc-code="0" op-status=
Feb 21 21:42:26 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/crmd/98,
version=0.108.45)
Feb 21 21:42:26 [24024] IVM-1 crmd: info: match_graph_event: Action
nataaagwip_monitor_10000 (228) confirmed on IVM-1 (rc=0)
Feb 21 21:42:26 [24024] IVM-1 crmd: info: process_lrm_event: Result
of monitor operation for alarmSnmpd on IVM-1: 8 (master) | call=159
key=alarmSnmpd_monitor_20000 confirmed=false cib-update=99
Feb 21 21:42:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/crmd/99)
Feb 21 21:42:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.45 2
Feb 21 21:42:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.46 (null)
Feb 21 21:42:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=46
Feb 21 21:42:26 [24019] IVM-1 cib: info: cib_perform_op: ++
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/
lrm_resource[@id='alarmSnmpd']: <lrm_rsc_op id="alarmSnmpd_monitor_20000"
operation_key="alarmSnmpd_monitor_20000" operation="monitor"
crm-debug-origin="do_update_resource" crm_feature_set="3.0.11"
transition-key="110:1:8:e27e11f8-fc4e-493e-a4f2-80a02cb1f079"
transition-magic="0:8;110:1:8:e27e11f8-fc4e-493e-a4f2-80a02cb1f079"
on_node="IVM-1" call-id="159" rc-code="8" op-status=
Feb 21 21:42:26 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/crmd/99,
version=0.108.46)
Feb 21 21:42:26 [24024] IVM-1 crmd: info: match_graph_event: Action
alarmSnmpd_monitor_20000 (110) confirmed on IVM-1 (rc=8)
Feb 21 21:42:26 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
61 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
IPaddr2(aaaip)[13459]: 2018/02/21_21:42:26 INFO: Adding inet address
10.53.201.176/27 with broadcast address + to device aaa
Feb 21 21:42:26 [24024] IVM-1 crmd: info: process_lrm_event: Result
of monitor operation for stateful_consul on IVM-1: 8 (master) | call=158
key=stateful_consul_monitor_5000 confirmed=false cib-update=100
Feb 21 21:42:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/crmd/100)
Feb 21 21:42:26 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
62 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
IPaddr2(aaaip)[13459]: 2018/02/21_21:42:26 INFO: Bringing device aaa up
Feb 21 21:42:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.46 2
Feb 21 21:42:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.47 (null)
Feb 21 21:42:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=47
Feb 21 21:42:26 [24019] IVM-1 cib: info: cib_perform_op: ++
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/
lrm_resource[@id='stateful_consul']: <lrm_rsc_op
id="stateful_consul_monitor_5000" operation_key="stateful_consul_monitor_5000"
operation="monitor" crm-debug-origin="do_update_resource"
crm_feature_set="3.0.11"
transition-key="54:1:8:e27e11f8-fc4e-493e-a4f2-80a02cb1f079"
transition-magic="0:8;54:1:8:e27e11f8-fc4e-493e-a4f2-80a02cb1f079"
on_node="IVM-1" call-id="158" rc-code="8"
Feb 21 21:42:26 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/crmd/100,
version=0.108.47)
Feb 21 21:42:26 [24024] IVM-1 crmd: info: match_graph_event: Action
stateful_consul_monitor_5000 (54) confirmed on IVM-1 (rc=8)
Feb 21 21:42:26 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
63 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
IPaddr2(aaaip)[13459]: 2018/02/21_21:42:26 INFO:
/usr/libexec/heartbeat/send_arp -i 200 -r 5 -p
/usr/var/run/resource-agents/send_arp-10.53.201.176
aaa 10.53.201.176 auto not_used not_used
Feb 21 21:42:26 [24021] IVM-1 lrmd: info: log_finished: finished
- rsc:aaaip action:start call_id:162 pid:13459 exit-code:0 exec-time:194ms
queue-time:0ms
Feb 21 21:42:26 [24024] IVM-1 crmd: info: action_synced_wait: Managed
IPaddr2_meta-data_0 process 13896 exited with rc=0
Feb 21 21:42:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/crmd/101)
Feb 21 21:42:26 [24024] IVM-1 crmd: notice: process_lrm_event: Result
of start operation for aaaip on IVM-1: 0 (ok) | call=162 key=aaaip_start_0
confirmed=true cib-update=101
Feb 21 21:42:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.47 2
Feb 21 21:42:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.48 (null)
Feb 21 21:42:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=48
Feb 21 21:42:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_
resources/lrm_resource[@id='aaaip']/lrm_rsc_op[@id='aaaip_last_0']:
@operation_key=aaaip_start_0, @operation=start,
@crm-debug-origin=do_update_resource, @transition-key=229:1:0:
e27e11f8-fc4e-493e-a4f2-80a02cb1f079, @transition-magic=0:0;229:1:0:
e27e11f8-fc4e-493e-a4f2-80a02cb1f079, @call-id=162, @rc-code=0,
@last-run=1519229546, @last-rc-change=1519229546, @exec-time=194
Feb 21 21:42:26 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
64 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:42:26 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
65 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:42:26 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
66 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:42:26 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/crmd/101,
version=0.108.48)
Feb 21 21:42:26 [24024] IVM-1 crmd: info: match_graph_event: Action
aaaip_start_0 (229) confirmed on IVM-1 (rc=0)
Feb 21 21:42:26 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
monitor operation aaaip_monitor_10000 locally on IVM-1 | action 230
Feb 21 21:42:26 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=230:1:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079 op=aaaip_monitor_10000
Feb 21 21:42:26 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
start operation nataccessgwip_start_0 locally on IVM-1 | action 231
Feb 21 21:42:26 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=231:1:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079 op=nataccessgwip_start_0
Feb 21 21:42:26 [24021] IVM-1 lrmd: info: log_execute: executing
- rsc:nataccessgwip action:start call_id:167
Feb 21 21:42:26 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
67 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:42:26 [24024] IVM-1 crmd: info: process_lrm_event: Result
of monitor operation for aaaip on IVM-1: 0 (ok) | call=166
key=aaaip_monitor_10000 confirmed=false cib-update=102
Feb 21 21:42:26 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/crmd/102)
Feb 21 21:42:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.48 2
Feb 21 21:42:26 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.49 (null)
Feb 21 21:42:26 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=49
Feb 21 21:42:26 [24019] IVM-1 cib: info: cib_perform_op: ++
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='aaaip']:
<lrm_rsc_op id="aaaip_monitor_10000" operation_key="aaaip_monitor_10000"
operation="monitor" crm-debug-origin="do_update_resource"
crm_feature_set="3.0.11"
transition-key="230:1:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079"
transition-magic="0:0;230:1:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079"
on_node="IVM-1" call-id="166" rc-code="0" op-status="0" interval="1
Feb 21 21:42:26 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/crmd/102,
version=0.108.49)
Feb 21 21:42:26 [24024] IVM-1 crmd: info: match_graph_event: Action
aaaip_monitor_10000 (230) confirmed on IVM-1 (rc=0)
Feb 21 21:42:27 [24021] IVM-1 lrmd: info: log_finished: finished
- rsc:stateful_wildfly action:demote call_id:165 pid:13656 exit-code:0
exec-time:243ms queue-time:0ms
Feb 21 21:42:27 [24024] IVM-1 crmd: notice: process_lrm_event: Result
of demote operation for stateful_wildfly on IVM-1: 0 (ok) | call=165
key=stateful_wildfly_demote_0 confirmed=true cib-update=103
Feb 21 21:42:27 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/crmd/103)
Feb 21 21:42:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.49 2
Feb 21 21:42:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.50 (null)
Feb 21 21:42:27 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=50
Feb 21 21:42:27 [24019] IVM-1 cib: info: cib_perform_op: +
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_
resources/lrm_resource[@id='stateful_wildfly']/lrm_rsc_op[
@id='stateful_wildfly_last_0']: @operation_key=stateful_wildfly_demote_0,
@operation=demote, @transition-key=79:1:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079,
@transition-magic=0:0;79:1:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079,
@call-id=165, @rc-code=0, @op-status=0, @last-run=1519229546,
@last-rc-change=1519229546, @exec-time=243
Feb 21 21:42:27 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/crmd/103,
version=0.108.50)
Feb 21 21:42:27 [24024] IVM-1 crmd: info: match_graph_event: Action
stateful_wildfly_demote_0 (79) confirmed on IVM-1 (rc=0)
Feb 21 21:42:27 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
notify operation stateful_wildfly_post_notify_demote_0 locally on IVM-1 |
action 290
Feb 21 21:42:27 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=290:1:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079
op=stateful_wildfly_notify_0
Feb 21 21:42:27 [24021] IVM-1 lrmd: info: log_execute: executing
- rsc:stateful_wildfly action:notify call_id:168
IPaddr2(nataccessgwip)[13966]: 2018/02/21_21:42:27 INFO: Adding inet
address 192.168.3.1/24 with broadcast address + to device nataccess
IPaddr2(nataccessgwip)[13966]: 2018/02/21_21:42:27 INFO: Bringing device
nataccess up
IPaddr2(nataccessgwip)[13966]: 2018/02/21_21:42:27 INFO:
/usr/libexec/heartbeat/send_arp -i 200 -r 5 -p
/usr/var/run/resource-agents/send_arp-192.168.3.1
nataccess 192.168.3.1 auto not_used not_used
Feb 21 21:42:27 [24021] IVM-1 lrmd: info: log_finished: finished
- rsc:nataccessgwip action:start call_id:167 pid:13966 exit-code:0
exec-time:146ms queue-time:0ms
Feb 21 21:42:27 [24021] IVM-1 lrmd: info: log_finished: finished
- rsc:stateful_wildfly action:notify call_id:168 pid:14184 exit-code:0
exec-time:48ms queue-time:0ms
Feb 21 21:42:27 [24024] IVM-1 crmd: info: action_synced_wait: Managed
IPaddr2_meta-data_0 process 14272 exited with rc=0
Feb 21 21:42:27 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/crmd/104)
Feb 21 21:42:27 [24024] IVM-1 crmd: notice: process_lrm_event: Result
of start operation for nataccessgwip on IVM-1: 0 (ok) | call=167
key=nataccessgwip_start_0 confirmed=true cib-update=104
Feb 21 21:42:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.50 2
Feb 21 21:42:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.51 (null)
Feb 21 21:42:27 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=51
Feb 21 21:42:27 [24019] IVM-1 cib: info: cib_perform_op: +
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_
resources/lrm_resource[@id='nataccessgwip']/lrm_rsc_op[@id='nataccessgwip_last_0']:
@operation_key=nataccessgwip_start_0, @operation=start,
@crm-debug-origin=do_update_resource, @transition-key=231:1:0:
e27e11f8-fc4e-493e-a4f2-80a02cb1f079, @transition-magic=0:0;231:1:0:
e27e11f8-fc4e-493e-a4f2-80a02cb1f079, @call-id=167, @rc-code=0,
@last-run=1519229546, @last-rc-change=1519229546, @exec-
Feb 21 21:42:27 [24024] IVM-1 crmd: info: match_graph_event: Action
stateful_wildfly_notify_0 (290) confirmed on IVM-1 (rc=0)
Feb 21 21:42:27 [24024] IVM-1 crmd: notice: process_lrm_event: Result
of notify operation for stateful_wildfly on IVM-1: 0 (ok) | call=168
key=stateful_wildfly_notify_0 confirmed=true cib-update=0
Feb 21 21:42:27 [24024] IVM-1 crmd: info: match_graph_event: Action
nataccessgwip_start_0 (231) confirmed on IVM-1 (rc=0)
Feb 21 21:42:27 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
monitor operation nataccessgwip_monitor_10000 locally on IVM-1 | action 232
Feb 21 21:42:27 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=232:1:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079
op=nataccessgwip_monitor_10000
Feb 21 21:42:27 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/crmd/104,
version=0.108.51)
Feb 21 21:42:27 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
start operation accessip_start_0 locally on IVM-1 | action 233
Feb 21 21:42:27 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=233:1:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079 op=accessip_start_0
Feb 21 21:42:27 [24021] IVM-1 lrmd: info: log_execute: executing
- rsc:accessip action:start call_id:170
Feb 21 21:42:27 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
notify operation stateful_wildfly_pre_notify_stop_0 locally on IVM-1 |
action 288
Feb 21 21:42:27 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=288:1:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079
op=stateful_wildfly_notify_0
Feb 21 21:42:27 [24021] IVM-1 lrmd: info: log_execute: executing
- rsc:stateful_wildfly action:notify call_id:171
Feb 21 21:42:27 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
68 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:42:27 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
69 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:42:27 [24021] IVM-1 lrmd: info: log_finished: finished
- rsc:stateful_wildfly action:notify call_id:171 pid:14291 exit-code:0
exec-time:69ms queue-time:0ms
Feb 21 21:42:27 [24024] IVM-1 crmd: info: match_graph_event: Action
stateful_wildfly_notify_0 (288) confirmed on IVM-1 (rc=0)
Feb 21 21:42:27 [24024] IVM-1 crmd: notice: process_lrm_event: Result
of notify operation for stateful_wildfly on IVM-1: 0 (ok) | call=171
key=stateful_wildfly_notify_0 confirmed=true cib-update=0
Feb 21 21:42:27 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
stop operation stateful_wildfly_stop_0 locally on IVM-1 | action 3
Feb 21 21:42:27 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=3:1:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079 op=stateful_wildfly_stop_0
Feb 21 21:42:27 [24021] IVM-1 lrmd: info: log_execute: executing
- rsc:stateful_wildfly action:stop call_id:172
Feb 21 21:42:27 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
70 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:42:27 [24024] IVM-1 crmd: info: process_lrm_event: Result
of monitor operation for nataccessgwip on IVM-1: 0 (ok) | call=169
key=nataccessgwip_monitor_10000 confirmed=false cib-update=105
Feb 21 21:42:27 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/crmd/105)
Feb 21 21:42:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.51 2
Feb 21 21:42:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.52 (null)
Feb 21 21:42:27 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=52
Feb 21 21:42:27 [24019] IVM-1 cib: info: cib_perform_op: ++
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/
lrm_resource[@id='nataccessgwip']: <lrm_rsc_op
id="nataccessgwip_monitor_10000"
operation_key="nataccessgwip_monitor_10000" operation="monitor"
crm-debug-origin="do_update_resource" crm_feature_set="3.0.11"
transition-key="232:1:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079"
transition-magic="0:0;232:1:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079"
on_node="IVM-1" call-id="169" rc-code="0" o
Feb 21 21:42:27 [24024] IVM-1 crmd: info: match_graph_event: Action
nataccessgwip_monitor_10000 (232) confirmed on IVM-1 (rc=0)
Feb 21 21:42:27 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
71 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:42:27 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/crmd/105,
version=0.108.52)
IPaddr2(accessip)[14290]: 2018/02/21_21:42:27 INFO: Adding inet address
10.53.201.144/27 with broadcast address + to device access
IPaddr2(accessip)[14290]: 2018/02/21_21:42:27 INFO: Bringing device access
up
IPaddr2(accessip)[14290]: 2018/02/21_21:42:27 INFO:
/usr/libexec/heartbeat/send_arp -i 200 -r 5 -p
/usr/var/run/resource-agents/send_arp-10.53.201.144
access 10.53.201.144 auto not_used not_used
Feb 21 21:42:27 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
72 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:42:27 [24021] IVM-1 lrmd: info: log_finished: finished
- rsc:accessip action:start call_id:170 pid:14290 exit-code:0
exec-time:153ms queue-time:0ms
Feb 21 21:42:27 [24024] IVM-1 crmd: info: action_synced_wait: Managed
IPaddr2_meta-data_0 process 14588 exited with rc=0
Feb 21 21:42:27 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/crmd/106)
Feb 21 21:42:27 [24024] IVM-1 crmd: notice: process_lrm_event: Result
of start operation for accessip on IVM-1: 0 (ok) | call=170
key=accessip_start_0 confirmed=true cib-update=106
Feb 21 21:42:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.52 2
Feb 21 21:42:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.53 (null)
Feb 21 21:42:27 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=53
Feb 21 21:42:27 [24019] IVM-1 cib: info: cib_perform_op: +
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_
resources/lrm_resource[@id='accessip']/lrm_rsc_op[@id='accessip_last_0']:
@operation_key=accessip_start_0, @operation=start,
@crm-debug-origin=do_update_resource, @transition-key=233:1:0:
e27e11f8-fc4e-493e-a4f2-80a02cb1f079, @transition-magic=0:0;233:1:0:
e27e11f8-fc4e-493e-a4f2-80a02cb1f079, @call-id=170, @rc-code=0,
@last-run=1519229547, @last-rc-change=1519229547, @exec-time=153
Feb 21 21:42:27 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/crmd/106,
version=0.108.53)
Feb 21 21:42:27 [24024] IVM-1 crmd: info: match_graph_event: Action
accessip_start_0 (233) confirmed on IVM-1 (rc=0)
Feb 21 21:42:27 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
monitor operation accessip_monitor_10000 locally on IVM-1 | action 234
Feb 21 21:42:27 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=234:1:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079 op=accessip_monitor_10000
Feb 21 21:42:27 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
start operation natcpcoregwip_start_0 locally on IVM-1 | action 235
Feb 21 21:42:27 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=235:1:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079 op=natcpcoregwip_start_0
Feb 21 21:42:27 [24021] IVM-1 lrmd: info: log_execute: executing
- rsc:natcpcoregwip action:start call_id:174
Feb 21 21:42:27 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
73 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:42:27 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
74 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:42:27 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_apache2_monitor_30000:13430:stderr [ chmod: cannot access
'/etc/.certs/operator/key.pem': No such file or directory ]
Feb 21 21:42:27 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_apache2_monitor_30000:13430:stderr [ chmod: cannot access
'/etc/.certs/operator/certificate.pem': No such file or directory ]
Feb 21 21:42:27 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_apache2_monitor_30000:13430:stderr [ chmod: cannot access
'/etc/.certs/operator/intermediatetrustchain.pem': No such file or
directory ]
Feb 21 21:42:27 [24024] IVM-1 crmd: info: process_lrm_event: Result
of monitor operation for stateful_apache2 on IVM-1: 8 (master) | call=157
key=stateful_apache2_monitor_30000 confirmed=false cib-update=107
Feb 21 21:42:27 [24024] IVM-1 crmd: notice: process_lrm_event:
IVM-1-stateful_apache2_monitor_30000:157 [ chmod: cannot access
'/etc/.certs/operator/key.pem': No such file or directory\nchmod: cannot
access '/etc/.certs/operator/certificate.pem': No such file or
directory\nchmod: cannot access
'/etc/.certs/operator/intermediatetrustchain.pem':
No such file or directory\n ]
Feb 21 21:42:27 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/crmd/107)
Feb 21 21:42:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.53 2
Feb 21 21:42:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.54 (null)
Feb 21 21:42:27 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=54
Feb 21 21:42:27 [24019] IVM-1 cib: info: cib_perform_op: ++
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/
lrm_resource[@id='stateful_apache2']: <lrm_rsc_op
id="stateful_apache2_monitor_30000"
operation_key="stateful_apache2_monitor_30000"
operation="monitor" crm-debug-origin="do_update_resource"
crm_feature_set="3.0.11"
transition-key="15:1:8:e27e11f8-fc4e-493e-a4f2-80a02cb1f079"
transition-magic="0:8;15:1:8:e27e11f8-fc4e-493e-a4f2-80a02cb1f079"
on_node="IVM-1" call-id="157" rc-cod
Feb 21 21:42:27 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/crmd/107,
version=0.108.54)
Feb 21 21:42:27 [24024] IVM-1 crmd: info: match_graph_event: Action
stateful_apache2_monitor_30000 (15) confirmed on IVM-1 (rc=8)
Feb 21 21:42:27 [24024] IVM-1 crmd: info: process_lrm_event: Result
of monitor operation for accessip on IVM-1: 0 (ok) | call=173
key=accessip_monitor_10000 confirmed=false cib-update=108
Feb 21 21:42:27 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/crmd/108)
Feb 21 21:42:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.54 2
Feb 21 21:42:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.55 (null)
Feb 21 21:42:27 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=55
Feb 21 21:42:27 [24019] IVM-1 cib: info: cib_perform_op: ++
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='accessip']:
<lrm_rsc_op id="accessip_monitor_10000" operation_key="accessip_monitor_10000"
operation="monitor" crm-debug-origin="do_update_resource"
crm_feature_set="3.0.11"
transition-key="234:1:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079"
transition-magic="0:0;234:1:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079"
on_node="IVM-1" call-id="173" rc-code="0" op-status="0" in
Feb 21 21:42:27 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/crmd/108,
version=0.108.55)
Feb 21 21:42:27 [24024] IVM-1 crmd: info: match_graph_event: Action
accessip_monitor_10000 (234) confirmed on IVM-1 (rc=0)
Feb 21 21:42:27 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_stop_0:14415:stderr [ kill: no process ID specified ]
Feb 21 21:42:27 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_stop_0:14415:stderr [ Try 'kill --help' for more
information. ]
Feb 21 21:42:27 [24021] IVM-1 lrmd: info: log_finished: finished
- rsc:stateful_wildfly action:stop call_id:172 pid:14415 exit-code:0
exec-time:283ms queue-time:0ms
Feb 21 21:42:27 [24024] IVM-1 crmd: notice: process_lrm_event: Result
of stop operation for stateful_wildfly on IVM-1: 0 (ok) | call=172
key=stateful_wildfly_stop_0 confirmed=true cib-update=109
Feb 21 21:42:27 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/crmd/109)
Feb 21 21:42:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.55 2
Feb 21 21:42:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.56 (null)
Feb 21 21:42:27 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=56
IPaddr2(natcpcoregwip)[14676]: 2018/02/21_21:42:27 INFO: Adding inet
address 192.168.4.1/24 with broadcast address + to device natcpcore
Feb 21 21:42:27 [24019] IVM-1 cib: info: cib_perform_op: +
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_
resources/lrm_resource[@id='stateful_wildfly']/lrm_rsc_op[
@id='stateful_wildfly_last_0']: @operation_key=stateful_wildfly_stop_0,
@operation=stop, @transition-key=3:1:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079,
@transition-magic=0:0;3:1:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079,
@call-id=172, @last-run=1519229547, @last-rc-change=1519229547,
@exec-time=283
Feb 21 21:42:27 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/crmd/109,
version=0.108.56)
Feb 21 21:42:27 [24024] IVM-1 crmd: info: match_graph_event: Action
stateful_wildfly_stop_0 (3) confirmed on IVM-1 (rc=0)
Feb 21 21:42:27 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
start operation stateful_wildfly_start_0 locally on IVM-1 | action 80
Feb 21 21:42:27 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=80:1:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079 op=stateful_wildfly_start_0
Feb 21 21:42:27 [24021] IVM-1 lrmd: info: log_execute: executing
- rsc:stateful_wildfly action:start call_id:175
Feb 21 21:42:27 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
75 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
IPaddr2(natcpcoregwip)[14676]: 2018/02/21_21:42:27 INFO: Bringing device
natcpcore up
Feb 21 21:42:27 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
76 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
IPaddr2(natcpcoregwip)[14676]: 2018/02/21_21:42:27 INFO:
/usr/libexec/heartbeat/send_arp -i 200 -r 5 -p
/usr/var/run/resource-agents/send_arp-192.168.4.1
natcpcore 192.168.4.1 auto not_used not_used
Feb 21 21:42:27 [24021] IVM-1 lrmd: info: log_finished: finished
- rsc:natcpcoregwip action:start call_id:174 pid:14676 exit-code:0
exec-time:276ms queue-time:0ms
Feb 21 21:42:27 [24024] IVM-1 crmd: info: action_synced_wait: Managed
IPaddr2_meta-data_0 process 15144 exited with rc=0
Feb 21 21:42:27 [24024] IVM-1 crmd: notice: process_lrm_event: Result
of start operation for natcpcoregwip on IVM-1: 0 (ok) | call=174
key=natcpcoregwip_start_0 confirmed=true cib-update=110
Feb 21 21:42:27 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/crmd/110)
Feb 21 21:42:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.56 2
Feb 21 21:42:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.57 (null)
Feb 21 21:42:27 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=57
Feb 21 21:42:27 [24019] IVM-1 cib: info: cib_perform_op: +
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_
resources/lrm_resource[@id='natcpcoregwip']/lrm_rsc_op[@id='natcpcoregwip_last_0']:
@operation_key=natcpcoregwip_start_0, @operation=start,
@crm-debug-origin=do_update_resource, @transition-key=235:1:0:
e27e11f8-fc4e-493e-a4f2-80a02cb1f079, @transition-magic=0:0;235:1:0:
e27e11f8-fc4e-493e-a4f2-80a02cb1f079, @call-id=174, @rc-code=0,
@last-run=1519229547, @last-rc-change=1519229547, @exec-
Feb 21 21:42:27 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/crmd/110,
version=0.108.57)
Feb 21 21:42:27 [24024] IVM-1 crmd: info: match_graph_event: Action
natcpcoregwip_start_0 (235) confirmed on IVM-1 (rc=0)
Feb 21 21:42:27 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
monitor operation natcpcoregwip_monitor_10000 locally on IVM-1 | action 236
Feb 21 21:42:27 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=236:1:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079
op=natcpcoregwip_monitor_10000
Feb 21 21:42:27 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
start operation cpcoreip_start_0 locally on IVM-1 | action 237
Feb 21 21:42:27 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=237:1:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079 op=cpcoreip_start_0
Feb 21 21:42:27 [24021] IVM-1 lrmd: info: log_execute: executing
- rsc:cpcoreip action:start call_id:177
Feb 21 21:42:27 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
77 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:42:27 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
78 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:42:27 [24024] IVM-1 crmd: info: process_lrm_event: Result
of monitor operation for natcpcoregwip on IVM-1: 0 (ok) | call=176
key=natcpcoregwip_monitor_10000 confirmed=false cib-update=111
Feb 21 21:42:27 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/crmd/111)
Feb 21 21:42:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.57 2
Feb 21 21:42:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.58 (null)
Feb 21 21:42:27 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=58
Feb 21 21:42:27 [24019] IVM-1 cib: info: cib_perform_op: ++
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/
lrm_resource[@id='natcpcoregwip']: <lrm_rsc_op
id="natcpcoregwip_monitor_10000"
operation_key="natcpcoregwip_monitor_10000" operation="monitor"
crm-debug-origin="do_update_resource" crm_feature_set="3.0.11"
transition-key="236:1:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079"
transition-magic="0:0;236:1:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079"
on_node="IVM-1" call-id="176" rc-code="0" o
Feb 21 21:42:27 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/crmd/111,
version=0.108.58)
Feb 21 21:42:27 [24024] IVM-1 crmd: info: match_graph_event: Action
natcpcoregwip_monitor_10000 (236) confirmed on IVM-1 (rc=0)
IPaddr2(cpcoreip)[15190]: 2018/02/21_21:42:27 INFO: Adding inet address
10.53.201.240/27 with broadcast address + to device cpcore
IPaddr2(cpcoreip)[15190]: 2018/02/21_21:42:27 INFO: Bringing device cpcore
up
IPaddr2(cpcoreip)[15190]: 2018/02/21_21:42:27 INFO:
/usr/libexec/heartbeat/send_arp -i 200 -r 5 -p
/usr/var/run/resource-agents/send_arp-10.53.201.240
cpcore 10.53.201.240 auto not_used not_used
Feb 21 21:42:27 [24021] IVM-1 lrmd: info: log_finished: finished
- rsc:cpcoreip action:start call_id:177 pid:15190 exit-code:0
exec-time:115ms queue-time:0ms
Feb 21 21:42:27 [24024] IVM-1 crmd: info: action_synced_wait: Managed
IPaddr2_meta-data_0 process 15473 exited with rc=0
Feb 21 21:42:27 [24024] IVM-1 crmd: notice: process_lrm_event: Result
of start operation for cpcoreip on IVM-1: 0 (ok) | call=177
key=cpcoreip_start_0 confirmed=true cib-update=112
Feb 21 21:42:27 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/crmd/112)
Feb 21 21:42:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.58 2
Feb 21 21:42:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.59 (null)
Feb 21 21:42:27 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=59
Feb 21 21:42:27 [24019] IVM-1 cib: info: cib_perform_op: +
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_
resources/lrm_resource[@id='cpcoreip']/lrm_rsc_op[@id='cpcoreip_last_0']:
@operation_key=cpcoreip_start_0, @operation=start,
@crm-debug-origin=do_update_resource, @transition-key=237:1:0:
e27e11f8-fc4e-493e-a4f2-80a02cb1f079, @transition-magic=0:0;237:1:0:
e27e11f8-fc4e-493e-a4f2-80a02cb1f079, @call-id=177, @rc-code=0,
@last-run=1519229547, @last-rc-change=1519229547, @exec-time=115
Feb 21 21:42:27 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/crmd/112,
version=0.108.59)
Feb 21 21:42:27 [24024] IVM-1 crmd: info: match_graph_event: Action
cpcoreip_start_0 (237) confirmed on IVM-1 (rc=0)
Feb 21 21:42:27 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
monitor operation cpcoreip_monitor_10000 locally on IVM-1 | action 238
Feb 21 21:42:27 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=238:1:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079 op=cpcoreip_monitor_10000
Feb 21 21:42:27 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
promote operation RoutingManager_promote_0 locally on IVM-1 | action 196
Feb 21 21:42:27 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=196:1:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079 op=RoutingManager_promote_0
Feb 21 21:42:27 [24021] IVM-1 lrmd: info: log_execute: executing
- rsc:RoutingManager action:promote call_id:179
Feb 21 21:42:27 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
79 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:42:27 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
80 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:42:27 [24024] IVM-1 crmd: info: process_lrm_event: Result
of monitor operation for cpcoreip on IVM-1: 0 (ok) | call=178
key=cpcoreip_monitor_10000 confirmed=false cib-update=113
Feb 21 21:42:27 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/crmd/113)
Feb 21 21:42:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.59 2
Feb 21 21:42:27 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.60 (null)
Feb 21 21:42:27 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=60
Feb 21 21:42:27 [24019] IVM-1 cib: info: cib_perform_op: ++
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='cpcoreip']:
<lrm_rsc_op id="cpcoreip_monitor_10000" operation_key="cpcoreip_monitor_10000"
operation="monitor" crm-debug-origin="do_update_resource"
crm_feature_set="3.0.11"
transition-key="238:1:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079"
transition-magic="0:0;238:1:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079"
on_node="IVM-1" call-id="178" rc-code="0" op-status="0" in
Feb 21 21:42:27 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/crmd/113,
version=0.108.60)
Feb 21 21:42:27 [24024] IVM-1 crmd: info: match_graph_event: Action
cpcoreip_monitor_10000 (238) confirmed on IVM-1 (rc=0)
Feb 21 21:42:27 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
81 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:42:28 [24022] IVM-1 attrd: info: attrd_peer_update: Setting
master-RoutingManager[IVM-1]: 5 -> 10 from IVM-1
Feb 21 21:42:28 [24022] IVM-1 attrd: info: write_attribute: Sent
update 45 with 1 changes for master-RoutingManager, id=<n/a>, set=(null)
Feb 21 21:42:28 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/attrd/45)
Feb 21 21:42:28 [24021] IVM-1 lrmd: info: log_finished: finished
- rsc:RoutingManager action:promote call_id:179 pid:15494 exit-code:0
exec-time:498ms queue-time:0ms
Feb 21 21:42:28 [24024] IVM-1 crmd: notice: process_lrm_event: Result
of promote operation for RoutingManager on IVM-1: 0 (ok) | call=179
key=RoutingManager_promote_0 confirmed=true cib-update=114
Feb 21 21:42:28 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.60 2
Feb 21 21:42:28 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.61 (null)
Feb 21 21:42:28 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=61
Feb 21 21:42:28 [24019] IVM-1 cib: info: cib_perform_op: +
/cib/status/node_state[@id='2']/transient_attributes[@id='
2']/instance_attributes[@id='status-2']/nvpair[@id='status-2-master-RoutingManager']:
@value=10
Feb 21 21:42:28 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/attrd/45,
version=0.108.61)
Feb 21 21:42:28 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
45 for master-RoutingManager: OK (0)
Feb 21 21:42:28 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/crmd/114)
Feb 21 21:42:28 [24022] IVM-1 attrd: info: attrd_cib_callback: Update
45 for master-RoutingManager[IVM-1]=10: OK (0)
Feb 21 21:42:28 [24024] IVM-1 crmd: notice:
abort_transition_graph: Transition
aborted by status-2-master-RoutingManager doing modify
master-RoutingManager=10: Transient attribute change | cib=0.108.61
source=abort_unless_down:331 path=/cib/status/node_state[@
id='2']/transient_attributes[@id='2']/instance_attributes[@
id='status-2']/nvpair[@id='status-2-master-RoutingManager'] complete=false
Feb 21 21:42:28 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.61 2
Feb 21 21:42:28 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.62 (null)
Feb 21 21:42:28 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=62
Feb 21 21:42:28 [24019] IVM-1 cib: info: cib_perform_op: +
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_
resources/lrm_resource[@id='RoutingManager']/lrm_rsc_op[@id='RoutingManager_last_0']:
@operation_key=RoutingManager_promote_0, @operation=promote,
@crm-debug-origin=do_update_resource, @transition-key=196:1:0:
e27e11f8-fc4e-493e-a4f2-80a02cb1f079, @transition-magic=0:0;196:1:0:
e27e11f8-fc4e-493e-a4f2-80a02cb1f079, @call-id=179, @last-run=1519229547,
@last-rc-change=1519229547, @exec-time=
Feb 21 21:42:28 [24024] IVM-1 crmd: info: match_graph_event: Action
RoutingManager_promote_0 (196) confirmed on IVM-1 (rc=0)
Feb 21 21:42:28 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
notify operation RoutingManager_post_notify_promote_0 locally on IVM-1 |
action 312
Feb 21 21:42:28 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=312:1:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079 op=RoutingManager_notify_0
Feb 21 21:42:28 [24021] IVM-1 lrmd: info: log_execute: executing
- rsc:RoutingManager action:notify call_id:180
Feb 21 21:42:28 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/crmd/114,
version=0.108.62)
Feb 21 21:42:28 [24021] IVM-1 lrmd: info: log_finished: finished
- rsc:RoutingManager action:notify call_id:180 pid:15907 exit-code:0
exec-time:21ms queue-time:0ms
Feb 21 21:42:28 [24024] IVM-1 crmd: info: match_graph_event: Action
RoutingManager_notify_0 (312) confirmed on IVM-1 (rc=0)
Feb 21 21:42:28 [24024] IVM-1 crmd: notice: process_lrm_event: Result
of notify operation for RoutingManager on IVM-1: 0 (ok) | call=180
key=RoutingManager_notify_0 confirmed=true cib-update=0
Feb 21 21:42:28 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
82 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:42:28 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
83 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:42:33 [24019] IVM-1 cib: info: cib_process_ping: Reporting
our current digest to IVM-1: 326c7b46bb344a0023abf1edf7ea848f for 0.108.62
(0x558e28d66b70 0)
Feb 21 21:42:57 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_apache2_monitor_30000:20950:stderr [ chmod: cannot access
'/etc/.certs/operator/key.pem': No such file or directory ]
Feb 21 21:42:57 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_apache2_monitor_30000:20950:stderr [ chmod: cannot access
'/etc/.certs/operator/certificate.pem': No such file or directory ]
Feb 21 21:42:57 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_apache2_monitor_30000:20950:stderr [ chmod: cannot access
'/etc/.certs/operator/intermediatetrustchain.pem': No such file or
directory ]
Feb 21 21:42:58 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:21167:stderr [
/opt/vplat/modules/pacemaker/clusterMon.sh:
line 127: [: too many arguments ]
Feb 21 21:42:58 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:21167:stderr [ resource stateful_consul is NOT
running ]
Feb 21 21:42:58 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:21167:stderr [ resource RoutingManager is NOT
running ]
Feb 21 21:42:58 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:21167:stderr [ resource haproxy is NOT running ]
Feb 21 21:42:58 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:21167:stderr [
/opt/vplat/modules/pacemaker/clusterMon.sh:
line 127: [: too many arguments ]
Feb 21 21:42:58 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:21167:stderr [ resource stateful_wildfly is NOT
running ]
Feb 21 21:42:58 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:21167:stderr [ resource stateful_wildfly is NOT
running ]
Feb 21 21:42:58 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:21167:stderr [ resource stateful_apache2 is NOT
running ]
Feb 21 21:43:28 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_apache2_monitor_30000:26522:stderr [ chmod: cannot access
'/etc/.certs/operator/key.pem': No such file or directory ]
Feb 21 21:43:28 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_apache2_monitor_30000:26522:stderr [ chmod: cannot access
'/etc/.certs/operator/certificate.pem': No such file or directory ]
Feb 21 21:43:28 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_apache2_monitor_30000:26522:stderr [ chmod: cannot access
'/etc/.certs/operator/intermediatetrustchain.pem': No such file or
directory ]
Feb 21 21:43:58 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_apache2_monitor_30000:32309:stderr [ chmod: cannot access
'/etc/.certs/operator/key.pem': No such file or directory ]
Feb 21 21:43:58 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_apache2_monitor_30000:32309:stderr [ chmod: cannot access
'/etc/.certs/operator/certificate.pem': No such file or directory ]
Feb 21 21:43:58 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_apache2_monitor_30000:32309:stderr [ chmod: cannot access
'/etc/.certs/operator/intermediatetrustchain.pem': No such file or
directory ]
Feb 21 21:43:59 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:32286:stderr [
/opt/vplat/modules/pacemaker/clusterMon.sh:
line 127: [: too many arguments ]
Feb 21 21:43:59 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:32286:stderr [ resource stateful_consul is NOT
running ]
Feb 21 21:43:59 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:32286:stderr [ resource RoutingManager is NOT
running ]
Feb 21 21:43:59 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:32286:stderr [ resource haproxy is NOT running ]
Feb 21 21:43:59 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:32286:stderr [
/opt/vplat/modules/pacemaker/clusterMon.sh:
line 127: [: too many arguments ]
Feb 21 21:43:59 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:32286:stderr [ resource stateful_wildfly is NOT
running ]
Feb 21 21:43:59 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:32286:stderr [ resource stateful_wildfly is NOT
running ]
Feb 21 21:43:59 [24021] IVM-1 lrmd: notice: operation_finished:
ClusterMon_monitor_60000:32286:stderr [ resource stateful_apache2 is NOT
running ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ ln: failed to create symbolic link
'/usr/java/jre/jre1.8.0_152': File exists ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ unable to write 'random state' ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ Importing keystore GUI_CAPPORT.p12
to GUI_CAPPORT_STORE.jks... ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ Entry for alias 1 successfully
imported. ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ Import command completed: 1
entries successfully imported, 0 entries failed or cancelled ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ Warning: ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ The JKS keystore uses a proprietary
format. It is recommended to migrate to PKCS12 which is an industry
standard format using "keytool -importkeystore -srckeystore
GUI_CAPPORT_STORE.jks -destkeystore GUI_CAPPORT_STORE.jks -deststoretype
pkcs12". ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ Warning: ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ The JCEKS keystore uses a
proprietary format. It is recommended to migrate to PKCS12 which is an
industry standard format using "keytool -importkeystore -srckeystore
/opt/wildfly/cwlcvault/vault.keystore -destkeystore
/opt/wildfly/cwlcvault/vault.keystore -deststoretype pkcs12". ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ Feb 21, 2018 9:42:35 PM
org.picketbox.plugins.vault.PicketBoxSecurityVault init ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ INFO: PBOX00361: Default Security
Vault Implementation Initialized and Ready ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ awk: cmd. line:1: warning: escape
sequence `\/' treated as plain `/' ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ awk: cmd. line:1: warning: escape
sequence `\/' treated as plain `/' ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ awk: cmd. line:1: warning: escape
sequence `\/' treated as plain `/' ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ awk: cmd. line:1: warning: escape
sequence `\$' treated as plain `$' ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ awk: cmd. line:1: warning: escape
sequence `\/' treated as plain `/' ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ awk: cmd. line:1: warning: escape
sequence `\/' treated as plain `/' ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ awk: cmd. line:1: warning: escape
sequence `\/' treated as plain `/' ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ awk: cmd. line:1: warning: escape
sequence `\/' treated as plain `/' ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ awk: cmd. line:1: warning: escape
sequence `\/' treated as plain `/' ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ awk: cmd. line:1: warning: escape
sequence `\/' treated as plain `/' ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ awk: cmd. line:1: warning: escape
sequence `\/' treated as plain `/' ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ awk: cmd. line:1: warning: escape
sequence `\/' treated as plain `/' ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ awk: cmd. line:1: warning: escape
sequence `\/' treated as plain `/' ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ awk: cmd. line:1: warning: escape
sequence `\/' treated as plain `/' ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ awk: cmd. line:1: warning: escape
sequence `\/' treated as plain `/' ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ awk: cmd. line:1: warning: escape
sequence `\/' treated as plain `/' ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ awk: cmd. line:1: warning: escape
sequence `\/' treated as plain `/' ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ awk: cmd. line:1: warning: escape
sequence `\/' treated as plain `/' ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ awk: cmd. line:1: warning: escape
sequence `\$' treated as plain `$' ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ awk: cmd. line:1: warning: escape
sequence `\/' treated as plain `/' ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ awk: cmd. line:1: warning: escape
sequence `\/' treated as plain `/' ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ awk: cmd. line:1: warning: escape
sequence `\$' treated as plain `$' ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ awk: cmd. line:1: warning: escape
sequence `\/' treated as plain `/' ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ awk: cmd. line:1: warning: escape
sequence `\/' treated as plain `/' ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ awk: cmd. line:1: warning: escape
sequence `\$' treated as plain `$' ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ awk: cmd. line:1: warning: escape
sequence `\/' treated as plain `/' ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ awk: cmd. line:1: warning: escape
sequence `\$' treated as plain `$' ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ awk: cmd. line:1: warning: escape
sequence `\/' treated as plain `/' ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ awk: cmd. line:1: warning: escape
sequence `\/' treated as plain `/' ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ awk: cmd. line:1: warning: escape
sequence `\/' treated as plain `/' ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ awk: cmd. line:1: warning: escape
sequence `\/' treated as plain `/' ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ awk: cmd. line:1: warning: escape
sequence `\/' treated as plain `/' ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ awk: cmd. line:1: warning: escape
sequence `\/' treated as plain `/' ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ awk: cmd. line:1: warning: escape
sequence `\$' treated as plain `$' ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ awk: cmd. line:1: warning: escape
sequence `\/' treated as plain `/' ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ awk: cmd. line:1: warning: escape
sequence `\/' treated as plain `/' ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ awk: cmd. line:1: warning: escape
sequence `\/' treated as plain `/' ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ awk: cmd. line:1: warning: escape
sequence `\/' treated as plain `/' ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ awk: cmd. line:1: warning: escape
sequence `\/' treated as plain `/' ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ awk: cmd. line:1: warning: escape
sequence `\/' treated as plain `/' ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ awk: cmd. line:1: warning: escape
sequence `\/' treated as plain `/' ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ awk: cmd. line:1: warning: escape
sequence `\/' treated as plain `/' ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: notice: operation_finished:
stateful_wildfly_start_0:14963:stderr [ awk: cmd. line:1: warning: escape
sequence `\/' treated as plain `/' ]
Feb 21 21:44:01 [24021] IVM-1 lrmd: info: log_finished: finished
- rsc:stateful_wildfly action:start call_id:175 pid:14963 exit-code:0
exec-time:94019ms queue-time:0ms
Feb 21 21:44:01 [24024] IVM-1 crmd: info: action_synced_wait: Managed
wildfly_meta-data_0 process 731 exited with rc=0
Feb 21 21:44:01 [24024] IVM-1 crmd: notice: process_lrm_event: Result
of start operation for stateful_wildfly on IVM-1: 0 (ok) | call=175
key=stateful_wildfly_start_0 confirmed=true cib-update=115
Feb 21 21:44:01 [24019] IVM-1 cib: info:
cib_process_request: Forwarding
cib_modify operation for section status to all (origin=local/crmd/115)
Feb 21 21:44:01 [24019] IVM-1 cib: info: cib_perform_op: Diff:
--- 0.108.62 2
Feb 21 21:44:01 [24019] IVM-1 cib: info: cib_perform_op: Diff:
+++ 0.108.63 (null)
Feb 21 21:44:01 [24019] IVM-1 cib: info: cib_perform_op: +
/cib: @num_updates=63
Feb 21 21:44:01 [24019] IVM-1 cib: info: cib_perform_op: +
/cib/status/node_state[@id='2']/lrm[@id='2']/lrm_
resources/lrm_resource[@id='stateful_wildfly']/lrm_rsc_op[
@id='stateful_wildfly_last_0']: @operation_key=stateful_wildfly_start_0,
@operation=start, @transition-key=80:1:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079,
@transition-magic=0:0;80:1:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079,
@call-id=175, @exec-time=94019
Feb 21 21:44:01 [24019] IVM-1 cib: info:
cib_process_request: Completed
cib_modify operation for section status: OK (rc=0, origin=IVM-1/crmd/115,
version=0.108.63)
Feb 21 21:44:01 [24024] IVM-1 crmd: info: match_graph_event: Action
stateful_wildfly_start_0 (80) confirmed on IVM-1 (rc=0)
Feb 21 21:44:01 [24024] IVM-1 crmd: notice: te_rsc_command: Initiating
notify operation stateful_wildfly_post_notify_start_0 locally on IVM-1 |
action 287
Feb 21 21:44:01 [24024] IVM-1 crmd: info: do_lrm_rsc_op: Performing
key=287:1:0:e27e11f8-fc4e-493e-a4f2-80a02cb1f079
op=stateful_wildfly_notify_0
Feb 21 21:44:01 [24021] IVM-1 lrmd: info: log_execute: executing
- rsc:stateful_wildfly action:notify call_id:181
Feb 21 21:44:01 [24021] IVM-1 lrmd: info: log_finished: finished
- rsc:stateful_wildfly action:notify call_id:181 pid:762 exit-code:0
exec-time:54ms queue-time:0ms
Feb 21 21:44:01 [24024] IVM-1 crmd: info: match_graph_event: Action
stateful_wildfly_notify_0 (287) confirmed on IVM-1 (rc=0)
Feb 21 21:44:01 [24024] IVM-1 crmd: notice: process_lrm_event: Result
of notify operation for stateful_wildfly on IVM-1: 0 (ok) | call=181
key=stateful_wildfly_notify_0 confirmed=true cib-update=0
Feb 21 21:44:01 [24024] IVM-1 crmd: notice: run_graph: Transition 1
(Complete=52, Pending=0, Fired=0, Skipped=2, Incomplete=2,
Source=/usr/var/lib/pacemaker/pengine/pe-input-107.bz2): Stopped
Feb 21 21:44:01 [24024] IVM-1 crmd: info: do_state_transition: State
transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC
cause=C_FSA_INTERNAL origin=notify_crmd
Feb 21 21:44:01 [24024] IVM-1 crmd: info: crmd_notify_complete: Alert
84 (/opt/vplat/modules/pacemaker/pacemakerNotificationHandler) complete
Feb 21 21:44:01 [24023] IVM-1 pengine: notice: unpack_config: On loss
of CCM Quorum: Ignore
Feb 21 21:44:01 [24023] IVM-1 pengine: info:
determine_online_status: Node
IVM-1 is online
Feb 21 21:44:01 [24023] IVM-1 pengine: info:
determine_op_status: Operation
monitor found resource haproxy:0 active on IVM-1
Feb 21 21:44:01 [24023] IVM-1 pengine: info:
get_failcount_full: stateful_wildfly:0
has failed 2 times on IVM-1
Feb 21 21:44:01 [24023] IVM-1 pengine: info:
get_failcount_full: stateful_wildfly:0
has failed 2 times on IVM-1
Feb 21 21:44:01 [24023] IVM-1 pengine: warning:
unpack_rsc_op_failure: Processing
failed op promote for stateful_wildfly:0 on IVM-1: unknown error (1)
Feb 21 21:44:01 [24023] IVM-1 pengine: info: clone_print: Master/Slave
Set: apache2_MS [stateful_apache2]
Feb 21 21:44:01 [24023] IVM-1 pengine: info: short_print:
Masters: [ IVM-1 ]
Feb 21 21:44:01 [24023] IVM-1 pengine: info: short_print:
Stopped: [ IVM-0 ]
Feb 21 21:44:01 [24023] IVM-1 pengine: info: native_print: natomsip
(ocf::heartbeat:IPaddr2): Started IVM-1
Feb 21 21:44:01 [24023] IVM-1 pengine: info: native_print:
natcpcoreip (ocf::heartbeat:IPaddr2): Started IVM-1
Feb 21 21:44:01 [24023] IVM-1 pengine: info: clone_print: Clone Set:
ClusterMon-clone [ClusterMon]
Feb 21 21:44:01 [24023] IVM-1 pengine: info: short_print:
Started: [ IVM-1 ]
Feb 21 21:44:01 [24023] IVM-1 pengine: info: short_print:
Stopped: [ IVM-0 ]
Feb 21 21:44:01 [24023] IVM-1 pengine: info: clone_print: Master/Slave
Set: stateful_ms_set [stateful_consul]
Feb 21 21:44:01 [24023] IVM-1 pengine: info: short_print:
Masters: [ IVM-1 ]
Feb 21 21:44:01 [24023] IVM-1 pengine: info: short_print:
Stopped: [ IVM-0 ]
Feb 21 21:44:01 [24023] IVM-1 pengine: info: clone_print: Master/Slave
Set: wildfly_MS [stateful_wildfly]
Feb 21 21:44:01 [24023] IVM-1 pengine: info: short_print:
Slaves: [ IVM-1 ]
Feb 21 21:44:01 [24023] IVM-1 pengine: info: short_print:
Stopped: [ IVM-0 ]
Feb 21 21:44:01 [24023] IVM-1 pengine: info: clone_print: Master/Slave
Set: alarmSnmpd_MS [alarmSnmpd]
Feb 21 21:44:01 [24023] IVM-1 pengine: info: short_print:
Masters: [ IVM-1 ]
Feb 21 21:44:01 [24023] IVM-1 pengine: info: short_print:
Stopped: [ IVM-0 ]
Feb 21 21:44:01 [24023] IVM-1 pengine: info: native_print:
natomsip_for_snmpd (ocf::heartbeat:IPaddr2): Started IVM-1
Feb 21 21:44:01 [24023] IVM-1 pengine: info: clone_print: Master/Slave
Set: haproxy_MS [haproxy]
Feb 21 21:44:01 [24023] IVM-1 pengine: info: short_print:
Masters: [ IVM-1 ]
Feb 21 21:44:01 [24023] IVM-1 pengine: info: short_print:
Stopped: [ IVM-0 ]
Feb 21 21:44:01 [24023] IVM-1 pengine: info: native_print: nataaaip
(ocf::heartbeat:IPaddr2): Started IVM-1
Feb 21 21:44:01 [24023] IVM-1 pengine: info: native_print:
nataccessip (ocf::heartbeat:IPaddr2): Started IVM-1
Feb 21 21:44:01 [24023] IVM-1 pengine: info: clone_print: Master/Slave
Set: netactagent_MS [netactagent]
Feb 21 21:44:01 [24023] IVM-1 pengine: info: short_print:
Stopped: [ IVM-0 IVM-1 ]
Feb 21 21:44:01 [24023] IVM-1 pengine: info: native_print:
natomsip_for_netact (ocf::heartbeat:IPaddr2): Stopped
Feb 21 21:44:01 [24023] IVM-1 pengine: info: clone_print: Master/Slave
Set: RoutingManager_MS [RoutingManager]
Feb 21 21:44:01 [24023] IVM-1 pengine: info: short_print:
Masters: [ IVM-1 ]
Feb 21 21:44:01 [24023] IVM-1 pengine: info: short_print:
Stopped: [ IVM-0 ]
Feb 21 21:44:01 [24023] IVM-1 pengine: info: group_print: Resource
Group: floatingips
Feb 21 21:44:01 [24023] IVM-1 pengine: info: native_print:
natomsgwip (ocf::heartbeat:IPaddr2): Started IVM-1
Feb 21 21:44:01 [24023] IVM-1 pengine: info: native_print: omsip
(ocf::heartbeat:IPaddr2): Started IVM-1
Feb 21 21:44:01 [24023] IVM-1 pengine: info: native_print:
nataaagwip (ocf::heartbeat:IPaddr2): Started IVM-1
Feb 21 21:44:01 [24023] IVM-1 pengine: info: native_print: aaaip
(ocf::heartbeat:IPaddr2): Started IVM-1
Feb 21 21:44:01 [24023] IVM-1 pengine: info: native_print:
nataccessgwip (ocf::heartbeat:IPaddr2): Started IVM-1
Feb 21 21:44:01 [24023] IVM-1 pengine: info: native_print:
accessip (ocf::heartbeat:IPaddr2): Started IVM-1
Feb 21 21:44:01 [24023] IVM-1 pengine: info: native_print:
natcpcoregwip (ocf::heartbeat:IPaddr2): Started IVM-1
Feb 21 21:44:01 [24023] IVM-1 pengine: info: native_print:
cpcoreip (ocf::heartbeat:IPaddr2): Started IVM-1
--
Thanks,
Avinash Sharma
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20180223/6d0e249d/attachment-0001.html>
More information about the Users
mailing list