[ClusterLabs] Resources not starting some times after node reboot
Pritam Kharat
pritam.kharat at oneconvergence.com
Thu Oct 29 13:42:48 EDT 2015
Hi All,
I have single node with 5 resources running on it. When I rebooted node
sometimes I saw resources in stopped state though node comes online.
When looked in to the logs, one difference found in success and failure
case is, when
*Election Trigger (I_DC_TIMEOUT) just popped (20000ms) *occurred LRM did
not start the resources instead jumped to monitor action and then onwards
it did not start the resources at all.
But in success case this Election timeout did not come and first action
taken by LRM was start the resource and then start monitoring it making all
the resources started properly.
I have attached both the success and failure logs. Could some one please
explain the reason for this issue and how to solve this ?
My CRM configuration is -
root at sc-node-2:~# crm configure show
node $id="2" sc-node-2
primitive oc-fw-agent upstart:oc-fw-agent \
meta allow-migrate="true" migration-threshold="5" failure-timeout="120s" \
op monitor interval="15s" timeout="60s"
primitive oc-lb-agent upstart:oc-lb-agent \
meta allow-migrate="true" migration-threshold="5" failure-timeout="120s" \
op monitor interval="15s" timeout="60s"
primitive oc-service-manager upstart:oc-service-manager \
meta allow-migrate="true" migration-threshold="5" failure-timeout="120s" \
op monitor interval="15s" timeout="60s"
primitive oc-vpn-agent upstart:oc-vpn-agent \
meta allow-migrate="true" migration-threshold="5" failure-timeout="120s" \
op monitor interval="15s" timeout="60s"
primitive sc_vip ocf:heartbeat:IPaddr2 \
params ip="200.10.10.188" cidr_netmask="24" nic="eth1" \
op monitor interval="15s"
group sc-resources sc_vip oc-service-manager oc-fw-agent oc-lb-agent
oc-vpn-agent
property $id="cib-bootstrap-options" \
dc-version="1.1.10-42f2063" \
cluster-infrastructure="corosync" \
stonith-enabled="false" \
cluster-recheck-interval="3min" \
default-action-timeout="180s"
--
Thanks and Regards,
Pritam Kharat.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20151029/83854571/attachment-0002.html>
-------------- next part --------------
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ]
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: crmd_join_phase_log: join-1: sc-node-2=integrated
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: do_dc_join_finalize: join-1: Syncing our CIB to the rest of the cluster
Oct 29 13:02:15 [1016] sc-node-2 cib: notice: corosync_node_name: Unable to get node name for nodeid 2
Oct 29 13:02:15 [1016] sc-node-2 cib: notice: get_node_name: Defaulting to uname -n for the local corosync node name
Oct 29 13:02:15 [1016] sc-node-2 cib: info: cib_process_request: Completed cib_sync operation for section 'all': OK (rc=0, origin=local/crmd/14, version=0.11.0)
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: crm_update_peer_join: finalize_join_for: Node sc-node-2[2] - join-1 phase 2 -> 3
Oct 29 13:02:15 [1016] sc-node-2 cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=local/crmd/15, version=0.11.0)
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: erase_status_tag: Deleting xpath: //node_state[@uname='sc-node-2']/transient_attributes
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: update_attrd: Connecting to attrd... 5 retries remaining
Oct 29 13:02:15 [1016] sc-node-2 cib: info: cib_process_request: Completed cib_delete operation for section //node_state[@uname='sc-node-2']/transient_attributes: OK (rc=0, origin=local/crmd/16, version=0.11.0)
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: crm_update_peer_join: do_dc_join_ack: Node sc-node-2[2] - join-1 phase 3 -> 4
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: do_dc_join_ack: join-1: Updating node state to member for sc-node-2
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: erase_status_tag: Deleting xpath: //node_state[@uname='sc-node-2']/lrm
Oct 29 13:02:15 [1016] sc-node-2 cib: info: cib_process_request: Completed cib_delete operation for section //node_state[@uname='sc-node-2']/lrm: OK (rc=0, origin=local/crmd/17, version=0.11.0)
Oct 29 13:02:15 [1016] sc-node-2 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/18, version=0.11.1)
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: do_state_transition: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ]
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: abort_transition_graph: do_te_invoke:151 - Triggered transition abort (complete=1) : Peer Cancelled
Oct 29 13:02:15 [1019] sc-node-2 attrd: notice: attrd_local_callback: Sending full refresh (origin=crmd)
Oct 29 13:02:15 [1016] sc-node-2 cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=local/crmd/19, version=0.11.1)
Oct 29 13:02:15 [1016] sc-node-2 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/20, version=0.11.1)
Oct 29 13:02:15 [1016] sc-node-2 cib: info: cib_process_request: Completed cib_modify operation for section cib: OK (rc=0, origin=local/crmd/21, version=0.11.2)
Oct 29 13:02:15 [1016] sc-node-2 cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/22, version=0.11.2)
Oct 29 13:02:15 [1020] sc-node-2 pengine: info: determine_online_status: Node sc-node-2 is online
Oct 29 13:02:15 [1020] sc-node-2 pengine: info: group_print: Resource Group: sc-resources
Oct 29 13:02:15 [1020] sc-node-2 pengine: info: native_print: sc_vip (ocf::heartbeat:IPaddr2): Stopped
Oct 29 13:02:15 [1020] sc-node-2 pengine: info: native_print: oc-service-manager (upstart:oc-service-manager): Stopped
Oct 29 13:02:15 [1020] sc-node-2 pengine: info: native_print: oc-fw-agent (upstart:oc-fw-agent): Stopped
Oct 29 13:02:15 [1020] sc-node-2 pengine: info: native_print: oc-lb-agent (upstart:oc-lb-agent): Stopped
Oct 29 13:02:15 [1020] sc-node-2 pengine: info: native_print: oc-vpn-agent (upstart:oc-vpn-agent): Stopped
Oct 29 13:02:15 [1020] sc-node-2 pengine: info: RecurringOp: Start recurring monitor (15s) for sc_vip on sc-node-2
Oct 29 13:02:15 [1020] sc-node-2 pengine: info: RecurringOp: Start recurring monitor (15s) for oc-service-manager on sc-node-2
Oct 29 13:02:15 [1020] sc-node-2 pengine: info: RecurringOp: Start recurring monitor (15s) for oc-fw-agent on sc-node-2
Oct 29 13:02:15 [1020] sc-node-2 pengine: info: RecurringOp: Start recurring monitor (15s) for oc-lb-agent on sc-node-2
Oct 29 13:02:15 [1020] sc-node-2 pengine: info: RecurringOp: Start recurring monitor (15s) for oc-vpn-agent on sc-node-2
Oct 29 13:02:15 [1020] sc-node-2 pengine: notice: LogActions: Start sc_vip (sc-node-2)
Oct 29 13:02:15 [1020] sc-node-2 pengine: notice: LogActions: Start oc-service-manager (sc-node-2)
Oct 29 13:02:15 [1020] sc-node-2 pengine: notice: LogActions: Start oc-fw-agent (sc-node-2)
Oct 29 13:02:15 [1020] sc-node-2 pengine: notice: LogActions: Start oc-lb-agent (sc-node-2)
Oct 29 13:02:15 [1020] sc-node-2 pengine: notice: LogActions: Start oc-vpn-agent (sc-node-2)
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: do_te_invoke: Processing graph 0 (ref=pe_calc-dc-1446138135-7) derived from /var/lib/pacemaker/pengine/pe-input-340.bz2
Oct 29 13:02:15 [1021] sc-node-2 crmd: notice: te_rsc_command: Initiating action 4: monitor sc_vip_monitor_0 on sc-node-2 (local)
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: process_lrmd_get_rsc_info: Resource 'sc_vip' not found (0 active resources)
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: process_lrmd_rsc_register: Added 'sc_vip' to the rsc list (1 active resources)
Oct 29 13:02:15 [1020] sc-node-2 pengine: notice: process_pe_message: Calculated Transition 0: /var/lib/pacemaker/pengine/pe-input-340.bz2
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: do_lrm_rsc_op: Performing key=4:0:7:419f8af1-d7a7-4913-b197-2d89ff519e66 op=sc_vip_monitor_0
Oct 29 13:02:15 [1021] sc-node-2 crmd: notice: te_rsc_command: Initiating action 5: monitor oc-service-manager_monitor_0 on sc-node-2 (local)
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: process_lrmd_get_rsc_info: Resource 'oc-service-manager' not found (1 active resources)
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: process_lrmd_rsc_register: Added 'oc-service-manager' to the rsc list (2 active resources)
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: do_lrm_rsc_op: Performing key=5:0:7:419f8af1-d7a7-4913-b197-2d89ff519e66 op=oc-service-manager_monitor_0
Oct 29 13:02:15 [1021] sc-node-2 crmd: notice: te_rsc_command: Initiating action 6: monitor oc-fw-agent_monitor_0 on sc-node-2 (local)
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: get_first_instance: Result: (null)
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: upstart_job_running: oc-service-manager is not running
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: process_lrmd_get_rsc_info: Resource 'oc-fw-agent' not found (2 active resources)
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: process_lrmd_rsc_register: Added 'oc-fw-agent' to the rsc list (3 active resources)
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: do_lrm_rsc_op: Performing key=6:0:7:419f8af1-d7a7-4913-b197-2d89ff519e66 op=oc-fw-agent_monitor_0
Oct 29 13:02:15 [1021] sc-node-2 crmd: notice: te_rsc_command: Initiating action 7: monitor oc-lb-agent_monitor_0 on sc-node-2 (local)
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: get_first_instance: Result: (null)
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: upstart_job_running: oc-fw-agent is not running
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: process_lrmd_get_rsc_info: Resource 'oc-lb-agent' not found (3 active resources)
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: process_lrmd_rsc_register: Added 'oc-lb-agent' to the rsc list (4 active resources)
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: do_lrm_rsc_op: Performing key=7:0:7:419f8af1-d7a7-4913-b197-2d89ff519e66 op=oc-lb-agent_monitor_0
Oct 29 13:02:15 [1021] sc-node-2 crmd: notice: te_rsc_command: Initiating action 8: monitor oc-vpn-agent_monitor_0 on sc-node-2 (local)
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: get_first_instance: Result: (null)
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: upstart_job_running: oc-lb-agent is not running
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: process_lrmd_get_rsc_info: Resource 'oc-vpn-agent' not found (4 active resources)
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: process_lrmd_rsc_register: Added 'oc-vpn-agent' to the rsc list (5 active resources)
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: do_lrm_rsc_op: Performing key=8:0:7:419f8af1-d7a7-4913-b197-2d89ff519e66 op=oc-vpn-agent_monitor_0
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: get_first_instance: Result: (null)
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: upstart_job_running: oc-vpn-agent is not running
Oct 29 13:02:15 [1016] sc-node-2 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/23, version=0.11.3)
Oct 29 13:02:15 [1021] sc-node-2 crmd: notice: process_lrm_event: LRM operation oc-service-manager_monitor_0 (call=9, rc=7, cib-update=23, confirmed=true) not running
Oct 29 13:02:15 [1021] sc-node-2 crmd: notice: process_lrm_event: LRM operation oc-fw-agent_monitor_0 (call=13, rc=7, cib-update=24, confirmed=true) not running
Oct 29 13:02:15 [1016] sc-node-2 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/24, version=0.11.4)
Oct 29 13:02:15 [1021] sc-node-2 crmd: notice: process_lrm_event: LRM operation oc-lb-agent_monitor_0 (call=17, rc=7, cib-update=25, confirmed=true) not running
Oct 29 13:02:15 [1016] sc-node-2 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/25, version=0.11.5)
Oct 29 13:02:15 [1021] sc-node-2 crmd: notice: process_lrm_event: LRM operation oc-vpn-agent_monitor_0 (call=21, rc=7, cib-update=26, confirmed=true) not running
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: match_graph_event: Action oc-service-manager_monitor_0 (5) confirmed on sc-node-2 (rc=0)
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: match_graph_event: Action oc-fw-agent_monitor_0 (6) confirmed on sc-node-2 (rc=0)
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: match_graph_event: Action oc-lb-agent_monitor_0 (7) confirmed on sc-node-2 (rc=0)
Oct 29 13:02:15 [1016] sc-node-2 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/26, version=0.11.6)
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: match_graph_event: Action oc-vpn-agent_monitor_0 (8) confirmed on sc-node-2 (rc=0)
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: services_os_action_execute: Managed IPaddr2_meta-data_0 process 1336 exited with rc=0
Oct 29 13:02:15 [1021] sc-node-2 crmd: notice: process_lrm_event: LRM operation sc_vip_monitor_0 (call=5, rc=7, cib-update=27, confirmed=true) not running
Oct 29 13:02:15 [1016] sc-node-2 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/27, version=0.11.7)
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: match_graph_event: Action sc_vip_monitor_0 (4) confirmed on sc-node-2 (rc=0)
Oct 29 13:02:15 [1021] sc-node-2 crmd: notice: te_rsc_command: Initiating action 3: probe_complete probe_complete on sc-node-2 (local) - no waiting
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: te_rsc_command: Action 3 confirmed - no wait
Oct 29 13:02:15 [1019] sc-node-2 attrd: notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
Oct 29 13:02:15 [1021] sc-node-2 crmd: notice: te_rsc_command: Initiating action 9: start sc_vip_start_0 on sc-node-2 (local)
Oct 29 13:02:15 [1016] sc-node-2 cib: info: cib_process_request: Completed cib_query operation for section //cib/status//node_state[@id='2']//transient_attributes//nvpair[@name='probe_complete']: No such device or address (rc=-6, origin=local/attrd/2, version=0.11.7)
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: do_lrm_rsc_op: Performing key=9:0:0:419f8af1-d7a7-4913-b197-2d89ff519e66 op=sc_vip_start_0
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: log_execute: executing - rsc:sc_vip action:start call_id:28
Oct 29 13:02:15 [1016] sc-node-2 cib: info: cib_process_request: Completed cib_query operation for section /cib: OK (rc=0, origin=local/attrd/3, version=0.11.7)
Oct 29 13:02:15 [1019] sc-node-2 attrd: notice: attrd_perform_update: Sent update 4: probe_complete=true
Oct 29 13:02:15 [1016] sc-node-2 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/attrd/4, version=0.11.8)
Oct 29 13:02:15 [1019] sc-node-2 attrd: notice: corosync_node_name: Unable to get node name for nodeid 2
Oct 29 13:02:15 [1019] sc-node-2 attrd: notice: get_node_name: Defaulting to uname -n for the local corosync node name
IPaddr2(sc_vip)[1340]: 2015/10/29_13:02:15 INFO: Adding IPv4 address 200.10.10.188/24 with broadcast address 200.10.10.255 to device eth1
IPaddr2(sc_vip)[1340]: 2015/10/29_13:02:15 INFO: Bringing device eth1 up
IPaddr2(sc_vip)[1340]: 2015/10/29_13:02:15 INFO: /usr/lib/heartbeat/send_arp -i 200 -r 5 -p /var/run/resource-agents/send_arp-200.10.10.188 eth1 200.10.10.188 auto not_used not_used
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: log_finished: finished - rsc:sc_vip action:start call_id:28 pid:1340 exit-code:0 exec-time:38ms queue-time:0ms
Oct 29 13:02:15 [1021] sc-node-2 crmd: notice: process_lrm_event: LRM operation sc_vip_start_0 (call=28, rc=0, cib-update=28, confirmed=true) ok
Oct 29 13:02:15 [1016] sc-node-2 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/28, version=0.11.9)
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: match_graph_event: Action sc_vip_start_0 (9) confirmed on sc-node-2 (rc=0)
Oct 29 13:02:15 [1021] sc-node-2 crmd: notice: te_rsc_command: Initiating action 10: monitor sc_vip_monitor_15000 on sc-node-2 (local)
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: do_lrm_rsc_op: Performing key=10:0:0:419f8af1-d7a7-4913-b197-2d89ff519e66 op=sc_vip_monitor_15000
Oct 29 13:02:15 [1021] sc-node-2 crmd: notice: te_rsc_command: Initiating action 11: start oc-service-manager_start_0 on sc-node-2 (local)
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: do_lrm_rsc_op: Performing key=11:0:0:419f8af1-d7a7-4913-b197-2d89ff519e66 op=oc-service-manager_start_0
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: log_execute: executing - rsc:oc-service-manager action:start call_id:33
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: upstart_job_exec_done: Call to start passed: type '(o)' /com/ubuntu/Upstart/jobs/oc_2dservice_2dmanager/_
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: log_finished: finished - rsc:oc-service-manager action:start call_id:33 exit-code:0 exec-time:19ms queue-time:0ms
Oct 29 13:02:15 [1021] sc-node-2 crmd: notice: process_lrm_event: LRM operation oc-service-manager_start_0 (call=33, rc=0, cib-update=29, confirmed=true) ok
Oct 29 13:02:15 [1016] sc-node-2 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/29, version=0.11.10)
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: match_graph_event: Action oc-service-manager_start_0 (11) confirmed on sc-node-2 (rc=0)
Oct 29 13:02:15 [1021] sc-node-2 crmd: notice: te_rsc_command: Initiating action 12: monitor oc-service-manager_monitor_15000 on sc-node-2 (local)
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: do_lrm_rsc_op: Performing key=12:0:0:419f8af1-d7a7-4913-b197-2d89ff519e66 op=oc-service-manager_monitor_15000
Oct 29 13:02:15 [1021] sc-node-2 crmd: notice: te_rsc_command: Initiating action 13: start oc-fw-agent_start_0 on sc-node-2 (local)
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: get_first_instance: Result: /com/ubuntu/Upstart/jobs/oc_2dservice_2dmanager/_
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: upstart_job_property: Calling GetAll on /com/ubuntu/Upstart/jobs/oc_2dservice_2dmanager/_
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: upstart_job_property: Call to GetAll passed: type '(a{sv})' 1
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: upstart_job_property: Got value 'running' for /com/ubuntu/Upstart/jobs/oc_2dservice_2dmanager/_[state]
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: upstart_job_running: State of oc-service-manager: running
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: upstart_job_running: oc-service-manager is running
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: do_lrm_rsc_op: Performing key=13:0:0:419f8af1-d7a7-4913-b197-2d89ff519e66 op=oc-fw-agent_start_0
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: log_execute: executing - rsc:oc-fw-agent action:start call_id:38
Oct 29 13:02:15 [1021] sc-node-2 crmd: notice: process_lrm_event: LRM operation oc-service-manager_monitor_15000 (call=36, rc=0, cib-update=30, confirmed=false) ok
Oct 29 13:02:15 [1016] sc-node-2 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/30, version=0.11.11)
Oct 29 13:02:15 [1021] sc-node-2 crmd: notice: process_lrm_event: LRM operation sc_vip_monitor_15000 (call=31, rc=0, cib-update=31, confirmed=false) ok
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: match_graph_event: Action oc-service-manager_monitor_15000 (12) confirmed on sc-node-2 (rc=0)
Oct 29 13:02:15 [1016] sc-node-2 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/31, version=0.11.12)
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: match_graph_event: Action sc_vip_monitor_15000 (10) confirmed on sc-node-2 (rc=0)
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: upstart_job_exec_done: Call to start passed: type '(o)' /com/ubuntu/Upstart/jobs/oc_2dfw_2dagent/_
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: log_finished: finished - rsc:oc-fw-agent action:start call_id:38 exit-code:0 exec-time:29ms queue-time:0ms
Oct 29 13:02:15 [1021] sc-node-2 crmd: notice: process_lrm_event: LRM operation oc-fw-agent_start_0 (call=38, rc=0, cib-update=32, confirmed=true) ok
Oct 29 13:02:15 [1016] sc-node-2 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/32, version=0.11.13)
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: match_graph_event: Action oc-fw-agent_start_0 (13) confirmed on sc-node-2 (rc=0)
Oct 29 13:02:15 [1021] sc-node-2 crmd: notice: te_rsc_command: Initiating action 14: monitor oc-fw-agent_monitor_15000 on sc-node-2 (local)
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: do_lrm_rsc_op: Performing key=14:0:0:419f8af1-d7a7-4913-b197-2d89ff519e66 op=oc-fw-agent_monitor_15000
Oct 29 13:02:15 [1021] sc-node-2 crmd: notice: te_rsc_command: Initiating action 15: start oc-lb-agent_start_0 on sc-node-2 (local)
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: get_first_instance: Result: /com/ubuntu/Upstart/jobs/oc_2dfw_2dagent/_
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: upstart_job_property: Calling GetAll on /com/ubuntu/Upstart/jobs/oc_2dfw_2dagent/_
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: upstart_job_property: Call to GetAll passed: type '(a{sv})' 1
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: upstart_job_property: Got value 'running' for /com/ubuntu/Upstart/jobs/oc_2dfw_2dagent/_[state]
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: upstart_job_running: State of oc-fw-agent: running
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: upstart_job_running: oc-fw-agent is running
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: do_lrm_rsc_op: Performing key=15:0:0:419f8af1-d7a7-4913-b197-2d89ff519e66 op=oc-lb-agent_start_0
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: log_execute: executing - rsc:oc-lb-agent action:start call_id:45
Oct 29 13:02:15 [1021] sc-node-2 crmd: notice: process_lrm_event: LRM operation oc-fw-agent_monitor_15000 (call=43, rc=0, cib-update=33, confirmed=false) ok
Oct 29 13:02:15 [1016] sc-node-2 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/33, version=0.11.14)
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: match_graph_event: Action oc-fw-agent_monitor_15000 (14) confirmed on sc-node-2 (rc=0)
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: upstart_job_exec_done: Call to start passed: type '(o)' /com/ubuntu/Upstart/jobs/oc_2dlb_2dagent/_
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: log_finished: finished - rsc:oc-lb-agent action:start call_id:45 exit-code:0 exec-time:23ms queue-time:0ms
Oct 29 13:02:15 [1021] sc-node-2 crmd: notice: process_lrm_event: LRM operation oc-lb-agent_start_0 (call=45, rc=0, cib-update=34, confirmed=true) ok
Oct 29 13:02:15 [1016] sc-node-2 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/34, version=0.11.15)
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: match_graph_event: Action oc-lb-agent_start_0 (15) confirmed on sc-node-2 (rc=0)
Oct 29 13:02:15 [1021] sc-node-2 crmd: notice: te_rsc_command: Initiating action 16: monitor oc-lb-agent_monitor_15000 on sc-node-2 (local)
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: do_lrm_rsc_op: Performing key=16:0:0:419f8af1-d7a7-4913-b197-2d89ff519e66 op=oc-lb-agent_monitor_15000
Oct 29 13:02:15 [1021] sc-node-2 crmd: notice: te_rsc_command: Initiating action 17: start oc-vpn-agent_start_0 on sc-node-2 (local)
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: get_first_instance: Result: /com/ubuntu/Upstart/jobs/oc_2dlb_2dagent/_
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: upstart_job_property: Calling GetAll on /com/ubuntu/Upstart/jobs/oc_2dlb_2dagent/_
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: upstart_job_property: Call to GetAll passed: type '(a{sv})' 1
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: upstart_job_property: Got value 'running' for /com/ubuntu/Upstart/jobs/oc_2dlb_2dagent/_[state]
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: upstart_job_running: State of oc-lb-agent: running
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: upstart_job_running: oc-lb-agent is running
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: do_lrm_rsc_op: Performing key=17:0:0:419f8af1-d7a7-4913-b197-2d89ff519e66 op=oc-vpn-agent_start_0
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: log_execute: executing - rsc:oc-vpn-agent action:start call_id:51
Oct 29 13:02:15 [1021] sc-node-2 crmd: notice: process_lrm_event: LRM operation oc-lb-agent_monitor_15000 (call=49, rc=0, cib-update=35, confirmed=false) ok
Oct 29 13:02:15 [1016] sc-node-2 cib: info: crm_client_new: Connecting 0x7f02b54c2890 for uid=0 gid=0 pid=1439 id=96b0787a-1720-403f-bdb6-647988cbe4b7
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: upstart_job_exec_done: Call to start passed: type '(o)' /com/ubuntu/Upstart/jobs/oc_2dvpn_2dagent/_
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: log_finished: finished - rsc:oc-vpn-agent action:start call_id:51 exit-code:0 exec-time:15ms queue-time:0ms
Oct 29 13:02:15 [1021] sc-node-2 crmd: notice: process_lrm_event: LRM operation oc-vpn-agent_start_0 (call=51, rc=0, cib-update=36, confirmed=true) ok
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: match_graph_event: Action oc-lb-agent_monitor_15000 (16) confirmed on sc-node-2 (rc=0)
Oct 29 13:02:15 [1016] sc-node-2 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/35, version=0.11.16)
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: match_graph_event: Action oc-vpn-agent_start_0 (17) confirmed on sc-node-2 (rc=0)
Oct 29 13:02:15 [1021] sc-node-2 crmd: notice: te_rsc_command: Initiating action 18: monitor oc-vpn-agent_monitor_15000 on sc-node-2 (local)
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: do_lrm_rsc_op: Performing key=18:0:0:419f8af1-d7a7-4913-b197-2d89ff519e66 op=oc-vpn-agent_monitor_15000
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: get_first_instance: Result: /com/ubuntu/Upstart/jobs/oc_2dvpn_2dagent/_
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: upstart_job_property: Calling GetAll on /com/ubuntu/Upstart/jobs/oc_2dvpn_2dagent/_
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: upstart_job_property: Call to GetAll passed: type '(a{sv})' 1
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: upstart_job_property: Got value 'running' for /com/ubuntu/Upstart/jobs/oc_2dvpn_2dagent/_[state]
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: upstart_job_running: State of oc-vpn-agent: running
Oct 29 13:02:15 [1018] sc-node-2 lrmd: info: upstart_job_running: oc-vpn-agent is running
Oct 29 13:02:15 [1021] sc-node-2 crmd: notice: process_lrm_event: LRM operation oc-vpn-agent_monitor_15000 (call=55, rc=0, cib-update=37, confirmed=false) ok
Oct 29 13:02:15 [1016] sc-node-2 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/36, version=0.11.17)
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: match_graph_event: Action oc-vpn-agent_monitor_15000 (18) confirmed on sc-node-2 (rc=0)
Oct 29 13:02:15 [1021] sc-node-2 crmd: notice: run_graph: Transition 0 (Complete=19, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-340.bz2): Complete
Oct 29 13:02:15 [1021] sc-node-2 crmd: info: do_log: FSA: Input I_TE_SUCCESS from notify_crmd() received in state S_TRANSITION_ENGINE
Oct 29 13:02:15 [1021] sc-node-2 crmd: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
Oct 29 13:02:15 [1016] sc-node-2 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/37, version=0.11.18)
Oct 29 13:02:15 [1016] sc-node-2 cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crm_mon/2, version=0.11.18)
Oct 29 13:02:15 [1016] sc-node-2 cib: info: crm_client_destroy: Destroying 0 events
Oct 29 13:02:17 [1016] sc-node-2 cib: info: crm_client_new: Connecting 0x7f02b54c2890 for uid=0 gid=0 pid=1463 id=b07521fd-f2bb-4267-8c2d-3ced76636afe
Oct 29 13:02:17 [1016] sc-node-2 cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crm_mon/2, version=0.11.18)
Oct 29 13:02:17 [1016] sc-node-2 cib: info: crm_client_destroy: Destroying 0 events
-------------- next part --------------
Oct 29 13:10:03 [1023] sc-node-2 cib: info: crm_client_destroy: Destroying 0 events
Oct 29 13:10:05 [1023] sc-node-2 cib: info: crm_client_new: Connecting 0x7f890dc4ad00 for uid=0 gid=0 pid=1291 id=43b404f1-46da-48a4-b0d2-94ada6bf98eb
Oct 29 13:10:05 [1023] sc-node-2 cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crm_mon/2, version=0.11.0)
Oct 29 13:10:05 [1023] sc-node-2 cib: info: crm_client_destroy: Destroying 0 events
Oct 29 13:10:06 [1028] sc-node-2 crmd: info: crm_timer_popped: Election Trigger (I_DC_TIMEOUT) just popped (20000ms)
Oct 29 13:10:06 [1028] sc-node-2 crmd: warning: do_log: FSA: Input I_DC_TIMEOUT from crm_timer_popped() received in state S_PENDING
Oct 29 13:10:06 [1028] sc-node-2 crmd: info: do_state_transition: State transition S_PENDING -> S_ELECTION [ input=I_DC_TIMEOUT cause=C_TIMER_POPPED origin=crm_timer_popped ]
Oct 29 13:10:06 [1028] sc-node-2 crmd: info: do_log: FSA: Input I_ELECTION_DC from do_election_check() received in state S_ELECTION
Oct 29 13:10:06 [1028] sc-node-2 crmd: notice: do_state_transition: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=do_election_check ]
Oct 29 13:10:06 [1028] sc-node-2 crmd: info: do_te_control: Registering TE UUID: e130e80d-75e0-416f-ac5a-d318fcdd6ec2
Oct 29 13:10:06 [1028] sc-node-2 crmd: info: set_graph_functions: Setting custom graph functions
Oct 29 13:10:06 [1027] sc-node-2 pengine: info: crm_client_new: Connecting 0x7f5a24271a60 for uid=106 gid=113 pid=1028 id=03ff4b05-5b82-41b8-a5fc-71025419cd50
Oct 29 13:10:06 [1028] sc-node-2 crmd: info: do_dc_takeover: Taking over DC status for this partition
Oct 29 13:10:06 [1023] sc-node-2 cib: info: cib_process_readwrite: We are now in R/W mode
Oct 29 13:10:06 [1023] sc-node-2 cib: info: cib_process_request: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/6, version=0.11.0)
Oct 29 13:10:06 [1023] sc-node-2 cib: info: cib_process_request: Completed cib_modify operation for section cib: OK (rc=0, origin=local/crmd/7, version=0.11.0)
Oct 29 13:10:06 [1023] sc-node-2 cib: info: cib_process_request: Completed cib_query operation for section //cib/configuration/crm_config//cluster_property_set//nvpair[@name='dc-version']: OK (rc=0, origin=local/crmd/8, version=0.11.0)
Oct 29 13:10:06 [1023] sc-node-2 cib: info: cib_process_request: Completed cib_modify operation for section crm_config: OK (rc=0, origin=local/crmd/9, version=0.11.0)
Oct 29 13:10:06 [1023] sc-node-2 cib: info: cib_process_request: Completed cib_query operation for section //cib/configuration/crm_config//cluster_property_set//nvpair[@name='cluster-infrastructure']: OK (rc=0, origin=local/crmd/10, version=0.11.0)
Oct 29 13:10:06 [1028] sc-node-2 crmd: info: join_make_offer: Making join offers based on membership 2088
Oct 29 13:10:06 [1028] sc-node-2 crmd: info: join_make_offer: join-1: Sending offer to sc-node-2
Oct 29 13:10:06 [1028] sc-node-2 crmd: info: crm_update_peer_join: join_make_offer: Node sc-node-2[2] - join-1 phase 0 -> 1
Oct 29 13:10:06 [1028] sc-node-2 crmd: info: do_dc_join_offer_all: join-1: Waiting on 1 outstanding join acks
Oct 29 13:10:06 [1023] sc-node-2 cib: info: cib_process_request: Completed cib_modify operation for section crm_config: OK (rc=0, origin=local/crmd/11, version=0.11.0)
Oct 29 13:10:06 [1023] sc-node-2 cib: info: cib_process_request: Completed cib_query operation for section crm_config: OK (rc=0, origin=local/crmd/12, version=0.11.0)
Oct 29 13:10:06 [1028] sc-node-2 crmd: info: update_dc: Set DC to sc-node-2 (3.0.7)
Oct 29 13:10:06 [1023] sc-node-2 cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/13, version=0.11.0)
Oct 29 13:10:06 [1028] sc-node-2 crmd: info: crm_update_peer_join: do_dc_join_filter_offer: Node sc-node-2[2] - join-1 phase 1 -> 2
Oct 29 13:10:06 [1028] sc-node-2 crmd: info: crm_update_peer_expected: do_dc_join_filter_offer: Node sc-node-2[2] - expected state is now member
Oct 29 13:10:06 [1028] sc-node-2 crmd: info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ]
Oct 29 13:10:06 [1028] sc-node-2 crmd: info: crmd_join_phase_log: join-1: sc-node-2=integrated
Oct 29 13:10:06 [1028] sc-node-2 crmd: info: do_dc_join_finalize: join-1: Syncing our CIB to the rest of the cluster
Oct 29 13:10:06 [1023] sc-node-2 cib: notice: corosync_node_name: Unable to get node name for nodeid 2
Oct 29 13:10:06 [1023] sc-node-2 cib: notice: get_node_name: Defaulting to uname -n for the local corosync node name
Oct 29 13:10:06 [1023] sc-node-2 cib: info: cib_process_request: Completed cib_sync operation for section 'all': OK (rc=0, origin=local/crmd/14, version=0.11.0)
Oct 29 13:10:06 [1028] sc-node-2 crmd: info: crm_update_peer_join: finalize_join_for: Node sc-node-2[2] - join-1 phase 2 -> 3
Oct 29 13:10:06 [1023] sc-node-2 cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=local/crmd/15, version=0.11.0)
Oct 29 13:10:06 [1028] sc-node-2 crmd: info: erase_status_tag: Deleting xpath: //node_state[@uname='sc-node-2']/transient_attributes
Oct 29 13:10:06 [1028] sc-node-2 crmd: info: update_attrd: Connecting to attrd... 5 retries remaining
Oct 29 13:10:06 [1023] sc-node-2 cib: info: cib_process_request: Completed cib_delete operation for section //node_state[@uname='sc-node-2']/transient_attributes: OK (rc=0, origin=local/crmd/16, version=0.11.0)
Oct 29 13:10:06 [1028] sc-node-2 crmd: info: crm_update_peer_join: do_dc_join_ack: Node sc-node-2[2] - join-1 phase 3 -> 4
Oct 29 13:10:06 [1028] sc-node-2 crmd: info: do_dc_join_ack: join-1: Updating node state to member for sc-node-2
Oct 29 13:10:06 [1028] sc-node-2 crmd: info: erase_status_tag: Deleting xpath: //node_state[@uname='sc-node-2']/lrm
Oct 29 13:10:06 [1023] sc-node-2 cib: info: cib_process_request: Completed cib_delete operation for section //node_state[@uname='sc-node-2']/lrm: OK (rc=0, origin=local/crmd/17, version=0.11.0)
Oct 29 13:10:06 [1023] sc-node-2 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/18, version=0.11.1)
Oct 29 13:10:06 [1028] sc-node-2 crmd: info: do_state_transition: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ]
Oct 29 13:10:06 [1028] sc-node-2 crmd: info: abort_transition_graph: do_te_invoke:151 - Triggered transition abort (complete=1) : Peer Cancelled
Oct 29 13:10:06 [1026] sc-node-2 attrd: notice: attrd_local_callback: Sending full refresh (origin=crmd)
Oct 29 13:10:06 [1023] sc-node-2 cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=local/crmd/19, version=0.11.1)
Oct 29 13:10:06 [1023] sc-node-2 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/20, version=0.11.1)
Oct 29 13:10:06 [1023] sc-node-2 cib: info: cib_process_request: Completed cib_modify operation for section cib: OK (rc=0, origin=local/crmd/21, version=0.11.2)
Oct 29 13:10:06 [1023] sc-node-2 cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crmd/22, version=0.11.2)
Oct 29 13:10:06 [1027] sc-node-2 pengine: info: determine_online_status: Node sc-node-2 is online
Oct 29 13:10:06 [1027] sc-node-2 pengine: info: group_print: Resource Group: sc-resources
Oct 29 13:10:06 [1027] sc-node-2 pengine: info: native_print: sc_vip (ocf::heartbeat:IPaddr2): Stopped
Oct 29 13:10:06 [1027] sc-node-2 pengine: info: native_print: oc-service-manager (upstart:oc-service-manager): Stopped
Oct 29 13:10:06 [1027] sc-node-2 pengine: info: native_print: oc-fw-agent (upstart:oc-fw-agent): Stopped
Oct 29 13:10:06 [1027] sc-node-2 pengine: info: native_print: oc-lb-agent (upstart:oc-lb-agent): Stopped
Oct 29 13:10:06 [1027] sc-node-2 pengine: info: native_print: oc-vpn-agent (upstart:oc-vpn-agent): Stopped
Oct 29 13:10:06 [1027] sc-node-2 pengine: info: RecurringOp: Start recurring monitor (15s) for sc_vip on sc-node-2
Oct 29 13:10:06 [1027] sc-node-2 pengine: info: RecurringOp: Start recurring monitor (15s) for oc-service-manager on sc-node-2
Oct 29 13:10:06 [1027] sc-node-2 pengine: info: RecurringOp: Start recurring monitor (15s) for oc-fw-agent on sc-node-2
Oct 29 13:10:06 [1027] sc-node-2 pengine: info: RecurringOp: Start recurring monitor (15s) for oc-lb-agent on sc-node-2
Oct 29 13:10:06 [1027] sc-node-2 pengine: info: RecurringOp: Start recurring monitor (15s) for oc-vpn-agent on sc-node-2
Oct 29 13:10:06 [1027] sc-node-2 pengine: notice: LogActions: Start sc_vip (sc-node-2)
Oct 29 13:10:06 [1027] sc-node-2 pengine: notice: LogActions: Start oc-service-manager (sc-node-2)
Oct 29 13:10:06 [1027] sc-node-2 pengine: notice: LogActions: Start oc-fw-agent (sc-node-2)
Oct 29 13:10:06 [1027] sc-node-2 pengine: notice: LogActions: Start oc-lb-agent (sc-node-2)
Oct 29 13:10:06 [1027] sc-node-2 pengine: notice: LogActions: Start oc-vpn-agent (sc-node-2)
Oct 29 13:10:06 [1028] sc-node-2 crmd: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response ]
Oct 29 13:10:06 [1028] sc-node-2 crmd: info: do_te_invoke: Processing graph 0 (ref=pe_calc-dc-1446138606-7) derived from /var/lib/pacemaker/pengine/pe-input-347.bz2
Oct 29 13:10:06 [1027] sc-node-2 pengine: notice: process_pe_message: Calculated Transition 0: /var/lib/pacemaker/pengine/pe-input-347.bz2
Oct 29 13:10:06 [1028] sc-node-2 crmd: notice: te_rsc_command: Initiating action 4: monitor sc_vip_monitor_0 on sc-node-2 (local)
Oct 29 13:10:06 [1025] sc-node-2 lrmd: info: process_lrmd_get_rsc_info: Resource 'sc_vip' not found (0 active resources)
Oct 29 13:10:06 [1025] sc-node-2 lrmd: info: process_lrmd_rsc_register: Added 'sc_vip' to the rsc list (1 active resources)
Oct 29 13:10:06 [1028] sc-node-2 crmd: info: do_lrm_rsc_op: Performing key=4:0:7:e130e80d-75e0-416f-ac5a-d318fcdd6ec2 op=sc_vip_monitor_0
Oct 29 13:10:06 [1028] sc-node-2 crmd: notice: te_rsc_command: Initiating action 5: monitor oc-service-manager_monitor_0 on sc-node-2 (local)
Oct 29 13:10:06 [1025] sc-node-2 lrmd: info: process_lrmd_get_rsc_info: Resource 'oc-service-manager' not found (1 active resources)
Oct 29 13:10:06 [1025] sc-node-2 lrmd: info: process_lrmd_rsc_register: Added 'oc-service-manager' to the rsc list (2 active resources)
Oct 29 13:10:06 [1028] sc-node-2 crmd: info: do_lrm_rsc_op: Performing key=5:0:7:e130e80d-75e0-416f-ac5a-d318fcdd6ec2 op=oc-service-manager_monitor_0
Oct 29 13:10:06 [1028] sc-node-2 crmd: notice: te_rsc_command: Initiating action 6: monitor oc-fw-agent_monitor_0 on sc-node-2 (local)
Oct 29 13:10:06 [1025] sc-node-2 lrmd: info: get_first_instance: Result: (null)
Oct 29 13:10:06 [1025] sc-node-2 lrmd: info: upstart_job_running: oc-service-manager is not running
Oct 29 13:10:06 [1025] sc-node-2 lrmd: info: process_lrmd_get_rsc_info: Resource 'oc-fw-agent' not found (2 active resources)
Oct 29 13:10:06 [1025] sc-node-2 lrmd: info: process_lrmd_rsc_register: Added 'oc-fw-agent' to the rsc list (3 active resources)
Oct 29 13:10:06 [1028] sc-node-2 crmd: info: do_lrm_rsc_op: Performing key=6:0:7:e130e80d-75e0-416f-ac5a-d318fcdd6ec2 op=oc-fw-agent_monitor_0
Oct 29 13:10:06 [1028] sc-node-2 crmd: notice: te_rsc_command: Initiating action 7: monitor oc-lb-agent_monitor_0 on sc-node-2 (local)
Oct 29 13:10:06 [1025] sc-node-2 lrmd: info: get_first_instance: Result: (null)
Oct 29 13:10:06 [1025] sc-node-2 lrmd: info: upstart_job_running: oc-fw-agent is not running
Oct 29 13:10:06 [1025] sc-node-2 lrmd: info: process_lrmd_get_rsc_info: Resource 'oc-lb-agent' not found (3 active resources)
Oct 29 13:10:06 [1025] sc-node-2 lrmd: info: process_lrmd_rsc_register: Added 'oc-lb-agent' to the rsc list (4 active resources)
Oct 29 13:10:06 [1028] sc-node-2 crmd: info: do_lrm_rsc_op: Performing key=7:0:7:e130e80d-75e0-416f-ac5a-d318fcdd6ec2 op=oc-lb-agent_monitor_0
Oct 29 13:10:06 [1028] sc-node-2 crmd: notice: te_rsc_command: Initiating action 8: monitor oc-vpn-agent_monitor_0 on sc-node-2 (local)
Oct 29 13:10:06 [1025] sc-node-2 lrmd: info: get_first_instance: Result: (null)
Oct 29 13:10:06 [1025] sc-node-2 lrmd: info: upstart_job_running: oc-lb-agent is not running
Oct 29 13:10:06 [1025] sc-node-2 lrmd: info: process_lrmd_get_rsc_info: Resource 'oc-vpn-agent' not found (4 active resources)
Oct 29 13:10:06 [1025] sc-node-2 lrmd: info: process_lrmd_rsc_register: Added 'oc-vpn-agent' to the rsc list (5 active resources)
Oct 29 13:10:06 [1028] sc-node-2 crmd: info: do_lrm_rsc_op: Performing key=8:0:7:e130e80d-75e0-416f-ac5a-d318fcdd6ec2 op=oc-vpn-agent_monitor_0
Oct 29 13:10:06 [1025] sc-node-2 lrmd: info: get_first_instance: Result: (null)
Oct 29 13:10:06 [1025] sc-node-2 lrmd: info: upstart_job_running: oc-vpn-agent is not running
Oct 29 13:10:06 [1023] sc-node-2 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/23, version=0.11.3)
Oct 29 13:10:06 [1028] sc-node-2 crmd: notice: process_lrm_event: LRM operation oc-service-manager_monitor_0 (call=9, rc=7, cib-update=23, confirmed=true) not running
Oct 29 13:10:06 [1028] sc-node-2 crmd: notice: process_lrm_event: LRM operation oc-fw-agent_monitor_0 (call=13, rc=7, cib-update=24, confirmed=true) not running
Oct 29 13:10:06 [1023] sc-node-2 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/24, version=0.11.4)
Oct 29 13:10:06 [1028] sc-node-2 crmd: notice: process_lrm_event: LRM operation oc-lb-agent_monitor_0 (call=17, rc=7, cib-update=25, confirmed=true) not running
Oct 29 13:10:06 [1023] sc-node-2 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/25, version=0.11.5)
Oct 29 13:10:06 [1028] sc-node-2 crmd: notice: process_lrm_event: LRM operation oc-vpn-agent_monitor_0 (call=21, rc=7, cib-update=26, confirmed=true) not running
Oct 29 13:10:06 [1028] sc-node-2 crmd: info: match_graph_event: Action oc-service-manager_monitor_0 (5) confirmed on sc-node-2 (rc=0)
Oct 29 13:10:06 [1028] sc-node-2 crmd: info: match_graph_event: Action oc-fw-agent_monitor_0 (6) confirmed on sc-node-2 (rc=0)
Oct 29 13:10:06 [1028] sc-node-2 crmd: info: match_graph_event: Action oc-lb-agent_monitor_0 (7) confirmed on sc-node-2 (rc=0)
Oct 29 13:10:06 [1023] sc-node-2 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=local/crmd/26, version=0.11.6)
Oct 29 13:10:07 [1023] sc-node-2 cib: info: crm_client_new: Connecting 0x7f890dc5c0b0 for uid=0 gid=0 pid=1360 id=bb7daa59-f0d5-44ea-99a1-360169ea9b78
Oct 29 13:10:07 [1023] sc-node-2 cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crm_mon/2, version=0.11.6)
Oct 29 13:10:07 [1023] sc-node-2 cib: info: crm_client_destroy: Destroying 0 events
Oct 29 13:10:09 [1023] sc-node-2 cib: info: crm_client_new: Connecting 0x7f890dc5c0b0 for uid=0 gid=0 pid=1363 id=2da2d19d-b56d-405c-a936-c287aec2cb24
Oct 29 13:10:09 [1023] sc-node-2 cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crm_mon/2, version=0.11.6)
Oct 29 13:10:09 [1023] sc-node-2 cib: info: crm_client_destroy: Destroying 0 events
Oct 29 13:10:11 [1023] sc-node-2 cib: info: crm_client_new: Connecting 0x7f890dc5c0b0 for uid=0 gid=0 pid=1366 id=8e5d0619-1d42-47a2-93db-8d2f63476f85
Oct 29 13:10:11 [1023] sc-node-2 cib: info: cib_process_request: Completed cib_query operation for section 'all': OK (rc=0, origin=local/crm_mon/2, version=0.11.6)
More information about the Users
mailing list