[ClusterLabs] Can't See Why This Cluster Failed Over
Eric Robinson
eric.robinson at psmnv.com
Fri Apr 7 13:39:17 EDT 2017
Somebody want to look at this log and tell me why the cluster failed over? All we did was add a new resource. We've done it many times before without any problems.
--
Apr 03 08:50:30 [22762] ha14a cib: info: cib_process_request: Forwarding cib_apply_diff operation for section 'all' to master (origin=local/cibadmin/2)
Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.605.2 2
Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.0 65654c97e62cd549f22f777a5290fe3a
Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: + /cib: @epoch=607, @num_updates=0
Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ /cib/configuration/resources: <primitive id="p_mysql_745" class="lsb" type="mysql_745"/>
Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ /cib/configuration/resources: <primitive id="p_mysql_746" class="lsb" type="mysql_746"/>
Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ /cib/configuration/constraints/rsc_colocation[@id='c_clust19']/resource_set[@id='c_clust19-0']: <resource_ref id="p_mysql_745"/>
Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ /cib/configuration/constraints/rsc_colocation[@id='c_clust19']/resource_set[@id='c_clust19-0']: <resource_ref id="p_mysql_746"/>
Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ /cib/configuration/constraints/rsc_order[@id='o_clust19']/resource_set[@id='o_clust19-3']: <resource_ref id="p_mysql_745"/>
Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ /cib/configuration/constraints/rsc_order[@id='o_clust19']/resource_set[@id='o_clust19-3']: <resource_ref id="p_mysql_746"/>
Apr 03 08:50:30 [22762] ha14a cib: info: cib_process_request: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=ha14a/cibadmin/2, version=0.607.0)
Apr 03 08:50:30 [22762] ha14a cib: info: write_cib_contents: Archived previous version as /var/lib/pacemaker/cib/cib-36.raw
Apr 03 08:50:30 [22762] ha14a cib: info: write_cib_contents: Wrote version 0.607.0 of the CIB to disk (digest: 1afdb9e480f870a095aa9e39719d29c4)
Apr 03 08:50:30 [22762] ha14a cib: info: retrieveCib: Reading cluster configuration from: /var/lib/pacemaker/cib/cib.DkIgSs (digest: /var/lib/pacemaker/cib/cib.hPwa66)
Apr 03 08:50:30 [22764] ha14a lrmd: info: process_lrmd_get_rsc_info: Resource 'p_mysql_745' not found (17 active resources)
Apr 03 08:50:30 [22764] ha14a lrmd: info: process_lrmd_rsc_register: Added 'p_mysql_745' to the rsc list (18 active resources)
Apr 03 08:50:30 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=10:7484:7:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_745_monitor_0
Apr 03 08:50:30 [22764] ha14a lrmd: info: process_lrmd_get_rsc_info: Resource 'p_mysql_746' not found (18 active resources)
Apr 03 08:50:30 [22764] ha14a lrmd: info: process_lrmd_rsc_register: Added 'p_mysql_746' to the rsc list (19 active resources)
Apr 03 08:50:30 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=11:7484:7:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_746_monitor_0
Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.0 2
Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.1 (null)
Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=1
Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources: <lrm_resource id="p_mysql_745" type="mysql_745" class="lsb"/>
Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_mysql_745_last_0" operation_key="p_mysql_745_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.9" transition-key="13:7484:7:91ef4b03-8769-47a1-a364-060569c46e52" transition-magic="0:7;13:7484:7:91ef4b03-8769-47a1-a364-060569c46e52" call-id="142" rc-code="7" op-status="0" interval="0" last-run="1491234630" las
Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ </lrm_resource>
Apr 03 08:50:30 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7665, version=0.607.1)
Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.1 2
Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.2 (null)
Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=2
Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources: <lrm_resource id="p_mysql_746" type="mysql_746" class="lsb"/>
Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_mysql_746_last_0" operation_key="p_mysql_746_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.9" transition-key="14:7484:7:91ef4b03-8769-47a1-a364-060569c46e52" transition-magic="0:7;14:7484:7:91ef4b03-8769-47a1-a364-060569c46e52" call-id="146" rc-code="7" op-status="0" interval="0" last-run="1491234630" las
Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ </lrm_resource>
Apr 03 08:50:30 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7666, version=0.607.2)
Apr 03 08:50:30 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_745_monitor_0: not running (node=ha14a, call=142, rc=7, cib-update=88, confirmed=true)
Apr 03 08:50:30 [22767] ha14a crmd: notice: process_lrm_event: ha14a-p_mysql_745_monitor_0:142 [ not started\n ]
Apr 03 08:50:30 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/88)
Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.2 2
Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.3 (null)
Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=3
Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources: <lrm_resource id="p_mysql_745" type="mysql_745" class="lsb"/>
Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_mysql_745_last_0" operation_key="p_mysql_745_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.9" transition-key="10:7484:7:91ef4b03-8769-47a1-a364-060569c46e52" transition-magic="0:7;10:7484:7:91ef4b03-8769-47a1-a364-060569c46e52" call-id="142" rc-code="7" op-status="0" interval="0" last-run="1491234630" las
Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ </lrm_resource>
Apr 03 08:50:30 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/88, version=0.607.3)
Apr 03 08:50:30 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_746_monitor_0: not running (node=ha14a, call=146, rc=7, cib-update=89, confirmed=true)
Apr 03 08:50:30 [22767] ha14a crmd: notice: process_lrm_event: ha14a-p_mysql_746_monitor_0:146 [ not started\n ]
Apr 03 08:50:30 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/89)
Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.3 2
Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.4 (null)
Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=4
Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources: <lrm_resource id="p_mysql_746" type="mysql_746" class="lsb"/>
Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_mysql_746_last_0" operation_key="p_mysql_746_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.9" transition-key="11:7484:7:91ef4b03-8769-47a1-a364-060569c46e52" transition-magic="0:7;11:7484:7:91ef4b03-8769-47a1-a364-060569c46e52" call-id="146" rc-code="7" op-status="0" interval="0" last-run="1491234630" las
Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ </lrm_resource>
Apr 03 08:50:30 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/89, version=0.607.4)
Apr 03 08:50:30 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=105:7484:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_745_start_0
Apr 03 08:50:30 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_mysql_745 action:start call_id:147
Apr 03 08:50:30 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=106:7484:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_746_start_0
Apr 03 08:50:30 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_mysql_746 action:start call_id:148
Apr 03 08:50:31 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_mysql_745 action:start call_id:147 pid:24627 exit-code:0 exec-time:1064ms queue-time:0ms
Apr 03 08:50:31 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_745_start_0: ok (node=ha14a, call=147, rc=0, cib-update=90, confirmed=true)
Apr 03 08:50:31 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/90)
Apr 03 08:50:31 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.4 2
Apr 03 08:50:31 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.5 (null)
Apr 03 08:50:31 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=5
Apr 03 08:50:31 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_mysql_745']/lrm_rsc_op[@id='p_mysql_745_last_0']: @operation_key=p_mysql_745_start_0, @operation=start, @transition-key=105:7484:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;105:7484:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=147, @rc-code=0, @exec-time=1064
Apr 03 08:50:31 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/90, version=0.607.5)
Apr 03 08:50:36 [22762] ha14a cib: info: cib_process_ping: Reporting our current digest to ha14b: a8fdd49311821ec2fcc7be040e4b02b0 for 0.607.5 (0x21e09d0 0)
Apr 03 08:50:40 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_mysql_746 action:start call_id:148 pid:24628 exit-code:1 exec-time:10064ms queue-time:0ms
Apr 03 08:50:40 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_746_start_0: unknown error (node=ha14a, call=148, rc=1, cib-update=91, confirmed=true)
Apr 03 08:50:40 [22767] ha14a crmd: notice: process_lrm_event: ha14a-p_mysql_746_start_0:148 [ starting mysqld... \ngave up waiting\n ]
Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/91)
Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.5 2
Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.6 (null)
Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=6
Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_mysql_746']/lrm_rsc_op[@id='p_mysql_746_last_0']: @operation_key=p_mysql_746_start_0, @operation=start, @transition-key=106:7484:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:1;106:7484:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=148, @rc-code=1, @exec-time=10064
Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: ++ /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_mysql_746']: <lrm_rsc_op id="p_mysql_746_last_failure_0" operation_key="p_mysql_746_start_0" operation="start" crm-debug-origin="do_update_resource" crm_feature_set="3.0.9" transition-key="106:7484:0:91ef4b03-8769-47a1-a364-060569c46e52" transition-magic="0:1;106:7484:0:91ef4b03-8769-47a1-a364-060569c46e52" call-id="148" rc-code="1" op-status="0" in
Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/91, version=0.607.6)
Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_cs_dispatch: Update relayed from ha14b
Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-p_mysql_746 (INFINITY)
Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_perform_update: Sent update 83: fail-count-p_mysql_746=INFINITY
Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/83)
Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_cs_dispatch: Update relayed from ha14b
Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-p_mysql_746 (1491234640)
Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_perform_update: Sent update 85: last-failure-p_mysql_746=1491234640
Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/85)
Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_cs_dispatch: Update relayed from ha14b
Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-p_mysql_746 (INFINITY)
Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.6 2
Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.7 (null)
Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=7
Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: ++ /cib/status/node_state[@id='ha14a']/transient_attributes[@id='ha14a']/instance_attributes[@id='status-ha14a']: <nvpair id="status-ha14a-fail-count-p_mysql_746" name="fail-count-p_mysql_746" value="INFINITY"/>
Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/83, version=0.607.7)
Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_perform_update: Sent update 87: fail-count-p_mysql_746=INFINITY
Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/87)
Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_cs_dispatch: Update relayed from ha14b
Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-p_mysql_746 (1491234640)
Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.7 2
Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.8 (null)
Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=8
Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: ++ /cib/status/node_state[@id='ha14a']/transient_attributes[@id='ha14a']/instance_attributes[@id='status-ha14a']: <nvpair id="status-ha14a-last-failure-p_mysql_746" name="last-failure-p_mysql_746" value="1491234640"/>
Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/85, version=0.607.8)
Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_perform_update: Sent update 89: last-failure-p_mysql_746=1491234640
Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/89)
Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_cs_dispatch: Update relayed from ha14b
Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-p_mysql_746 (INFINITY)
Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_perform_update: Sent update 91: fail-count-p_mysql_746=INFINITY
Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_cs_dispatch: Update relayed from ha14b
Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-p_mysql_746 (1491234640)
Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/87, version=0.607.8)
Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/91)
Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_perform_update: Sent update 93: last-failure-p_mysql_746=1491234640
Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/93)
Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_cs_dispatch: Update relayed from ha14b
Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-p_mysql_746 (INFINITY)
Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_perform_update: Sent update 95: fail-count-p_mysql_746=INFINITY
Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_cs_dispatch: Update relayed from ha14b
Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/89, version=0.607.8)
Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-p_mysql_746 (1491234640)
Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/91, version=0.607.8)
Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/93, version=0.607.8)
Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/95)
Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_perform_update: Sent update 97: last-failure-p_mysql_746=1491234640
Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/95, version=0.607.8)
Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/97)
Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/97, version=0.607.8)
Apr 03 08:50:40 [22764] ha14a lrmd: info: cancel_recurring_action: Cancelling operation p_drbd0_monitor_30000
Apr 03 08:50:40 [22767] ha14a crmd: info: process_lrm_event: Operation p_drbd0_monitor_30000: Cancelled (node=ha14a, call=92, confirmed=true)
Apr 03 08:50:40 [22767] ha14a crmd: notice: process_lrm_event: ha14a-p_drbd0_monitor_30000:92 [ \n ]
Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Forwarding cib_delete operation for section status to master (origin=local/crmd/92)
Apr 03 08:50:40 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=87:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_057_stop_0
Apr 03 08:50:40 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_mysql_057 action:stop call_id:150
Apr 03 08:50:40 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=89:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_092_stop_0
Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.8 2
Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.9 (null)
Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: -- /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_drbd0']/lrm_rsc_op[@id='p_drbd0_monitor_30000']
Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=9
Apr 03 08:50:40 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_mysql_092 action:stop call_id:151
Apr 03 08:50:40 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=91:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_187_stop_0
Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Completed cib_delete operation for section status: OK (rc=0, origin=ha14a/crmd/92, version=0.607.9)
Apr 03 08:50:40 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_mysql_187 action:stop call_id:152
Apr 03 08:50:40 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=93:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_213_stop_0
Apr 03 08:50:40 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_mysql_213 action:stop call_id:153
Apr 03 08:50:40 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=95:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_702_stop_0
Apr 03 08:50:40 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_mysql_702 action:stop call_id:154
Apr 03 08:50:40 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=97:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_743_stop_0
Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.9 2
Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.10 (null)
Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: -- /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_drbd0']/lrm_rsc_op[@id='p_drbd0_monitor_31000']
Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=10
Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Completed cib_delete operation for section status: OK (rc=0, origin=ha14b/crmd/7670, version=0.607.10)
Apr 03 08:50:40 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_mysql_743 action:stop call_id:155
Apr 03 08:50:40 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=99:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_312_stop_0
Apr 03 08:50:40 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_mysql_312 action:stop call_id:156
Apr 03 08:50:40 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=101:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_289_stop_0
Apr 03 08:50:40 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_mysql_289 action:stop call_id:157
Apr 03 08:50:40 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=103:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_250_stop_0
Apr 03 08:50:40 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_mysql_250 action:stop call_id:158
Apr 03 08:50:40 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=105:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_745_stop_0
Apr 03 08:50:40 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_mysql_745 action:stop call_id:159
Apr 03 08:50:40 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=5:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_746_stop_0
Apr 03 08:50:40 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_mysql_746 action:stop call_id:160
Apr 03 08:50:40 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=123:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_drbd0_notify_0
Apr 03 08:50:40 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_drbd0 action:notify call_id:161
Apr 03 08:50:40 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_drbd0 action:notify call_id:161 pid:25907 exit-code:0 exec-time:25ms queue-time:0ms
Apr 03 08:50:40 [22767] ha14a crmd: notice: process_lrm_event: Operation p_drbd0_notify_0: ok (node=ha14a, call=161, rc=0, cib-update=0, confirmed=true)
Apr 03 08:50:40 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_mysql_746 action:stop call_id:160 pid:25792 exit-code:0 exec-time:41ms queue-time:0ms
Apr 03 08:50:40 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_746_stop_0: ok (node=ha14a, call=160, rc=0, cib-update=93, confirmed=true)
Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/93)
Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.10 2
Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.11 (null)
Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=11
Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_mysql_746']/lrm_rsc_op[@id='p_mysql_746_last_0']: @operation_key=p_mysql_746_stop_0, @operation=stop, @transition-key=5:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;5:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=160, @rc-code=0, @last-run=1491234640, @last-rc-change=1491234640, @exec-time=41
Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/93, version=0.607.11)
Apr 03 08:50:42 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_mysql_743 action:stop call_id:155 pid:25787 exit-code:0 exec-time:2105ms queue-time:0ms
Apr 03 08:50:42 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_743_stop_0: ok (node=ha14a, call=155, rc=0, cib-update=94, confirmed=true)
Apr 03 08:50:42 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/94)
Apr 03 08:50:42 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.11 2
Apr 03 08:50:42 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.12 (null)
Apr 03 08:50:42 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=12
Apr 03 08:50:42 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_mysql_743']/lrm_rsc_op[@id='p_mysql_743_last_0']: @operation_key=p_mysql_743_stop_0, @operation=stop, @transition-key=97:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;97:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=155, @last-run=1491234640, @last-rc-change=1491234640, @exec-time=2105
Apr 03 08:50:42 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/94, version=0.607.12)
Apr 03 08:50:43 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_mysql_745 action:stop call_id:159 pid:25791 exit-code:0 exec-time:3107ms queue-time:0ms
Apr 03 08:50:43 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_745_stop_0: ok (node=ha14a, call=159, rc=0, cib-update=95, confirmed=true)
Apr 03 08:50:43 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/95)
Apr 03 08:50:43 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.12 2
Apr 03 08:50:43 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.13 (null)
Apr 03 08:50:43 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=13
Apr 03 08:50:43 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_mysql_745']/lrm_rsc_op[@id='p_mysql_745_last_0']: @operation_key=p_mysql_745_stop_0, @operation=stop, @transition-key=105:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;105:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=159, @last-run=1491234640, @last-rc-change=1491234640, @exec-time=3107
Apr 03 08:50:43 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/95, version=0.607.13)
Apr 03 08:50:44 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_mysql_213 action:stop call_id:153 pid:25785 exit-code:0 exec-time:4109ms queue-time:0ms
Apr 03 08:50:44 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_mysql_312 action:stop call_id:156 pid:25788 exit-code:0 exec-time:4108ms queue-time:0ms
Apr 03 08:50:44 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_mysql_289 action:stop call_id:157 pid:25789 exit-code:0 exec-time:4108ms queue-time:0ms
Apr 03 08:50:44 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_mysql_092 action:stop call_id:151 pid:25783 exit-code:0 exec-time:4110ms queue-time:0ms
Apr 03 08:50:44 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_213_stop_0: ok (node=ha14a, call=153, rc=0, cib-update=96, confirmed=true)
Apr 03 08:50:44 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/96)
Apr 03 08:50:44 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_312_stop_0: ok (node=ha14a, call=156, rc=0, cib-update=97, confirmed=true)
Apr 03 08:50:44 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/97)
Apr 03 08:50:44 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_289_stop_0: ok (node=ha14a, call=157, rc=0, cib-update=98, confirmed=true)
Apr 03 08:50:44 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/98)
Apr 03 08:50:44 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_092_stop_0: ok (node=ha14a, call=151, rc=0, cib-update=99, confirmed=true)
Apr 03 08:50:44 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/99)
Apr 03 08:50:44 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_mysql_057 action:stop call_id:150 pid:25782 exit-code:0 exec-time:4111ms queue-time:0ms
Apr 03 08:50:44 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_mysql_187 action:stop call_id:152 pid:25784 exit-code:0 exec-time:4110ms queue-time:0ms
Apr 03 08:50:44 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_mysql_702 action:stop call_id:154 pid:25786 exit-code:0 exec-time:4110ms queue-time:0ms
Apr 03 08:50:44 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_057_stop_0: ok (node=ha14a, call=150, rc=0, cib-update=100, confirmed=true)
Apr 03 08:50:44 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_187_stop_0: ok (node=ha14a, call=152, rc=0, cib-update=101, confirmed=true)
Apr 03 08:50:44 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_702_stop_0: ok (node=ha14a, call=154, rc=0, cib-update=102, confirmed=true)
Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.13 2
Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.14 (null)
Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=14
Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_mysql_213']/lrm_rsc_op[@id='p_mysql_213_last_0']: @operation_key=p_mysql_213_stop_0, @operation=stop, @transition-key=93:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;93:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=153, @last-run=1491234640, @last-rc-change=1491234640, @exec-time=4109
Apr 03 08:50:44 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/96, version=0.607.14)
Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.14 2
Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.15 (null)
Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=15
Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_mysql_312']/lrm_rsc_op[@id='p_mysql_312_last_0']: @operation_key=p_mysql_312_stop_0, @operation=stop, @transition-key=99:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;99:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=156, @last-run=1491234640, @last-rc-change=1491234640, @exec-time=4108
Apr 03 08:50:44 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/97, version=0.607.15)
Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.15 2
Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.16 (null)
Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=16
Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_mysql_289']/lrm_rsc_op[@id='p_mysql_289_last_0']: @operation_key=p_mysql_289_stop_0, @operation=stop, @transition-key=101:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;101:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=157, @last-run=1491234640, @last-rc-change=1491234640, @exec-time=4108
Apr 03 08:50:44 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/98, version=0.607.16)
Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.16 2
Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.17 (null)
Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=17
Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_mysql_092']/lrm_rsc_op[@id='p_mysql_092_last_0']: @operation_key=p_mysql_092_stop_0, @operation=stop, @transition-key=89:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;89:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=151, @last-run=1491234640, @last-rc-change=1491234640, @exec-time=4110
Apr 03 08:50:44 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/99, version=0.607.17)
Apr 03 08:50:44 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/100)
Apr 03 08:50:44 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/101)
Apr 03 08:50:44 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/102)
Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.17 2
Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.18 (null)
Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=18
Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_mysql_057']/lrm_rsc_op[@id='p_mysql_057_last_0']: @operation_key=p_mysql_057_stop_0, @operation=stop, @transition-key=87:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;87:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=150, @last-run=1491234640, @last-rc-change=1491234640, @exec-time=4111
Apr 03 08:50:44 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/100, version=0.607.18)
Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.18 2
Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.19 (null)
Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=19
Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_mysql_187']/lrm_rsc_op[@id='p_mysql_187_last_0']: @operation_key=p_mysql_187_stop_0, @operation=stop, @transition-key=91:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;91:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=152, @last-run=1491234640, @last-rc-change=1491234640, @exec-time=4110
Apr 03 08:50:44 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/101, version=0.607.19)
Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.19 2
Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.20 (null)
Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=20
Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_mysql_702']/lrm_rsc_op[@id='p_mysql_702_last_0']: @operation_key=p_mysql_702_stop_0, @operation=stop, @transition-key=95:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;95:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=154, @last-run=1491234640, @last-rc-change=1491234640, @exec-time=4110
Apr 03 08:50:44 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/102, version=0.607.20)
Apr 03 08:50:45 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_mysql_250 action:stop call_id:158 pid:25790 exit-code:0 exec-time:5108ms queue-time:0ms
Apr 03 08:50:45 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_250_stop_0: ok (node=ha14a, call=158, rc=0, cib-update=103, confirmed=true)
Apr 03 08:50:45 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/103)
Apr 03 08:50:45 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.20 2
Apr 03 08:50:45 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.21 (null)
Apr 03 08:50:45 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=21
Apr 03 08:50:45 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_mysql_250']/lrm_rsc_op[@id='p_mysql_250_last_0']: @operation_key=p_mysql_250_stop_0, @operation=stop, @transition-key=103:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;103:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=158, @last-run=1491234640, @last-rc-change=1491234640, @exec-time=5108
Apr 03 08:50:45 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/103, version=0.607.21)
Apr 03 08:50:45 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=78:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_fs_clust19_stop_0
Apr 03 08:50:45 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_fs_clust19 action:stop call_id:162
Apr 03 08:50:45 [22764] ha14a lrmd: info: cancel_recurring_action: Cancelling operation p_vip_clust19_monitor_30000
Apr 03 08:50:45 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=82:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_vip_clust19_stop_0
Apr 03 08:50:45 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_vip_clust19 action:stop call_id:164
Apr 03 08:50:45 [22767] ha14a crmd: info: process_lrm_event: Operation p_vip_clust19_monitor_30000: Cancelled (node=ha14a, call=96, confirmed=true)
Apr 03 08:50:45 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_vip_clust19 action:stop call_id:164 pid:26174 exit-code:0 exec-time:44ms queue-time:0ms
Apr 03 08:50:45 [22767] ha14a crmd: notice: process_lrm_event: Operation p_vip_clust19_stop_0: ok (node=ha14a, call=164, rc=0, cib-update=104, confirmed=true)
Apr 03 08:50:45 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/104)
Apr 03 08:50:45 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.21 2
Apr 03 08:50:45 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.22 (null)
Apr 03 08:50:45 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=22
Apr 03 08:50:45 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_vip_clust19']/lrm_rsc_op[@id='p_vip_clust19_last_0']: @operation_key=p_vip_clust19_stop_0, @operation=stop, @transition-key=82:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;82:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=164, @last-run=1491234645, @last-rc-change=1491234645, @exec-time=44
Apr 03 08:50:45 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/104, version=0.607.22)
Apr 03 08:50:50 [22762] ha14a cib: info: cib_process_ping: Reporting our current digest to ha14b: bca4bc21567f887c56ae7ece4ac72814 for 0.607.22 (0x21e09d0 0)
Apr 03 08:50:58 [22764] ha14a lrmd: notice: operation_finished: p_fs_clust19_stop_0:26173:stderr [ umount: /ha01_mysql: device is busy. ]
Apr 03 08:50:58 [22764] ha14a lrmd: notice: operation_finished: p_fs_clust19_stop_0:26173:stderr [ (In some cases useful info about processes that use ]
Apr 03 08:50:58 [22764] ha14a lrmd: notice: operation_finished: p_fs_clust19_stop_0:26173:stderr [ the device is found by lsof(8) or fuser(1)) ]
Apr 03 08:50:58 [22764] ha14a lrmd: notice: operation_finished: p_fs_clust19_stop_0:26173:stderr [ umount: /ha01_mysql: device is busy. ]
Apr 03 08:50:58 [22764] ha14a lrmd: notice: operation_finished: p_fs_clust19_stop_0:26173:stderr [ (In some cases useful info about processes that use ]
Apr 03 08:50:58 [22764] ha14a lrmd: notice: operation_finished: p_fs_clust19_stop_0:26173:stderr [ the device is found by lsof(8) or fuser(1)) ]
Apr 03 08:50:58 [22764] ha14a lrmd: notice: operation_finished: p_fs_clust19_stop_0:26173:stderr [ umount: /ha01_mysql: device is busy. ]
Apr 03 08:50:58 [22764] ha14a lrmd: notice: operation_finished: p_fs_clust19_stop_0:26173:stderr [ (In some cases useful info about processes that use ]
Apr 03 08:50:58 [22764] ha14a lrmd: notice: operation_finished: p_fs_clust19_stop_0:26173:stderr [ the device is found by lsof(8) or fuser(1)) ]
Apr 03 08:50:58 [22764] ha14a lrmd: notice: operation_finished: p_fs_clust19_stop_0:26173:stderr [ umount: /ha01_mysql: device is busy. ]
Apr 03 08:50:58 [22764] ha14a lrmd: notice: operation_finished: p_fs_clust19_stop_0:26173:stderr [ (In some cases useful info about processes that use ]
Apr 03 08:50:58 [22764] ha14a lrmd: notice: operation_finished: p_fs_clust19_stop_0:26173:stderr [ the device is found by lsof(8) or fuser(1)) ]
Apr 03 08:50:58 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_fs_clust19 action:stop call_id:162 pid:26173 exit-code:0 exec-time:13216ms queue-time:0ms
Apr 03 08:50:58 [22767] ha14a crmd: notice: process_lrm_event: Operation p_fs_clust19_stop_0: ok (node=ha14a, call=162, rc=0, cib-update=105, confirmed=true)
Apr 03 08:50:58 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/105)
Apr 03 08:50:58 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.22 2
Apr 03 08:50:58 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.23 (null)
Apr 03 08:50:58 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=23
Apr 03 08:50:58 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_fs_clust19']/lrm_rsc_op[@id='p_fs_clust19_last_0']: @operation_key=p_fs_clust19_stop_0, @operation=stop, @transition-key=78:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;78:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=162, @last-run=1491234645, @last-rc-change=1491234645, @exec-time=13216
Apr 03 08:50:58 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/105, version=0.607.23)
Apr 03 08:50:58 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=74:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_lv_on_drbd0_stop_0
Apr 03 08:50:58 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_lv_on_drbd0 action:stop call_id:165
Apr 03 08:50:58 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_lv_on_drbd0 action:stop call_id:165 pid:26495 exit-code:0 exec-time:88ms queue-time:0ms
Apr 03 08:50:58 [22767] ha14a crmd: notice: process_lrm_event: Operation p_lv_on_drbd0_stop_0: ok (node=ha14a, call=165, rc=0, cib-update=106, confirmed=true)
Apr 03 08:50:58 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/106)
Apr 03 08:50:58 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.23 2
Apr 03 08:50:58 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.24 (null)
Apr 03 08:50:58 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=24
Apr 03 08:50:58 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_lv_on_drbd0']/lrm_rsc_op[@id='p_lv_on_drbd0_last_0']: @operation_key=p_lv_on_drbd0_stop_0, @operation=stop, @transition-key=74:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;74:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=165, @last-run=1491234658, @last-rc-change=1491234658, @exec-time=88
Apr 03 08:50:58 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/106, version=0.607.24)
Apr 03 08:50:58 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=13:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_drbd0_demote_0
Apr 03 08:50:58 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_drbd0 action:demote call_id:166
Apr 03 08:50:58 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_drbd0 action:demote call_id:166 pid:26533 exit-code:0 exec-time:33ms queue-time:0ms
Apr 03 08:50:58 [22767] ha14a crmd: notice: process_lrm_event: Operation p_drbd0_demote_0: ok (node=ha14a, call=166, rc=0, cib-update=107, confirmed=true)
Apr 03 08:50:58 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/107)
Apr 03 08:50:58 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.24 2
Apr 03 08:50:58 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.25 (null)
Apr 03 08:50:58 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=25
Apr 03 08:50:58 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_drbd0']/lrm_rsc_op[@id='p_drbd0_last_0']: @operation_key=p_drbd0_demote_0, @operation=demote, @transition-key=13:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;13:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=166, @rc-code=0, @last-run=1491234658, @last-rc-change=1491234658, @exec-time=33
Apr 03 08:50:58 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/107, version=0.607.25)
Apr 03 08:50:58 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=124:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_drbd0_notify_0
Apr 03 08:50:58 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_drbd0 action:notify call_id:167
Apr 03 08:50:58 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_drbd0 action:notify call_id:167 pid:26567 exit-code:0 exec-time:42ms queue-time:0ms
Apr 03 08:50:58 [22767] ha14a crmd: notice: process_lrm_event: Operation p_drbd0_notify_0: ok (node=ha14a, call=167, rc=0, cib-update=0, confirmed=true)
Apr 03 08:50:58 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=119:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_drbd0_notify_0
Apr 03 08:50:58 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_drbd0 action:notify call_id:168
Apr 03 08:50:58 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_drbd0 action:notify call_id:168 pid:26594 exit-code:0 exec-time:27ms queue-time:0ms
Apr 03 08:50:58 [22767] ha14a crmd: notice: process_lrm_event: Operation p_drbd0_notify_0: ok (node=ha14a, call=168, rc=0, cib-update=0, confirmed=true)
Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.25 2
Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.26 (null)
Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=26
Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_drbd0']/lrm_rsc_op[@id='p_drbd0_last_0']: @operation_key=p_drbd0_promote_0, @operation=promote, @transition-key=18:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;18:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=151, @last-run=1491234658, @last-rc-change=1491234658, @exec-time=31
Apr 03 08:50:59 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7671, version=0.607.26)
Apr 03 08:50:59 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=120:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_drbd0_notify_0
Apr 03 08:50:59 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_drbd0 action:notify call_id:169
Apr 03 08:50:59 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_drbd0 action:notify call_id:169 pid:26616 exit-code:0 exec-time:42ms queue-time:0ms
Apr 03 08:50:59 [22767] ha14a crmd: notice: process_lrm_event: Operation p_drbd0_notify_0: ok (node=ha14a, call=169, rc=0, cib-update=0, confirmed=true)
Apr 03 08:50:59 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=15:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_drbd0_monitor_31000
Apr 03 08:50:59 [22767] ha14a crmd: notice: process_lrm_event: Operation p_drbd0_monitor_31000: ok (node=ha14a, call=170, rc=0, cib-update=108, confirmed=false)
Apr 03 08:50:59 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/108)
Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.26 2
Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.27 (null)
Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=27
Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: ++ /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_drbd0']: <lrm_rsc_op id="p_drbd0_monitor_30000" operation_key="p_drbd0_monitor_30000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.9" transition-key="19:7486:8:91ef4b03-8769-47a1-a364-060569c46e52" transition-magic="0:8;19:7486:8:91ef4b03-8769-47a1-a364-060569c46e52" call-id="154" rc-code="8" op-status="0" interval=
Apr 03 08:50:59 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7672, version=0.607.27)
Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.27 2
Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.28 (null)
Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=28
Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: ++ /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_drbd0']: <lrm_rsc_op id="p_drbd0_monitor_31000" operation_key="p_drbd0_monitor_31000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.9" transition-key="15:7486:0:91ef4b03-8769-47a1-a364-060569c46e52" transition-magic="0:0;15:7486:0:91ef4b03-8769-47a1-a364-060569c46e52" call-id="170" rc-code="0" op-status="0" interval=
Apr 03 08:50:59 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/108, version=0.607.28)
Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.28 2
Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.29 (null)
Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=29
Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_lv_on_drbd0']/lrm_rsc_op[@id='p_lv_on_drbd0_last_0']: @operation_key=p_lv_on_drbd0_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=75:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;75:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=153, @rc-code=0, @last-run=1491234658, @last-rc-change=14912
Apr 03 08:50:59 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7673, version=0.607.29)
Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.29 2
Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.30 (null)
Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=30
Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_vip_clust19']/lrm_rsc_op[@id='p_vip_clust19_last_0']: @operation_key=p_vip_clust19_start_0, @operation=start, @transition-key=83:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;83:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=156, @last-run=1491234659, @last-rc-change=1491234659, @exec-time=52
Apr 03 08:50:59 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7674, version=0.607.30)
Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.30 2
Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.31 (null)
Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=31
Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: ++ /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_vip_clust19']: <lrm_rsc_op id="p_vip_clust19_monitor_30000" operation_key="p_vip_clust19_monitor_30000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.9" transition-key="84:7486:0:91ef4b03-8769-47a1-a364-060569c46e52" transition-magic="0:0;84:7486:0:91ef4b03-8769-47a1-a364-060569c46e52" call-id="157" rc-code="0" op-st
Apr 03 08:50:59 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7675, version=0.607.31)
Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.31 2
Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.32 (null)
Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=32
Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_fs_clust19']/lrm_rsc_op[@id='p_fs_clust19_last_0']: @operation_key=p_fs_clust19_start_0, @operation=start, @transition-key=79:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;79:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=155, @rc-code=0, @last-run=1491234659, @last-rc-change=1491234659, @exec-time=123
Apr 03 08:50:59 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7676, version=0.607.32)
Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.32 2
Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.33 (null)
Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=33
Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_mysql_057']/lrm_rsc_op[@id='p_mysql_057_last_0']: @operation_key=p_mysql_057_start_0, @operation=start, @transition-key=88:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;88:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=158, @rc-code=0, @last-run=1491234659, @last-rc-change=1491234659, @exec-time=1050
Apr 03 08:51:00 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7677, version=0.607.33)
Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.33 2
Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.34 (null)
Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=34
Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_mysql_702']/lrm_rsc_op[@id='p_mysql_702_last_0']: @operation_key=p_mysql_702_start_0, @operation=start, @transition-key=96:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;96:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=162, @rc-code=0, @last-run=1491234659, @last-rc-change=1491234659, @exec-time=1051
Apr 03 08:51:00 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7678, version=0.607.34)
Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.34 2
Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.35 (null)
Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=35
Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_mysql_250']/lrm_rsc_op[@id='p_mysql_250_last_0']: @operation_key=p_mysql_250_start_0, @operation=start, @transition-key=104:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;104:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=166, @rc-code=0, @last-run=1491234659, @last-rc-change=1491234659, @exec-time=1050
Apr 03 08:51:00 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7679, version=0.607.35)
Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.35 2
Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.36 (null)
Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=36
Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_mysql_092']/lrm_rsc_op[@id='p_mysql_092_last_0']: @operation_key=p_mysql_092_start_0, @operation=start, @transition-key=90:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;90:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=159, @rc-code=0, @last-run=1491234659, @last-rc-change=1491234659, @exec-time=1053
Apr 03 08:51:00 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7680, version=0.607.36)
Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.36 2
Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.37 (null)
Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=37
Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_mysql_745']/lrm_rsc_op[@id='p_mysql_745_last_0']: @operation_key=p_mysql_745_start_0, @operation=start, @transition-key=106:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;106:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=167, @rc-code=0, @last-run=1491234659, @last-rc-change=1491234659, @exec-time=1052
Apr 03 08:51:00 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7681, version=0.607.37)
Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.37 2
Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.38 (null)
Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=38
Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_mysql_312']/lrm_rsc_op[@id='p_mysql_312_last_0']: @operation_key=p_mysql_312_start_0, @operation=start, @transition-key=100:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;100:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=164, @rc-code=0, @last-run=1491234659, @last-rc-change=1491234659, @exec-time=1053
Apr 03 08:51:00 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7682, version=0.607.38)
Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.38 2
Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.39 (null)
Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=39
Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_mysql_289']/lrm_rsc_op[@id='p_mysql_289_last_0']: @operation_key=p_mysql_289_start_0, @operation=start, @transition-key=102:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;102:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=165, @rc-code=0, @last-run=1491234659, @last-rc-change=1491234659, @exec-time=1055
Apr 03 08:51:00 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7683, version=0.607.39)
Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.39 2
Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.40 (null)
Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=40
Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_mysql_187']/lrm_rsc_op[@id='p_mysql_187_last_0']: @operation_key=p_mysql_187_start_0, @operation=start, @transition-key=92:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;92:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=160, @rc-code=0, @last-run=1491234659, @last-rc-change=1491234659, @exec-time=1056
Apr 03 08:51:00 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7684, version=0.607.40)
Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.40 2
Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.41 (null)
Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=41
Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_mysql_743']/lrm_rsc_op[@id='p_mysql_743_last_0']: @operation_key=p_mysql_743_start_0, @operation=start, @transition-key=98:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;98:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=163, @rc-code=0, @last-run=1491234659, @last-rc-change=1491234659, @exec-time=1055
Apr 03 08:51:00 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7685, version=0.607.41)
Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.41 2
Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.42 (null)
Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=42
Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_mysql_213']/lrm_rsc_op[@id='p_mysql_213_last_0']: @operation_key=p_mysql_213_start_0, @operation=start, @transition-key=94:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;94:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=161, @rc-code=0, @last-run=1491234659, @last-rc-change=1491234659, @exec-time=1063
Apr 03 08:51:00 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7686, version=0.607.42)
Apr 03 08:51:05 [22762] ha14a cib: info: cib_process_ping: Reporting our current digest to ha14b: 3b1748e1e0487004594b1da9d8e47da6 for 0.607.42 (0x21e09d0 0)
Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.42 2
Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.43 (null)
Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=43
Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_mysql_746']/lrm_rsc_op[@id='p_mysql_746_last_0']: @operation_key=p_mysql_746_start_0, @operation=start, @transition-key=107:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:1;107:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=168, @rc-code=1, @last-run=1491234659, @last-rc-change=1491234659, @exec-time=10060
Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: ++ /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_mysql_746']: <lrm_rsc_op id="p_mysql_746_last_failure_0" operation_key="p_mysql_746_start_0" operation="start" crm-debug-origin="do_update_resource" crm_feature_set="3.0.9" transition-key="107:7486:0:91ef4b03-8769-47a1-a364-060569c46e52" transition-magic="0:1;107:7486:0:91ef4b03-8769-47a1-a364-060569c46e52" call-id="168" rc-code="1" op-status="0" in
Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7687, version=0.607.43)
Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/99)
Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.43 2
Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.44 (null)
Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=44
Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: ++ /cib/status/node_state[@id='ha14b']/transient_attributes[@id='ha14b']/instance_attributes[@id='status-ha14b']: <nvpair id="status-ha14b-fail-count-p_mysql_746" name="fail-count-p_mysql_746" value="INFINITY"/>
Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/attrd/163, version=0.607.44)
Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.44 2
Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.45 (null)
Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=45
Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: ++ /cib/status/node_state[@id='ha14b']/transient_attributes[@id='ha14b']/instance_attributes[@id='status-ha14b']: <nvpair id="status-ha14b-last-failure-p_mysql_746" name="last-failure-p_mysql_746" value="1491234669"/>
Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/attrd/165, version=0.607.45)
Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/attrd/167, version=0.607.45)
Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/attrd/169, version=0.607.45)
Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/attrd/171, version=0.607.45)
Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/attrd/173, version=0.607.45)
Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/attrd/175, version=0.607.45)
Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/99, version=0.607.45)
Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/attrd/177, version=0.607.45)
Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/101)
Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/103)
Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/105)
Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/101, version=0.607.45)
Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/103, version=0.607.45)
Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/107)
Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/105, version=0.607.45)
Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/109)
Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/107, version=0.607.45)
Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/109, version=0.607.45)
Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/111)
Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/111, version=0.607.45)
Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/113)
Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/113, version=0.607.45)
Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.45 2
Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.46 (null)
Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=46
Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_mysql_746']/lrm_rsc_op[@id='p_mysql_746_last_0']: @operation_key=p_mysql_746_stop_0, @operation=stop, @transition-key=7:7488:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;7:7488:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=169, @rc-code=0, @last-run=1491234669, @last-rc-change=1491234669, @exec-time=24
Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7691, version=0.607.46)
Apr 03 08:51:14 [22762] ha14a cib: info: cib_process_ping: Reporting our current digest to ha14b: 5384604825dd17358b76d9b84f2be021 for 0.607.46 (0x21e09d0 0)
--
Eric Robinson
More information about the Users
mailing list