[ClusterLabs] Can't See Why This Cluster Failed Over
Ken Gaillot
kgaillot at redhat.com
Fri Apr 7 21:34:18 CEST 2017
On 04/07/2017 12:58 PM, Eric Robinson wrote:
> Somebody want to look at this log and tell me why the cluster failed over? All we did was add a new resource. We've done it many times before without any problems.
>
> --
>
> Apr 03 08:50:30 [22762] ha14a cib: info: cib_process_request: Forwarding cib_apply_diff operation for section 'all' to master (origin=local/cibadmin/2)
> Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.605.2 2
> Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.0 65654c97e62cd549f22f777a5290fe3a
> Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: + /cib: @epoch=607, @num_updates=0
> Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ /cib/configuration/resources: <primitive id="p_mysql_745" class="lsb" type="mysql_745"/>
> Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ /cib/configuration/resources: <primitive id="p_mysql_746" class="lsb" type="mysql_746"/>
> Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ /cib/configuration/constraints/rsc_colocation[@id='c_clust19']/resource_set[@id='c_clust19-0']: <resource_ref id="p_mysql_745"/>
> Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ /cib/configuration/constraints/rsc_colocation[@id='c_clust19']/resource_set[@id='c_clust19-0']: <resource_ref id="p_mysql_746"/>
> Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ /cib/configuration/constraints/rsc_order[@id='o_clust19']/resource_set[@id='o_clust19-3']: <resource_ref id="p_mysql_745"/>
> Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ /cib/configuration/constraints/rsc_order[@id='o_clust19']/resource_set[@id='o_clust19-3']: <resource_ref id="p_mysql_746"/>
> Apr 03 08:50:30 [22762] ha14a cib: info: cib_process_request: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=ha14a/cibadmin/2, version=0.607.0)
> Apr 03 08:50:30 [22762] ha14a cib: info: write_cib_contents: Archived previous version as /var/lib/pacemaker/cib/cib-36.raw
> Apr 03 08:50:30 [22762] ha14a cib: info: write_cib_contents: Wrote version 0.607.0 of the CIB to disk (digest: 1afdb9e480f870a095aa9e39719d29c4)
> Apr 03 08:50:30 [22762] ha14a cib: info: retrieveCib: Reading cluster configuration from: /var/lib/pacemaker/cib/cib.DkIgSs (digest: /var/lib/pacemaker/cib/cib.hPwa66)
> Apr 03 08:50:30 [22764] ha14a lrmd: info: process_lrmd_get_rsc_info: Resource 'p_mysql_745' not found (17 active resources)
> Apr 03 08:50:30 [22764] ha14a lrmd: info: process_lrmd_rsc_register: Added 'p_mysql_745' to the rsc list (18 active resources)
> Apr 03 08:50:30 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=10:7484:7:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_745_monitor_0
> Apr 03 08:50:30 [22764] ha14a lrmd: info: process_lrmd_get_rsc_info: Resource 'p_mysql_746' not found (18 active resources)
> Apr 03 08:50:30 [22764] ha14a lrmd: info: process_lrmd_rsc_register: Added 'p_mysql_746' to the rsc list (19 active resources)
> Apr 03 08:50:30 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=11:7484:7:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_746_monitor_0
> Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.0 2
> Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.1 (null)
> Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=1
> Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources: <lrm_resource id="p_mysql_745" type="mysql_745" class="lsb"/>
> Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_mysql_745_last_0" operation_key="p_mysql_745_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.9" transition-key="13:7484:7:91ef4b03-8769-47a1-a364-060569c46e52" transition-magic="0:7;13:7484:7:91ef4b03-8769-47a1-a364-060569c46e52" call-id="142" rc-code="7" op-status="0" interval="0" last-run="1491234630" las
> Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ </lrm_resource>
> Apr 03 08:50:30 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7665, version=0.607.1)
> Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.1 2
> Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.2 (null)
> Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=2
> Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources: <lrm_resource id="p_mysql_746" type="mysql_746" class="lsb"/>
> Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_mysql_746_last_0" operation_key="p_mysql_746_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.9" transition-key="14:7484:7:91ef4b03-8769-47a1-a364-060569c46e52" transition-magic="0:7;14:7484:7:91ef4b03-8769-47a1-a364-060569c46e52" call-id="146" rc-code="7" op-status="0" interval="0" last-run="1491234630" las
> Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ </lrm_resource>
> Apr 03 08:50:30 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7666, version=0.607.2)
> Apr 03 08:50:30 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_745_monitor_0: not running (node=ha14a, call=142, rc=7, cib-update=88, confirmed=true)
> Apr 03 08:50:30 [22767] ha14a crmd: notice: process_lrm_event: ha14a-p_mysql_745_monitor_0:142 [ not started\n ]
> Apr 03 08:50:30 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/88)
> Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.2 2
> Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.3 (null)
> Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=3
> Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources: <lrm_resource id="p_mysql_745" type="mysql_745" class="lsb"/>
> Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_mysql_745_last_0" operation_key="p_mysql_745_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.9" transition-key="10:7484:7:91ef4b03-8769-47a1-a364-060569c46e52" transition-magic="0:7;10:7484:7:91ef4b03-8769-47a1-a364-060569c46e52" call-id="142" rc-code="7" op-status="0" interval="0" last-run="1491234630" las
> Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ </lrm_resource>
> Apr 03 08:50:30 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/88, version=0.607.3)
> Apr 03 08:50:30 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_746_monitor_0: not running (node=ha14a, call=146, rc=7, cib-update=89, confirmed=true)
> Apr 03 08:50:30 [22767] ha14a crmd: notice: process_lrm_event: ha14a-p_mysql_746_monitor_0:146 [ not started\n ]
> Apr 03 08:50:30 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/89)
> Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.3 2
> Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.4 (null)
> Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=4
> Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources: <lrm_resource id="p_mysql_746" type="mysql_746" class="lsb"/>
> Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_mysql_746_last_0" operation_key="p_mysql_746_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.9" transition-key="11:7484:7:91ef4b03-8769-47a1-a364-060569c46e52" transition-magic="0:7;11:7484:7:91ef4b03-8769-47a1-a364-060569c46e52" call-id="146" rc-code="7" op-status="0" interval="0" last-run="1491234630" las
> Apr 03 08:50:30 [22762] ha14a cib: info: cib_perform_op: ++ </lrm_resource>
> Apr 03 08:50:30 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/89, version=0.607.4)
> Apr 03 08:50:30 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=105:7484:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_745_start_0
> Apr 03 08:50:30 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_mysql_745 action:start call_id:147
> Apr 03 08:50:30 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=106:7484:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_746_start_0
> Apr 03 08:50:30 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_mysql_746 action:start call_id:148
> Apr 03 08:50:31 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_mysql_745 action:start call_id:147 pid:24627 exit-code:0 exec-time:1064ms queue-time:0ms
> Apr 03 08:50:31 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_745_start_0: ok (node=ha14a, call=147, rc=0, cib-update=90, confirmed=true)
> Apr 03 08:50:31 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/90)
> Apr 03 08:50:31 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.4 2
> Apr 03 08:50:31 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.5 (null)
> Apr 03 08:50:31 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=5
> Apr 03 08:50:31 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_mysql_745']/lrm_rsc_op[@id='p_mysql_745_last_0']: @operation_key=p_mysql_745_start_0, @operation=start, @transition-key=105:7484:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;105:7484:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=147, @rc-code=0, @exec-time=1064
> Apr 03 08:50:31 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/90, version=0.607.5)
> Apr 03 08:50:36 [22762] ha14a cib: info: cib_process_ping: Reporting our current digest to ha14b: a8fdd49311821ec2fcc7be040e4b02b0 for 0.607.5 (0x21e09d0 0)
> Apr 03 08:50:40 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_mysql_746 action:start call_id:148 pid:24628 exit-code:1 exec-time:10064ms queue-time:0ms
> Apr 03 08:50:40 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_746_start_0: unknown error (node=ha14a, call=148, rc=1, cib-update=91, confirmed=true)
> Apr 03 08:50:40 [22767] ha14a crmd: notice: process_lrm_event: ha14a-p_mysql_746_start_0:148 [ starting mysqld... \ngave up waiting\n ]
This is where your trouble starts. p_mysql_746 fails to start; the init
script returned exit code 1, and showed the output in the above line.
I can't see the full configuration, but I'm guessing the colocation
added with the new resource means other resources now depend on it, so
they get stopped. Perhaps you got the sense of the colocation backwards,
i.e. you wanted the new resource to be placed with the old ones, rather
than vice versa?
The cluster status should show the failed start action.
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/91)
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.5 2
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.6 (null)
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=6
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_mysql_746']/lrm_rsc_op[@id='p_mysql_746_last_0']: @operation_key=p_mysql_746_start_0, @operation=start, @transition-key=106:7484:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:1;106:7484:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=148, @rc-code=1, @exec-time=10064
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: ++ /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_mysql_746']: <lrm_rsc_op id="p_mysql_746_last_failure_0" operation_key="p_mysql_746_start_0" operation="start" crm-debug-origin="do_update_resource" crm_feature_set="3.0.9" transition-key="106:7484:0:91ef4b03-8769-47a1-a364-060569c46e52" transition-magic="0:1;106:7484:0:91ef4b03-8769-47a1-a364-060569c46e52" call-id="148" rc-code="1" op-status="0" in
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/91, version=0.607.6)
> Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_cs_dispatch: Update relayed from ha14b
> Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-p_mysql_746 (INFINITY)
> Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_perform_update: Sent update 83: fail-count-p_mysql_746=INFINITY
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/83)
> Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_cs_dispatch: Update relayed from ha14b
> Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-p_mysql_746 (1491234640)
> Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_perform_update: Sent update 85: last-failure-p_mysql_746=1491234640
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/85)
> Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_cs_dispatch: Update relayed from ha14b
> Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-p_mysql_746 (INFINITY)
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.6 2
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.7 (null)
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=7
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: ++ /cib/status/node_state[@id='ha14a']/transient_attributes[@id='ha14a']/instance_attributes[@id='status-ha14a']: <nvpair id="status-ha14a-fail-count-p_mysql_746" name="fail-count-p_mysql_746" value="INFINITY"/>
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/83, version=0.607.7)
> Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_perform_update: Sent update 87: fail-count-p_mysql_746=INFINITY
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/87)
> Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_cs_dispatch: Update relayed from ha14b
> Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-p_mysql_746 (1491234640)
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.7 2
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.8 (null)
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=8
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: ++ /cib/status/node_state[@id='ha14a']/transient_attributes[@id='ha14a']/instance_attributes[@id='status-ha14a']: <nvpair id="status-ha14a-last-failure-p_mysql_746" name="last-failure-p_mysql_746" value="1491234640"/>
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/85, version=0.607.8)
> Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_perform_update: Sent update 89: last-failure-p_mysql_746=1491234640
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/89)
> Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_cs_dispatch: Update relayed from ha14b
> Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-p_mysql_746 (INFINITY)
> Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_perform_update: Sent update 91: fail-count-p_mysql_746=INFINITY
> Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_cs_dispatch: Update relayed from ha14b
> Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-p_mysql_746 (1491234640)
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/87, version=0.607.8)
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/91)
> Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_perform_update: Sent update 93: last-failure-p_mysql_746=1491234640
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/93)
> Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_cs_dispatch: Update relayed from ha14b
> Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-p_mysql_746 (INFINITY)
> Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_perform_update: Sent update 95: fail-count-p_mysql_746=INFINITY
> Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_cs_dispatch: Update relayed from ha14b
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/89, version=0.607.8)
> Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-p_mysql_746 (1491234640)
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/91, version=0.607.8)
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/93, version=0.607.8)
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/95)
> Apr 03 08:50:40 [22765] ha14a attrd: notice: attrd_perform_update: Sent update 97: last-failure-p_mysql_746=1491234640
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/95, version=0.607.8)
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/97)
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/97, version=0.607.8)
> Apr 03 08:50:40 [22764] ha14a lrmd: info: cancel_recurring_action: Cancelling operation p_drbd0_monitor_30000
> Apr 03 08:50:40 [22767] ha14a crmd: info: process_lrm_event: Operation p_drbd0_monitor_30000: Cancelled (node=ha14a, call=92, confirmed=true)
> Apr 03 08:50:40 [22767] ha14a crmd: notice: process_lrm_event: ha14a-p_drbd0_monitor_30000:92 [ \n ]
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Forwarding cib_delete operation for section status to master (origin=local/crmd/92)
> Apr 03 08:50:40 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=87:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_057_stop_0
> Apr 03 08:50:40 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_mysql_057 action:stop call_id:150
> Apr 03 08:50:40 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=89:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_092_stop_0
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.8 2
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.9 (null)
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: -- /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_drbd0']/lrm_rsc_op[@id='p_drbd0_monitor_30000']
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=9
> Apr 03 08:50:40 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_mysql_092 action:stop call_id:151
> Apr 03 08:50:40 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=91:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_187_stop_0
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Completed cib_delete operation for section status: OK (rc=0, origin=ha14a/crmd/92, version=0.607.9)
> Apr 03 08:50:40 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_mysql_187 action:stop call_id:152
> Apr 03 08:50:40 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=93:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_213_stop_0
> Apr 03 08:50:40 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_mysql_213 action:stop call_id:153
> Apr 03 08:50:40 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=95:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_702_stop_0
> Apr 03 08:50:40 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_mysql_702 action:stop call_id:154
> Apr 03 08:50:40 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=97:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_743_stop_0
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.9 2
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.10 (null)
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: -- /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_drbd0']/lrm_rsc_op[@id='p_drbd0_monitor_31000']
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=10
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Completed cib_delete operation for section status: OK (rc=0, origin=ha14b/crmd/7670, version=0.607.10)
> Apr 03 08:50:40 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_mysql_743 action:stop call_id:155
> Apr 03 08:50:40 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=99:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_312_stop_0
> Apr 03 08:50:40 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_mysql_312 action:stop call_id:156
> Apr 03 08:50:40 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=101:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_289_stop_0
> Apr 03 08:50:40 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_mysql_289 action:stop call_id:157
> Apr 03 08:50:40 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=103:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_250_stop_0
> Apr 03 08:50:40 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_mysql_250 action:stop call_id:158
> Apr 03 08:50:40 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=105:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_745_stop_0
> Apr 03 08:50:40 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_mysql_745 action:stop call_id:159
> Apr 03 08:50:40 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=5:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_mysql_746_stop_0
> Apr 03 08:50:40 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_mysql_746 action:stop call_id:160
> Apr 03 08:50:40 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=123:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_drbd0_notify_0
> Apr 03 08:50:40 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_drbd0 action:notify call_id:161
> Apr 03 08:50:40 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_drbd0 action:notify call_id:161 pid:25907 exit-code:0 exec-time:25ms queue-time:0ms
> Apr 03 08:50:40 [22767] ha14a crmd: notice: process_lrm_event: Operation p_drbd0_notify_0: ok (node=ha14a, call=161, rc=0, cib-update=0, confirmed=true)
> Apr 03 08:50:40 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_mysql_746 action:stop call_id:160 pid:25792 exit-code:0 exec-time:41ms queue-time:0ms
> Apr 03 08:50:40 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_746_stop_0: ok (node=ha14a, call=160, rc=0, cib-update=93, confirmed=true)
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/93)
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.10 2
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.11 (null)
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=11
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_mysql_746']/lrm_rsc_op[@id='p_mysql_746_last_0']: @operation_key=p_mysql_746_stop_0, @operation=stop, @transition-key=5:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;5:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=160, @rc-code=0, @last-run=1491234640, @last-rc-change=1491234640, @exec-time=41
> Apr 03 08:50:40 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/93, version=0.607.11)
> Apr 03 08:50:42 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_mysql_743 action:stop call_id:155 pid:25787 exit-code:0 exec-time:2105ms queue-time:0ms
> Apr 03 08:50:42 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_743_stop_0: ok (node=ha14a, call=155, rc=0, cib-update=94, confirmed=true)
> Apr 03 08:50:42 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/94)
> Apr 03 08:50:42 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.11 2
> Apr 03 08:50:42 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.12 (null)
> Apr 03 08:50:42 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=12
> Apr 03 08:50:42 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_mysql_743']/lrm_rsc_op[@id='p_mysql_743_last_0']: @operation_key=p_mysql_743_stop_0, @operation=stop, @transition-key=97:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;97:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=155, @last-run=1491234640, @last-rc-change=1491234640, @exec-time=2105
> Apr 03 08:50:42 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/94, version=0.607.12)
> Apr 03 08:50:43 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_mysql_745 action:stop call_id:159 pid:25791 exit-code:0 exec-time:3107ms queue-time:0ms
> Apr 03 08:50:43 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_745_stop_0: ok (node=ha14a, call=159, rc=0, cib-update=95, confirmed=true)
> Apr 03 08:50:43 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/95)
> Apr 03 08:50:43 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.12 2
> Apr 03 08:50:43 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.13 (null)
> Apr 03 08:50:43 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=13
> Apr 03 08:50:43 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_mysql_745']/lrm_rsc_op[@id='p_mysql_745_last_0']: @operation_key=p_mysql_745_stop_0, @operation=stop, @transition-key=105:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;105:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=159, @last-run=1491234640, @last-rc-change=1491234640, @exec-time=3107
> Apr 03 08:50:43 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/95, version=0.607.13)
> Apr 03 08:50:44 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_mysql_213 action:stop call_id:153 pid:25785 exit-code:0 exec-time:4109ms queue-time:0ms
> Apr 03 08:50:44 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_mysql_312 action:stop call_id:156 pid:25788 exit-code:0 exec-time:4108ms queue-time:0ms
> Apr 03 08:50:44 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_mysql_289 action:stop call_id:157 pid:25789 exit-code:0 exec-time:4108ms queue-time:0ms
> Apr 03 08:50:44 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_mysql_092 action:stop call_id:151 pid:25783 exit-code:0 exec-time:4110ms queue-time:0ms
> Apr 03 08:50:44 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_213_stop_0: ok (node=ha14a, call=153, rc=0, cib-update=96, confirmed=true)
> Apr 03 08:50:44 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/96)
> Apr 03 08:50:44 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_312_stop_0: ok (node=ha14a, call=156, rc=0, cib-update=97, confirmed=true)
> Apr 03 08:50:44 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/97)
> Apr 03 08:50:44 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_289_stop_0: ok (node=ha14a, call=157, rc=0, cib-update=98, confirmed=true)
> Apr 03 08:50:44 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/98)
> Apr 03 08:50:44 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_092_stop_0: ok (node=ha14a, call=151, rc=0, cib-update=99, confirmed=true)
> Apr 03 08:50:44 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/99)
> Apr 03 08:50:44 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_mysql_057 action:stop call_id:150 pid:25782 exit-code:0 exec-time:4111ms queue-time:0ms
> Apr 03 08:50:44 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_mysql_187 action:stop call_id:152 pid:25784 exit-code:0 exec-time:4110ms queue-time:0ms
> Apr 03 08:50:44 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_mysql_702 action:stop call_id:154 pid:25786 exit-code:0 exec-time:4110ms queue-time:0ms
> Apr 03 08:50:44 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_057_stop_0: ok (node=ha14a, call=150, rc=0, cib-update=100, confirmed=true)
> Apr 03 08:50:44 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_187_stop_0: ok (node=ha14a, call=152, rc=0, cib-update=101, confirmed=true)
> Apr 03 08:50:44 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_702_stop_0: ok (node=ha14a, call=154, rc=0, cib-update=102, confirmed=true)
> Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.13 2
> Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.14 (null)
> Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=14
> Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_mysql_213']/lrm_rsc_op[@id='p_mysql_213_last_0']: @operation_key=p_mysql_213_stop_0, @operation=stop, @transition-key=93:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;93:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=153, @last-run=1491234640, @last-rc-change=1491234640, @exec-time=4109
> Apr 03 08:50:44 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/96, version=0.607.14)
> Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.14 2
> Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.15 (null)
> Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=15
> Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_mysql_312']/lrm_rsc_op[@id='p_mysql_312_last_0']: @operation_key=p_mysql_312_stop_0, @operation=stop, @transition-key=99:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;99:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=156, @last-run=1491234640, @last-rc-change=1491234640, @exec-time=4108
> Apr 03 08:50:44 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/97, version=0.607.15)
> Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.15 2
> Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.16 (null)
> Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=16
> Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_mysql_289']/lrm_rsc_op[@id='p_mysql_289_last_0']: @operation_key=p_mysql_289_stop_0, @operation=stop, @transition-key=101:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;101:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=157, @last-run=1491234640, @last-rc-change=1491234640, @exec-time=4108
> Apr 03 08:50:44 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/98, version=0.607.16)
> Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.16 2
> Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.17 (null)
> Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=17
> Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_mysql_092']/lrm_rsc_op[@id='p_mysql_092_last_0']: @operation_key=p_mysql_092_stop_0, @operation=stop, @transition-key=89:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;89:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=151, @last-run=1491234640, @last-rc-change=1491234640, @exec-time=4110
> Apr 03 08:50:44 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/99, version=0.607.17)
> Apr 03 08:50:44 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/100)
> Apr 03 08:50:44 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/101)
> Apr 03 08:50:44 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/102)
> Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.17 2
> Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.18 (null)
> Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=18
> Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_mysql_057']/lrm_rsc_op[@id='p_mysql_057_last_0']: @operation_key=p_mysql_057_stop_0, @operation=stop, @transition-key=87:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;87:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=150, @last-run=1491234640, @last-rc-change=1491234640, @exec-time=4111
> Apr 03 08:50:44 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/100, version=0.607.18)
> Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.18 2
> Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.19 (null)
> Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=19
> Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_mysql_187']/lrm_rsc_op[@id='p_mysql_187_last_0']: @operation_key=p_mysql_187_stop_0, @operation=stop, @transition-key=91:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;91:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=152, @last-run=1491234640, @last-rc-change=1491234640, @exec-time=4110
> Apr 03 08:50:44 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/101, version=0.607.19)
> Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.19 2
> Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.20 (null)
> Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=20
> Apr 03 08:50:44 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_mysql_702']/lrm_rsc_op[@id='p_mysql_702_last_0']: @operation_key=p_mysql_702_stop_0, @operation=stop, @transition-key=95:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;95:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=154, @last-run=1491234640, @last-rc-change=1491234640, @exec-time=4110
> Apr 03 08:50:44 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/102, version=0.607.20)
> Apr 03 08:50:45 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_mysql_250 action:stop call_id:158 pid:25790 exit-code:0 exec-time:5108ms queue-time:0ms
> Apr 03 08:50:45 [22767] ha14a crmd: notice: process_lrm_event: Operation p_mysql_250_stop_0: ok (node=ha14a, call=158, rc=0, cib-update=103, confirmed=true)
> Apr 03 08:50:45 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/103)
> Apr 03 08:50:45 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.20 2
> Apr 03 08:50:45 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.21 (null)
> Apr 03 08:50:45 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=21
> Apr 03 08:50:45 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_mysql_250']/lrm_rsc_op[@id='p_mysql_250_last_0']: @operation_key=p_mysql_250_stop_0, @operation=stop, @transition-key=103:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;103:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=158, @last-run=1491234640, @last-rc-change=1491234640, @exec-time=5108
> Apr 03 08:50:45 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/103, version=0.607.21)
> Apr 03 08:50:45 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=78:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_fs_clust19_stop_0
> Apr 03 08:50:45 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_fs_clust19 action:stop call_id:162
> Apr 03 08:50:45 [22764] ha14a lrmd: info: cancel_recurring_action: Cancelling operation p_vip_clust19_monitor_30000
> Apr 03 08:50:45 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=82:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_vip_clust19_stop_0
> Apr 03 08:50:45 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_vip_clust19 action:stop call_id:164
> Apr 03 08:50:45 [22767] ha14a crmd: info: process_lrm_event: Operation p_vip_clust19_monitor_30000: Cancelled (node=ha14a, call=96, confirmed=true)
> Apr 03 08:50:45 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_vip_clust19 action:stop call_id:164 pid:26174 exit-code:0 exec-time:44ms queue-time:0ms
> Apr 03 08:50:45 [22767] ha14a crmd: notice: process_lrm_event: Operation p_vip_clust19_stop_0: ok (node=ha14a, call=164, rc=0, cib-update=104, confirmed=true)
> Apr 03 08:50:45 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/104)
> Apr 03 08:50:45 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.21 2
> Apr 03 08:50:45 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.22 (null)
> Apr 03 08:50:45 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=22
> Apr 03 08:50:45 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_vip_clust19']/lrm_rsc_op[@id='p_vip_clust19_last_0']: @operation_key=p_vip_clust19_stop_0, @operation=stop, @transition-key=82:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;82:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=164, @last-run=1491234645, @last-rc-change=1491234645, @exec-time=44
> Apr 03 08:50:45 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/104, version=0.607.22)
> Apr 03 08:50:50 [22762] ha14a cib: info: cib_process_ping: Reporting our current digest to ha14b: bca4bc21567f887c56ae7ece4ac72814 for 0.607.22 (0x21e09d0 0)
> Apr 03 08:50:58 [22764] ha14a lrmd: notice: operation_finished: p_fs_clust19_stop_0:26173:stderr [ umount: /ha01_mysql: device is busy. ]
> Apr 03 08:50:58 [22764] ha14a lrmd: notice: operation_finished: p_fs_clust19_stop_0:26173:stderr [ (In some cases useful info about processes that use ]
> Apr 03 08:50:58 [22764] ha14a lrmd: notice: operation_finished: p_fs_clust19_stop_0:26173:stderr [ the device is found by lsof(8) or fuser(1)) ]
> Apr 03 08:50:58 [22764] ha14a lrmd: notice: operation_finished: p_fs_clust19_stop_0:26173:stderr [ umount: /ha01_mysql: device is busy. ]
> Apr 03 08:50:58 [22764] ha14a lrmd: notice: operation_finished: p_fs_clust19_stop_0:26173:stderr [ (In some cases useful info about processes that use ]
> Apr 03 08:50:58 [22764] ha14a lrmd: notice: operation_finished: p_fs_clust19_stop_0:26173:stderr [ the device is found by lsof(8) or fuser(1)) ]
> Apr 03 08:50:58 [22764] ha14a lrmd: notice: operation_finished: p_fs_clust19_stop_0:26173:stderr [ umount: /ha01_mysql: device is busy. ]
> Apr 03 08:50:58 [22764] ha14a lrmd: notice: operation_finished: p_fs_clust19_stop_0:26173:stderr [ (In some cases useful info about processes that use ]
> Apr 03 08:50:58 [22764] ha14a lrmd: notice: operation_finished: p_fs_clust19_stop_0:26173:stderr [ the device is found by lsof(8) or fuser(1)) ]
> Apr 03 08:50:58 [22764] ha14a lrmd: notice: operation_finished: p_fs_clust19_stop_0:26173:stderr [ umount: /ha01_mysql: device is busy. ]
> Apr 03 08:50:58 [22764] ha14a lrmd: notice: operation_finished: p_fs_clust19_stop_0:26173:stderr [ (In some cases useful info about processes that use ]
> Apr 03 08:50:58 [22764] ha14a lrmd: notice: operation_finished: p_fs_clust19_stop_0:26173:stderr [ the device is found by lsof(8) or fuser(1)) ]
> Apr 03 08:50:58 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_fs_clust19 action:stop call_id:162 pid:26173 exit-code:0 exec-time:13216ms queue-time:0ms
> Apr 03 08:50:58 [22767] ha14a crmd: notice: process_lrm_event: Operation p_fs_clust19_stop_0: ok (node=ha14a, call=162, rc=0, cib-update=105, confirmed=true)
> Apr 03 08:50:58 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/105)
> Apr 03 08:50:58 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.22 2
> Apr 03 08:50:58 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.23 (null)
> Apr 03 08:50:58 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=23
> Apr 03 08:50:58 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_fs_clust19']/lrm_rsc_op[@id='p_fs_clust19_last_0']: @operation_key=p_fs_clust19_stop_0, @operation=stop, @transition-key=78:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;78:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=162, @last-run=1491234645, @last-rc-change=1491234645, @exec-time=13216
> Apr 03 08:50:58 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/105, version=0.607.23)
> Apr 03 08:50:58 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=74:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_lv_on_drbd0_stop_0
> Apr 03 08:50:58 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_lv_on_drbd0 action:stop call_id:165
> Apr 03 08:50:58 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_lv_on_drbd0 action:stop call_id:165 pid:26495 exit-code:0 exec-time:88ms queue-time:0ms
> Apr 03 08:50:58 [22767] ha14a crmd: notice: process_lrm_event: Operation p_lv_on_drbd0_stop_0: ok (node=ha14a, call=165, rc=0, cib-update=106, confirmed=true)
> Apr 03 08:50:58 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/106)
> Apr 03 08:50:58 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.23 2
> Apr 03 08:50:58 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.24 (null)
> Apr 03 08:50:58 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=24
> Apr 03 08:50:58 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_lv_on_drbd0']/lrm_rsc_op[@id='p_lv_on_drbd0_last_0']: @operation_key=p_lv_on_drbd0_stop_0, @operation=stop, @transition-key=74:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;74:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=165, @last-run=1491234658, @last-rc-change=1491234658, @exec-time=88
> Apr 03 08:50:58 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/106, version=0.607.24)
> Apr 03 08:50:58 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=13:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_drbd0_demote_0
> Apr 03 08:50:58 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_drbd0 action:demote call_id:166
> Apr 03 08:50:58 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_drbd0 action:demote call_id:166 pid:26533 exit-code:0 exec-time:33ms queue-time:0ms
> Apr 03 08:50:58 [22767] ha14a crmd: notice: process_lrm_event: Operation p_drbd0_demote_0: ok (node=ha14a, call=166, rc=0, cib-update=107, confirmed=true)
> Apr 03 08:50:58 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/107)
> Apr 03 08:50:58 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.24 2
> Apr 03 08:50:58 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.25 (null)
> Apr 03 08:50:58 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=25
> Apr 03 08:50:58 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_drbd0']/lrm_rsc_op[@id='p_drbd0_last_0']: @operation_key=p_drbd0_demote_0, @operation=demote, @transition-key=13:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;13:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=166, @rc-code=0, @last-run=1491234658, @last-rc-change=1491234658, @exec-time=33
> Apr 03 08:50:58 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/107, version=0.607.25)
> Apr 03 08:50:58 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=124:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_drbd0_notify_0
> Apr 03 08:50:58 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_drbd0 action:notify call_id:167
> Apr 03 08:50:58 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_drbd0 action:notify call_id:167 pid:26567 exit-code:0 exec-time:42ms queue-time:0ms
> Apr 03 08:50:58 [22767] ha14a crmd: notice: process_lrm_event: Operation p_drbd0_notify_0: ok (node=ha14a, call=167, rc=0, cib-update=0, confirmed=true)
> Apr 03 08:50:58 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=119:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_drbd0_notify_0
> Apr 03 08:50:58 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_drbd0 action:notify call_id:168
> Apr 03 08:50:58 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_drbd0 action:notify call_id:168 pid:26594 exit-code:0 exec-time:27ms queue-time:0ms
> Apr 03 08:50:58 [22767] ha14a crmd: notice: process_lrm_event: Operation p_drbd0_notify_0: ok (node=ha14a, call=168, rc=0, cib-update=0, confirmed=true)
> Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.25 2
> Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.26 (null)
> Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=26
> Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_drbd0']/lrm_rsc_op[@id='p_drbd0_last_0']: @operation_key=p_drbd0_promote_0, @operation=promote, @transition-key=18:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;18:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=151, @last-run=1491234658, @last-rc-change=1491234658, @exec-time=31
> Apr 03 08:50:59 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7671, version=0.607.26)
> Apr 03 08:50:59 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=120:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_drbd0_notify_0
> Apr 03 08:50:59 [22764] ha14a lrmd: info: log_execute: executing - rsc:p_drbd0 action:notify call_id:169
> Apr 03 08:50:59 [22764] ha14a lrmd: info: log_finished: finished - rsc:p_drbd0 action:notify call_id:169 pid:26616 exit-code:0 exec-time:42ms queue-time:0ms
> Apr 03 08:50:59 [22767] ha14a crmd: notice: process_lrm_event: Operation p_drbd0_notify_0: ok (node=ha14a, call=169, rc=0, cib-update=0, confirmed=true)
> Apr 03 08:50:59 [22767] ha14a crmd: info: do_lrm_rsc_op: Performing key=15:7486:0:91ef4b03-8769-47a1-a364-060569c46e52 op=p_drbd0_monitor_31000
> Apr 03 08:50:59 [22767] ha14a crmd: notice: process_lrm_event: Operation p_drbd0_monitor_31000: ok (node=ha14a, call=170, rc=0, cib-update=108, confirmed=false)
> Apr 03 08:50:59 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/crmd/108)
> Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.26 2
> Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.27 (null)
> Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=27
> Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: ++ /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_drbd0']: <lrm_rsc_op id="p_drbd0_monitor_30000" operation_key="p_drbd0_monitor_30000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.9" transition-key="19:7486:8:91ef4b03-8769-47a1-a364-060569c46e52" transition-magic="0:8;19:7486:8:91ef4b03-8769-47a1-a364-060569c46e52" call-id="154" rc-code="8" op-status="0" interval=
> Apr 03 08:50:59 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7672, version=0.607.27)
> Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.27 2
> Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.28 (null)
> Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=28
> Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: ++ /cib/status/node_state[@id='ha14a']/lrm[@id='ha14a']/lrm_resources/lrm_resource[@id='p_drbd0']: <lrm_rsc_op id="p_drbd0_monitor_31000" operation_key="p_drbd0_monitor_31000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.9" transition-key="15:7486:0:91ef4b03-8769-47a1-a364-060569c46e52" transition-magic="0:0;15:7486:0:91ef4b03-8769-47a1-a364-060569c46e52" call-id="170" rc-code="0" op-status="0" interval=
> Apr 03 08:50:59 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/crmd/108, version=0.607.28)
> Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.28 2
> Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.29 (null)
> Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=29
> Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_lv_on_drbd0']/lrm_rsc_op[@id='p_lv_on_drbd0_last_0']: @operation_key=p_lv_on_drbd0_start_0, @operation=start, @crm-debug-origin=do_update_resource, @transition-key=75:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;75:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=153, @rc-code=0, @last-run=1491234658, @last-rc-change=14912
> Apr 03 08:50:59 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7673, version=0.607.29)
> Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.29 2
> Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.30 (null)
> Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=30
> Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_vip_clust19']/lrm_rsc_op[@id='p_vip_clust19_last_0']: @operation_key=p_vip_clust19_start_0, @operation=start, @transition-key=83:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;83:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=156, @last-run=1491234659, @last-rc-change=1491234659, @exec-time=52
> Apr 03 08:50:59 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7674, version=0.607.30)
> Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.30 2
> Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.31 (null)
> Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=31
> Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: ++ /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_vip_clust19']: <lrm_rsc_op id="p_vip_clust19_monitor_30000" operation_key="p_vip_clust19_monitor_30000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.9" transition-key="84:7486:0:91ef4b03-8769-47a1-a364-060569c46e52" transition-magic="0:0;84:7486:0:91ef4b03-8769-47a1-a364-060569c46e52" call-id="157" rc-code="0" op-st
> Apr 03 08:50:59 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7675, version=0.607.31)
> Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.31 2
> Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.32 (null)
> Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=32
> Apr 03 08:50:59 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_fs_clust19']/lrm_rsc_op[@id='p_fs_clust19_last_0']: @operation_key=p_fs_clust19_start_0, @operation=start, @transition-key=79:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;79:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=155, @rc-code=0, @last-run=1491234659, @last-rc-change=1491234659, @exec-time=123
> Apr 03 08:50:59 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7676, version=0.607.32)
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.32 2
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.33 (null)
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=33
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_mysql_057']/lrm_rsc_op[@id='p_mysql_057_last_0']: @operation_key=p_mysql_057_start_0, @operation=start, @transition-key=88:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;88:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=158, @rc-code=0, @last-run=1491234659, @last-rc-change=1491234659, @exec-time=1050
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7677, version=0.607.33)
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.33 2
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.34 (null)
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=34
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_mysql_702']/lrm_rsc_op[@id='p_mysql_702_last_0']: @operation_key=p_mysql_702_start_0, @operation=start, @transition-key=96:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;96:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=162, @rc-code=0, @last-run=1491234659, @last-rc-change=1491234659, @exec-time=1051
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7678, version=0.607.34)
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.34 2
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.35 (null)
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=35
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_mysql_250']/lrm_rsc_op[@id='p_mysql_250_last_0']: @operation_key=p_mysql_250_start_0, @operation=start, @transition-key=104:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;104:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=166, @rc-code=0, @last-run=1491234659, @last-rc-change=1491234659, @exec-time=1050
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7679, version=0.607.35)
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.35 2
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.36 (null)
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=36
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_mysql_092']/lrm_rsc_op[@id='p_mysql_092_last_0']: @operation_key=p_mysql_092_start_0, @operation=start, @transition-key=90:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;90:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=159, @rc-code=0, @last-run=1491234659, @last-rc-change=1491234659, @exec-time=1053
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7680, version=0.607.36)
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.36 2
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.37 (null)
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=37
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_mysql_745']/lrm_rsc_op[@id='p_mysql_745_last_0']: @operation_key=p_mysql_745_start_0, @operation=start, @transition-key=106:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;106:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=167, @rc-code=0, @last-run=1491234659, @last-rc-change=1491234659, @exec-time=1052
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7681, version=0.607.37)
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.37 2
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.38 (null)
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=38
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_mysql_312']/lrm_rsc_op[@id='p_mysql_312_last_0']: @operation_key=p_mysql_312_start_0, @operation=start, @transition-key=100:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;100:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=164, @rc-code=0, @last-run=1491234659, @last-rc-change=1491234659, @exec-time=1053
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7682, version=0.607.38)
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.38 2
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.39 (null)
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=39
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_mysql_289']/lrm_rsc_op[@id='p_mysql_289_last_0']: @operation_key=p_mysql_289_start_0, @operation=start, @transition-key=102:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;102:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=165, @rc-code=0, @last-run=1491234659, @last-rc-change=1491234659, @exec-time=1055
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7683, version=0.607.39)
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.39 2
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.40 (null)
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=40
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_mysql_187']/lrm_rsc_op[@id='p_mysql_187_last_0']: @operation_key=p_mysql_187_start_0, @operation=start, @transition-key=92:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;92:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=160, @rc-code=0, @last-run=1491234659, @last-rc-change=1491234659, @exec-time=1056
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7684, version=0.607.40)
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.40 2
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.41 (null)
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=41
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_mysql_743']/lrm_rsc_op[@id='p_mysql_743_last_0']: @operation_key=p_mysql_743_start_0, @operation=start, @transition-key=98:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;98:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=163, @rc-code=0, @last-run=1491234659, @last-rc-change=1491234659, @exec-time=1055
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7685, version=0.607.41)
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.41 2
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.42 (null)
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=42
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_mysql_213']/lrm_rsc_op[@id='p_mysql_213_last_0']: @operation_key=p_mysql_213_start_0, @operation=start, @transition-key=94:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;94:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=161, @rc-code=0, @last-run=1491234659, @last-rc-change=1491234659, @exec-time=1063
> Apr 03 08:51:00 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7686, version=0.607.42)
> Apr 03 08:51:05 [22762] ha14a cib: info: cib_process_ping: Reporting our current digest to ha14b: 3b1748e1e0487004594b1da9d8e47da6 for 0.607.42 (0x21e09d0 0)
> Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.42 2
> Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.43 (null)
> Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=43
> Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_mysql_746']/lrm_rsc_op[@id='p_mysql_746_last_0']: @operation_key=p_mysql_746_start_0, @operation=start, @transition-key=107:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:1;107:7486:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=168, @rc-code=1, @last-run=1491234659, @last-rc-change=1491234659, @exec-time=10060
> Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: ++ /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_mysql_746']: <lrm_rsc_op id="p_mysql_746_last_failure_0" operation_key="p_mysql_746_start_0" operation="start" crm-debug-origin="do_update_resource" crm_feature_set="3.0.9" transition-key="107:7486:0:91ef4b03-8769-47a1-a364-060569c46e52" transition-magic="0:1;107:7486:0:91ef4b03-8769-47a1-a364-060569c46e52" call-id="168" rc-code="1" op-status="0" in
> Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7687, version=0.607.43)
> Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/99)
> Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.43 2
> Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.44 (null)
> Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=44
> Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: ++ /cib/status/node_state[@id='ha14b']/transient_attributes[@id='ha14b']/instance_attributes[@id='status-ha14b']: <nvpair id="status-ha14b-fail-count-p_mysql_746" name="fail-count-p_mysql_746" value="INFINITY"/>
> Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/attrd/163, version=0.607.44)
> Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.44 2
> Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.45 (null)
> Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=45
> Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: ++ /cib/status/node_state[@id='ha14b']/transient_attributes[@id='ha14b']/instance_attributes[@id='status-ha14b']: <nvpair id="status-ha14b-last-failure-p_mysql_746" name="last-failure-p_mysql_746" value="1491234669"/>
> Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/attrd/165, version=0.607.45)
> Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/attrd/167, version=0.607.45)
> Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/attrd/169, version=0.607.45)
> Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/attrd/171, version=0.607.45)
> Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/attrd/173, version=0.607.45)
> Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/attrd/175, version=0.607.45)
> Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/99, version=0.607.45)
> Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/attrd/177, version=0.607.45)
> Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/101)
> Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/103)
> Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/105)
> Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/101, version=0.607.45)
> Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/103, version=0.607.45)
> Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/107)
> Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/105, version=0.607.45)
> Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/109)
> Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/107, version=0.607.45)
> Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/109, version=0.607.45)
> Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/111)
> Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/111, version=0.607.45)
> Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Forwarding cib_modify operation for section status to master (origin=local/attrd/113)
> Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14a/attrd/113, version=0.607.45)
> Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: Diff: --- 0.607.45 2
> Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: Diff: +++ 0.607.46 (null)
> Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: + /cib: @num_updates=46
> Apr 03 08:51:09 [22762] ha14a cib: info: cib_perform_op: + /cib/status/node_state[@id='ha14b']/lrm[@id='ha14b']/lrm_resources/lrm_resource[@id='p_mysql_746']/lrm_rsc_op[@id='p_mysql_746_last_0']: @operation_key=p_mysql_746_stop_0, @operation=stop, @transition-key=7:7488:0:91ef4b03-8769-47a1-a364-060569c46e52, @transition-magic=0:0;7:7488:0:91ef4b03-8769-47a1-a364-060569c46e52, @call-id=169, @rc-code=0, @last-run=1491234669, @last-rc-change=1491234669, @exec-time=24
> Apr 03 08:51:09 [22762] ha14a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=ha14b/crmd/7691, version=0.607.46)
> Apr 03 08:51:14 [22762] ha14a cib: info: cib_process_ping: Reporting our current digest to ha14b: 5384604825dd17358b76d9b84f2be021 for 0.607.46 (0x21e09d0 0)
>
> --
> Eric Robinson
More information about the Users
mailing list