[ClusterLabs] Why Do All The Services Go Down When Just One Fails?
Eric Robinson
eric.robinson at psmnv.com
Sat Feb 16 16:03:43 EST 2019
Here are the relevant corosync logs.
It appears that the stop action for resource p_mysql_002 failed, and that caused a cascading series of service changes. However, I don't understand why, since no other resources are dependent on p_mysql_002.
[root at 001db01a cluster]# cat corosync_filtered.log
Feb 16 14:06:24 [3908] 001db01a cib: info: cib_process_request: Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2)
Feb 16 14:06:24 [3908] 001db01a cib: info: cib_perform_op: Diff: --- 0.345.30 2
Feb 16 14:06:24 [3908] 001db01a cib: info: cib_perform_op: Diff: +++ 0.346.0 cc0da1b030418ec8b7c72db1115e2af1
Feb 16 14:06:24 [3908] 001db01a cib: info: cib_perform_op: + /cib: @epoch=346, @num_updates=0
Feb 16 14:06:24 [3908] 001db01a cib: info: cib_perform_op: ++ /cib/configuration/resources/primitive[@id='p_mysql_002']: <meta_attributes id="p_mysql_002-meta_attributes"/>
Feb 16 14:06:24 [3908] 001db01a cib: info: cib_perform_op: ++ <nvpair id="p_mysql_002-meta_attributes-target-role" name="target-role" value="Stopped"/>
Feb 16 14:06:24 [3908] 001db01a cib: info: cib_perform_op: ++ </meta_attributes>
Feb 16 14:06:24 [3908] 001db01a cib: info: cib_process_request: Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=001db01a/cibadmin/2, version=0.346.0)
Feb 16 14:06:24 [3913] 001db01a crmd: info: abort_transition_graph: Transition aborted by meta_attributes.p_mysql_002-meta_attributes 'create': Configuration change | cib=0.346.0 source=te_update_diff:456 path=/cib/configuration/resources/primitive[@id='p_mysql_002'] complete=true
Feb 16 14:06:24 [3913] 001db01a crmd: notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph
Feb 16 14:06:24 [3912] 001db01a pengine: notice: unpack_config: On loss of CCM Quorum: Ignore
Feb 16 14:06:24 [3912] 001db01a pengine: info: determine_online_status: Node 001db01b is online
Feb 16 14:06:24 [3912] 001db01a pengine: info: determine_online_status: Node 001db01a is online
Feb 16 14:06:24 [3912] 001db01a pengine: info: determine_op_status: Operation monitor found resource p_drbd0:0 active in master mode on 001db01b
Feb 16 14:06:24 [3912] 001db01a pengine: info: determine_op_status: Operation monitor found resource p_drbd1:0 active on 001db01b
Feb 16 14:06:24 [3912] 001db01a pengine: info: determine_op_status: Operation monitor found resource p_mysql_004 active on 001db01a
Feb 16 14:06:24 [3912] 001db01a pengine: info: determine_op_status: Operation monitor found resource p_mysql_005 active on 001db01a
Feb 16 14:06:24 [3912] 001db01a pengine: info: determine_op_status: Operation monitor found resource p_drbd0:1 active on 001db01a
Feb 16 14:06:24 [3912] 001db01a pengine: info: determine_op_status: Operation monitor found resource p_drbd1:1 active on 001db01a
Feb 16 14:06:24 [3912] 001db01a pengine: info: determine_op_status: Operation monitor found resource p_mysql_001 active on 001db01a
Feb 16 14:06:24 [3912] 001db01a pengine: info: determine_op_status: Operation monitor found resource p_mysql_002 active on 001db01a
Feb 16 14:06:24 [3912] 001db01a pengine: info: determine_op_status: Operation monitor found resource p_mysql_002 active on 001db01a
Feb 16 14:06:24 [3912] 001db01a pengine: info: unpack_node_loop: Node 2 is already processed
Feb 16 14:06:24 [3912] 001db01a pengine: info: unpack_node_loop: Node 1 is already processed
Feb 16 14:06:24 [3912] 001db01a pengine: info: unpack_node_loop: Node 2 is already processed
Feb 16 14:06:24 [3912] 001db01a pengine: info: unpack_node_loop: Node 1 is already processed
Feb 16 14:06:24 [3912] 001db01a pengine: info: common_print: p_vip_clust01 (ocf::heartbeat:IPaddr2): Started 001db01a
Feb 16 14:06:24 [3912] 001db01a pengine: info: clone_print: Master/Slave Set: ms_drbd0 [p_drbd0]
Feb 16 14:06:24 [3912] 001db01a pengine: info: short_print: Masters: [ 001db01a ]
Feb 16 14:06:24 [3912] 001db01a pengine: info: short_print: Slaves: [ 001db01b ]
Feb 16 14:06:24 [3912] 001db01a pengine: info: clone_print: Master/Slave Set: ms_drbd1 [p_drbd1]
Feb 16 14:06:24 [3912] 001db01a pengine: info: short_print: Masters: [ 001db01b ]
Feb 16 14:06:24 [3912] 001db01a pengine: info: short_print: Slaves: [ 001db01a ]
Feb 16 14:06:24 [3912] 001db01a pengine: info: common_print: p_fs_clust01 (ocf::heartbeat:Filesystem): Started 001db01a
Feb 16 14:06:24 [3912] 001db01a pengine: info: common_print: p_fs_clust02 (ocf::heartbeat:Filesystem): Started 001db01b
Feb 16 14:06:24 [3912] 001db01a pengine: info: common_print: p_vip_clust02 (ocf::heartbeat:IPaddr2): Started 001db01b
Feb 16 14:06:24 [3912] 001db01a pengine: info: common_print: p_mysql_001 (lsb:mysql_001): Started 001db01a
Feb 16 14:06:24 [3912] 001db01a pengine: info: common_print: p_mysql_000 (lsb:mysql_000): Started 001db01a
Feb 16 14:06:24 [3912] 001db01a pengine: info: common_print: p_mysql_002 (lsb:mysql_002): Started 001db01a (disabled)
Feb 16 14:06:24 [3912] 001db01a pengine: info: common_print: p_mysql_003 (lsb:mysql_003): Started 001db01a
Feb 16 14:06:24 [3912] 001db01a pengine: info: common_print: p_mysql_004 (lsb:mysql_004): Started 001db01a
Feb 16 14:06:24 [3912] 001db01a pengine: info: common_print: p_mysql_005 (lsb:mysql_005): Started 001db01a
Feb 16 14:06:24 [3912] 001db01a pengine: info: common_print: p_mysql_006 (lsb:mysql_006): Started 001db01b
Feb 16 14:06:24 [3912] 001db01a pengine: info: common_print: p_mysql_007 (lsb:mysql_007): Started 001db01b
Feb 16 14:06:24 [3912] 001db01a pengine: info: common_print: p_mysql_008 (lsb:mysql_008): Started 001db01b
Feb 16 14:06:24 [3912] 001db01a pengine: info: common_print: p_mysql_622 (lsb:mysql_622): Started 001db01a
Feb 16 14:06:24 [3912] 001db01a pengine: info: master_color: Promoting p_drbd0:1 (Master 001db01a)
Feb 16 14:06:24 [3912] 001db01a pengine: info: master_color: ms_drbd0: Promoted 1 instances of a possible 1 to master
Feb 16 14:06:24 [3912] 001db01a pengine: info: master_color: Promoting p_drbd1:0 (Master 001db01b)
Feb 16 14:06:24 [3912] 001db01a pengine: info: master_color: ms_drbd1: Promoted 1 instances of a possible 1 to master
Feb 16 14:06:24 [3912] 001db01a pengine: info: native_color: Resource p_mysql_002 cannot run anywhere
Feb 16 14:06:24 [3912] 001db01a pengine: info: LogActions: Leave p_vip_clust01 (Started 001db01a)
Feb 16 14:06:24 [3912] 001db01a pengine: info: LogActions: Leave p_drbd0:0 (Slave 001db01b)
Feb 16 14:06:24 [3912] 001db01a pengine: info: LogActions: Leave p_drbd0:1 (Master 001db01a)
Feb 16 14:06:24 [3912] 001db01a pengine: info: LogActions: Leave p_drbd1:0 (Master 001db01b)
Feb 16 14:06:24 [3912] 001db01a pengine: info: LogActions: Leave p_drbd1:1 (Slave 001db01a)
Feb 16 14:06:24 [3912] 001db01a pengine: info: LogActions: Leave p_fs_clust01 (Started 001db01a)
Feb 16 14:06:24 [3912] 001db01a pengine: info: LogActions: Leave p_fs_clust02 (Started 001db01b)
Feb 16 14:06:24 [3912] 001db01a pengine: info: LogActions: Leave p_vip_clust02 (Started 001db01b)
Feb 16 14:06:24 [3912] 001db01a pengine: info: LogActions: Leave p_mysql_001 (Started 001db01a)
Feb 16 14:06:24 [3912] 001db01a pengine: info: LogActions: Leave p_mysql_000 (Started 001db01a)
Feb 16 14:06:24 [3912] 001db01a pengine: notice: LogAction: * Stop p_mysql_002 ( 001db01a ) due to node availability
Feb 16 14:06:24 [3912] 001db01a pengine: info: LogActions: Leave p_mysql_003 (Started 001db01a)
Feb 16 14:06:24 [3912] 001db01a pengine: info: LogActions: Leave p_mysql_004 (Started 001db01a)
Feb 16 14:06:24 [3912] 001db01a pengine: info: LogActions: Leave p_mysql_005 (Started 001db01a)
Feb 16 14:06:24 [3912] 001db01a pengine: info: LogActions: Leave p_mysql_006 (Started 001db01b)
Feb 16 14:06:24 [3912] 001db01a pengine: info: LogActions: Leave p_mysql_007 (Started 001db01b)
Feb 16 14:06:24 [3912] 001db01a pengine: info: LogActions: Leave p_mysql_008 (Started 001db01b)
Feb 16 14:06:24 [3912] 001db01a pengine: info: LogActions: Leave p_mysql_622 (Started 001db01a)
Feb 16 14:06:24 [3912] 001db01a pengine: notice: process_pe_message: Calculated transition 17789, saving inputs in /var/lib/pacemaker/pengine/pe-input-480.bz2
Feb 16 14:06:24 [3913] 001db01a crmd: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
Feb 16 14:06:24 [3913] 001db01a crmd: info: do_te_invoke: Processing graph 17789 (ref=pe_calc-dc-1550343984-19545) derived from /var/lib/pacemaker/pengine/pe-input-480.bz2
Feb 16 14:06:24 [3913] 001db01a crmd: notice: te_rsc_command: Initiating stop operation p_mysql_002_stop_0 locally on 001db01a | action 90
Feb 16 14:06:24 [3910] 001db01a lrmd: info: cancel_recurring_action: Cancelling lsb operation p_mysql_002_status_15000
Feb 16 14:06:24 [3913] 001db01a crmd: info: do_lrm_rsc_op: Performing key=90:17789:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5 op=p_mysql_002_stop_0
Feb 16 14:06:24 [3910] 001db01a lrmd: info: log_execute: executing - rsc:p_mysql_002 action:stop call_id:1094
Feb 16 14:06:24 [3913] 001db01a crmd: info: process_lrm_event: Result of monitor operation for p_mysql_002 on 001db01a: Cancelled | call=1088 key=p_mysql_002_monitor_15000 confirmed=true
Feb 16 14:06:24 [3908] 001db01a cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/20017)
Feb 16 14:06:24 [3908] 001db01a cib: info: cib_perform_op: Diff: --- 0.346.0 2
Feb 16 14:06:24 [3908] 001db01a cib: info: cib_perform_op: Diff: +++ 0.346.1 (null)
Feb 16 14:06:24 [3908] 001db01a cib: info: cib_perform_op: + /cib: @num_updates=1
Feb 16 14:06:24 [3908] 001db01a cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_002']/lrm_rsc_op[@id='p_mysql_002_last_0']: @operation_key=p_mysql_002_stop_0, @operation=stop, @transition-key=90:17789:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @transition-magic=-1:193;90:17789:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @call-id=-1, @rc-code=193, @op-status=-1, @last-run=1550343984, @last-rc-change=1550343984, @exec-time=0
Feb 16 14:06:24 [3908] 001db01a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=001db01a/crmd/20017, version=0.346.1)
Feb 16 14:06:24 [3908] 001db01a cib: info: cib_file_backup: Archived previous version as /var/lib/pacemaker/cib/cib-72.raw
Feb 16 14:06:24 [3908] 001db01a cib: info: cib_file_write_with_digest: Wrote version 0.346.0 of the CIB to disk (digest: 5efac46096f70afefaf0a677cf4ca823)
Feb 16 14:06:24 [3908] 001db01a cib: info: cib_file_write_with_digest: Reading cluster configuration file /var/lib/pacemaker/cib/cib.cp2XNp (digest: /var/lib/pacemaker/cib/cib.5hNGXZ)
Feb 16 14:06:29 [3908] 001db01a cib: info: cib_process_ping: Reporting our current digest to 001db01a: 1064b8ca173ab45545a221e13e4716d8 for 0.346.1 (0x55ac23ea4ef0 0)
Feb 16 14:06:39 [3910] 001db01a lrmd: warning: child_timeout_callback: p_mysql_002_stop_0 process (PID 36067) timed out
Feb 16 14:06:39 [3910] 001db01a lrmd: warning: operation_finished: p_mysql_002_stop_0:36067 - timed out after 15000ms
Feb 16 14:06:39 [3910] 001db01a lrmd: info: log_finished: finished - rsc:p_mysql_002 action:stop call_id:1094 pid:36067 exit-code:1 exec-time:15004ms queue-time:0ms
Feb 16 14:06:39 [3913] 001db01a crmd: error: process_lrm_event: Result of stop operation for p_mysql_002 on 001db01a: Timed Out | call=1094 key=p_mysql_002_stop_0 timeout=15000ms
Feb 16 14:06:39 [3913] 001db01a crmd: notice: process_lrm_event: 001db01a-p_mysql_002_stop_0:1094 [ killing mysqld with pid 69229\n.............. ]
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/20018)
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff: --- 0.346.1 2
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff: +++ 0.346.2 (null)
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: + /cib: @num_updates=2
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_002']/lrm_rsc_op[@id='p_mysql_002_last_0']: @transition-magic=2:1;90:17789:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @call-id=1094, @rc-code=1, @op-status=2, @exec-time=15004
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_002']/lrm_rsc_op[@id='p_mysql_002_last_failure_0']: @operation_key=p_mysql_002_stop_0, @operation=stop, @transition-key=90:17789:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @transition-magic=2:1;90:17789:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @call-id=1094, @rc-code=1, @op-status=2, @last-run=1550343984, @last-rc-change=1550343984, @exec-time=15004
Feb 16 14:06:39 [3913] 001db01a crmd: warning: status_from_rc: Action 90 (p_mysql_002_stop_0) on 001db01a failed (target: 0 vs. rc: 1): Error
Feb 16 14:06:39 [3913] 001db01a crmd: notice: abort_transition_graph: Transition aborted by operation p_mysql_002_stop_0 'modify' on 001db01a: Event failed | magic=2:1;90:17789:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5 cib=0.346.2 source=match_graph_event:310 complete=false
Feb 16 14:06:39 [3913] 001db01a crmd: info: match_graph_event: Action p_mysql_002_stop_0 (90) confirmed on 001db01a (rc=1)
Feb 16 14:06:39 [3913] 001db01a crmd: info: update_failcount: Updating failcount for p_mysql_002 on 001db01a after failed stop: rc=1 (update=INFINITY, time=1550343999)
Feb 16 14:06:39 [3913] 001db01a crmd: info: process_graph_event: Detected action (17789.90) p_mysql_002_stop_0.1094=unknown error: failed
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=001db01a/crmd/20018, version=0.346.2)
Feb 16 14:06:39 [3913] 001db01a crmd: warning: status_from_rc: Action 90 (p_mysql_002_stop_0) on 001db01a failed (target: 0 vs. rc: 1): Error
Feb 16 14:06:39 [3913] 001db01a crmd: info: abort_transition_graph: Transition aborted by operation p_mysql_002_stop_0 'modify' on 001db01a: Event failed | magic=2:1;90:17789:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5 cib=0.346.2 source=match_graph_event:310 complete=false
Feb 16 14:06:39 [3913] 001db01a crmd: info: match_graph_event: Action p_mysql_002_stop_0 (90) confirmed on 001db01a (rc=1)
Feb 16 14:06:39 [3913] 001db01a crmd: info: update_failcount: Updating failcount for p_mysql_002 on 001db01a after failed stop: rc=1 (update=INFINITY, time=1550343999)
Feb 16 14:06:39 [3911] 001db01a attrd: info: attrd_peer_update: Setting fail-count-p_mysql_002#stop_0[001db01a]: (null) -> INFINITY from 001db01a
Feb 16 14:06:39 [3911] 001db01a attrd: info: write_attribute: Sent update 666 with 2 changes for fail-count-p_mysql_002#stop_0, id=<n/a>, set=(null)
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/attrd/666)
Feb 16 14:06:39 [3911] 001db01a attrd: info: attrd_peer_update: Setting last-failure-p_mysql_002#stop_0[001db01a]: (null) -> 1550343999 from 001db01a
Feb 16 14:06:39 [3913] 001db01a crmd: info: process_graph_event: Detected action (17789.90) p_mysql_002_stop_0.1094=unknown error: failed
Feb 16 14:06:39 [3913] 001db01a crmd: notice: run_graph: Transition 17789 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=1, Source=/var/lib/pacemaker/pengine/pe-input-480.bz2): Complete
Feb 16 14:06:39 [3913] 001db01a crmd: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd
Feb 16 14:06:39 [3911] 001db01a attrd: info: write_attribute: Sent update 667 with 2 changes for last-failure-p_mysql_002#stop_0, id=<n/a>, set=(null)
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff: --- 0.346.2 2
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff: +++ 0.346.3 (null)
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: + /cib: @num_updates=3
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']: <nvpair id="status-1-fail-count-p_mysql_002.stop_0" name="fail-count-p_mysql_002#stop_0" value="INFINITY"/>
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=001db01a/attrd/666, version=0.346.3)
Feb 16 14:06:39 [3913] 001db01a crmd: info: abort_transition_graph: Transition aborted by status-1-fail-count-p_mysql_002.stop_0 doing create fail-count-p_mysql_002#stop_0=INFINITY: Transient attribute change | cib=0.346.3 source=abort_unless_down:343 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true
Feb 16 14:06:39 [3911] 001db01a attrd: info: attrd_cib_callback: Update 666 for fail-count-p_mysql_002#stop_0: OK (0)
Feb 16 14:06:39 [3911] 001db01a attrd: info: attrd_cib_callback: Update 666 for fail-count-p_mysql_002#stop_0[001db01a]=INFINITY: OK (0)
Feb 16 14:06:39 [3911] 001db01a attrd: info: attrd_cib_callback: Update 666 for fail-count-p_mysql_002#stop_0[001db01b]=(null): OK (0)
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/attrd/667)
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff: --- 0.346.3 2
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff: +++ 0.346.4 (null)
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: + /cib: @num_updates=4
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']: <nvpair id="status-1-last-failure-p_mysql_002.stop_0" name="last-failure-p_mysql_002#stop_0" value="1550343999"/>
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=001db01a/attrd/667, version=0.346.4)
Feb 16 14:06:39 [3911] 001db01a attrd: info: attrd_cib_callback: Update 667 for last-failure-p_mysql_002#stop_0: OK (0)
Feb 16 14:06:39 [3911] 001db01a attrd: info: attrd_cib_callback: Update 667 for last-failure-p_mysql_002#stop_0[001db01a]=1550343999: OK (0)
Feb 16 14:06:39 [3911] 001db01a attrd: info: attrd_cib_callback: Update 667 for last-failure-p_mysql_002#stop_0[001db01b]=(null): OK (0)
Feb 16 14:06:39 [3913] 001db01a crmd: info: abort_transition_graph: Transition aborted by status-1-last-failure-p_mysql_002.stop_0 doing create last-failure-p_mysql_002#stop_0=1550343999: Transient attribute change | cib=0.346.4 source=abort_unless_down:343 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true
Feb 16 14:06:39 [3912] 001db01a pengine: notice: unpack_config: On loss of CCM Quorum: Ignore
Feb 16 14:06:39 [3912] 001db01a pengine: info: determine_online_status: Node 001db01b is online
Feb 16 14:06:39 [3912] 001db01a pengine: info: determine_online_status: Node 001db01a is online
Feb 16 14:06:39 [3912] 001db01a pengine: info: determine_op_status: Operation monitor found resource p_drbd0:0 active in master mode on 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: determine_op_status: Operation monitor found resource p_drbd1:0 active on 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: determine_op_status: Operation monitor found resource p_mysql_004 active on 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: determine_op_status: Operation monitor found resource p_mysql_005 active on 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: determine_op_status: Operation monitor found resource p_drbd0:1 active on 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: determine_op_status: Operation monitor found resource p_drbd1:1 active on 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: determine_op_status: Operation monitor found resource p_mysql_001 active on 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: warning: unpack_rsc_op_failure: Processing failed op stop for p_mysql_002 on 001db01a: unknown error (1)
Feb 16 14:06:39 [3912] 001db01a pengine: warning: unpack_rsc_op_failure: Processing failed op stop for p_mysql_002 on 001db01a: unknown error (1)
Feb 16 14:06:39 [3912] 001db01a pengine: info: native_add_running: resource p_mysql_002 isn't managed
Feb 16 14:06:39 [3912] 001db01a pengine: info: unpack_node_loop: Node 2 is already processed
Feb 16 14:06:39 [3912] 001db01a pengine: info: unpack_node_loop: Node 1 is already processed
Feb 16 14:06:39 [3912] 001db01a pengine: info: unpack_node_loop: Node 2 is already processed
Feb 16 14:06:39 [3912] 001db01a pengine: info: unpack_node_loop: Node 1 is already processed
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print: p_vip_clust01 (ocf::heartbeat:IPaddr2): Started 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: clone_print: Master/Slave Set: ms_drbd0 [p_drbd0]
Feb 16 14:06:39 [3912] 001db01a pengine: info: short_print: Masters: [ 001db01a ]
Feb 16 14:06:39 [3912] 001db01a pengine: info: short_print: Slaves: [ 001db01b ]
Feb 16 14:06:39 [3912] 001db01a pengine: info: clone_print: Master/Slave Set: ms_drbd1 [p_drbd1]
Feb 16 14:06:39 [3912] 001db01a pengine: info: short_print: Masters: [ 001db01b ]
Feb 16 14:06:39 [3912] 001db01a pengine: info: short_print: Slaves: [ 001db01a ]
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print: p_fs_clust01 (ocf::heartbeat:Filesystem): Started 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print: p_fs_clust02 (ocf::heartbeat:Filesystem): Started 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print: p_vip_clust02 (ocf::heartbeat:IPaddr2): Started 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print: p_mysql_001 (lsb:mysql_001): Started 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print: p_mysql_000 (lsb:mysql_000): Started 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print: p_mysql_002 (lsb:mysql_002): FAILED 001db01a (disabled, blocked)
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print: p_mysql_003 (lsb:mysql_003): Started 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print: p_mysql_004 (lsb:mysql_004): Started 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print: p_mysql_005 (lsb:mysql_005): Started 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print: p_mysql_006 (lsb:mysql_006): Started 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print: p_mysql_007 (lsb:mysql_007): Started 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print: p_mysql_008 (lsb:mysql_008): Started 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print: p_mysql_622 (lsb:mysql_622): Started 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: pe_get_failcount: p_mysql_002 has failed INFINITY times on 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: warning: check_migration_threshold: Forcing p_mysql_002 away from 001db01a after 1000000 failures (max=1000000)
Feb 16 14:06:39 [3912] 001db01a pengine: info: rsc_merge_weights: p_drbd0:1: Rolling back scores from p_mysql_002
Feb 16 14:06:39 [3912] 001db01a pengine: info: master_color: Promoting p_drbd0:0 (Slave 001db01b)
Feb 16 14:06:39 [3912] 001db01a pengine: info: master_color: ms_drbd0: Promoted 1 instances of a possible 1 to master
Feb 16 14:06:39 [3912] 001db01a pengine: info: master_color: Promoting p_drbd1:0 (Master 001db01b)
Feb 16 14:06:39 [3912] 001db01a pengine: info: master_color: ms_drbd1: Promoted 1 instances of a possible 1 to master
Feb 16 14:06:39 [3912] 001db01a pengine: info: native_color: Unmanaged resource p_mysql_002 allocated to 001db01a: failed
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp: Start recurring monitor (30s) for p_vip_clust01 on 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp: Cancelling action p_drbd0:0_monitor_60000 (Slave vs. Master)
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp: Start recurring monitor (60s) for p_drbd0:1 on 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp: Cancelling action p_drbd0:0_monitor_60000 (Slave vs. Master)
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp: Start recurring monitor (60s) for p_drbd0:1 on 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp: Start recurring monitor (20s) for p_fs_clust01 on 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp: Start recurring monitor (15s) for p_mysql_001 on 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp: Start recurring monitor (15s) for p_mysql_000 on 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp: Start recurring monitor (15s) for p_mysql_003 on 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp: Start recurring monitor (15s) for p_mysql_004 on 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp: Start recurring monitor (15s) for p_mysql_005 on 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp: Start recurring monitor (15s) for p_mysql_622 on 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: notice: LogAction: * Stop p_vip_clust01 ( 001db01a ) blocked
Feb 16 14:06:39 [3912] 001db01a pengine: notice: LogAction: * Promote p_drbd0:0 ( Slave -> Master 001db01b )
Feb 16 14:06:39 [3912] 001db01a pengine: notice: LogAction: * Demote p_drbd0:1 ( Master -> Slave 001db01a )
Feb 16 14:06:39 [3912] 001db01a pengine: info: LogActions: Leave p_drbd1:0 (Master 001db01b)
Feb 16 14:06:39 [3912] 001db01a pengine: info: LogActions: Leave p_drbd1:1 (Slave 001db01a)
Feb 16 14:06:39 [3912] 001db01a pengine: notice: LogAction: * Move p_fs_clust01 ( 001db01a -> 001db01b )
Feb 16 14:06:39 [3912] 001db01a pengine: info: LogActions: Leave p_fs_clust02 (Started 001db01b)
Feb 16 14:06:39 [3912] 001db01a pengine: info: LogActions: Leave p_vip_clust02 (Started 001db01b)
Feb 16 14:06:39 [3912] 001db01a pengine: notice: LogAction: * Stop p_mysql_001 ( 001db01a ) due to colocation with p_vip_clust01
Feb 16 14:06:39 [3912] 001db01a pengine: notice: LogAction: * Stop p_mysql_000 ( 001db01a ) due to colocation with p_vip_clust01
Feb 16 14:06:39 [3912] 001db01a pengine: info: LogActions: Leave p_mysql_002 (Started unmanaged)
Feb 16 14:06:39 [3912] 001db01a pengine: notice: LogAction: * Stop p_mysql_003 ( 001db01a ) due to colocation with p_vip_clust01
Feb 16 14:06:39 [3912] 001db01a pengine: notice: LogAction: * Stop p_mysql_004 ( 001db01a ) due to colocation with p_vip_clust01
Feb 16 14:06:39 [3912] 001db01a pengine: notice: LogAction: * Stop p_mysql_005 ( 001db01a ) due to colocation with p_vip_clust01
Feb 16 14:06:39 [3912] 001db01a pengine: info: LogActions: Leave p_mysql_006 (Started 001db01b)
Feb 16 14:06:39 [3912] 001db01a pengine: info: LogActions: Leave p_mysql_007 (Started 001db01b)
Feb 16 14:06:39 [3912] 001db01a pengine: info: LogActions: Leave p_mysql_008 (Started 001db01b)
Feb 16 14:06:39 [3912] 001db01a pengine: notice: LogAction: * Stop p_mysql_622 ( 001db01a ) due to colocation with p_vip_clust01
Feb 16 14:06:39 [3912] 001db01a pengine: notice: process_pe_message: Calculated transition 17790, saving inputs in /var/lib/pacemaker/pengine/pe-input-481.bz2
Feb 16 14:06:39 [3913] 001db01a crmd: info: handle_response: pe_calc calculation pe_calc-dc-1550343999-19547 is obsolete
Feb 16 14:06:39 [3912] 001db01a pengine: notice: unpack_config: On loss of CCM Quorum: Ignore
Feb 16 14:06:39 [3912] 001db01a pengine: info: determine_online_status: Node 001db01b is online
Feb 16 14:06:39 [3912] 001db01a pengine: info: determine_online_status: Node 001db01a is online
Feb 16 14:06:39 [3912] 001db01a pengine: info: determine_op_status: Operation monitor found resource p_drbd0:0 active in master mode on 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: determine_op_status: Operation monitor found resource p_drbd1:0 active on 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: determine_op_status: Operation monitor found resource p_mysql_004 active on 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: determine_op_status: Operation monitor found resource p_mysql_005 active on 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: determine_op_status: Operation monitor found resource p_drbd0:1 active on 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: determine_op_status: Operation monitor found resource p_drbd1:1 active on 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: determine_op_status: Operation monitor found resource p_mysql_001 active on 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: warning: unpack_rsc_op_failure: Processing failed op stop for p_mysql_002 on 001db01a: unknown error (1)
Feb 16 14:06:39 [3912] 001db01a pengine: warning: unpack_rsc_op_failure: Processing failed op stop for p_mysql_002 on 001db01a: unknown error (1)
Feb 16 14:06:39 [3912] 001db01a pengine: info: native_add_running: resource p_mysql_002 isn't managed
Feb 16 14:06:39 [3912] 001db01a pengine: info: unpack_node_loop: Node 2 is already processed
Feb 16 14:06:39 [3912] 001db01a pengine: info: unpack_node_loop: Node 1 is already processed
Feb 16 14:06:39 [3912] 001db01a pengine: info: unpack_node_loop: Node 2 is already processed
Feb 16 14:06:39 [3912] 001db01a pengine: info: unpack_node_loop: Node 1 is already processed
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print: p_vip_clust01 (ocf::heartbeat:IPaddr2): Started 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: clone_print: Master/Slave Set: ms_drbd0 [p_drbd0]
Feb 16 14:06:39 [3912] 001db01a pengine: info: short_print: Masters: [ 001db01a ]
Feb 16 14:06:39 [3912] 001db01a pengine: info: short_print: Slaves: [ 001db01b ]
Feb 16 14:06:39 [3912] 001db01a pengine: info: clone_print: Master/Slave Set: ms_drbd1 [p_drbd1]
Feb 16 14:06:39 [3912] 001db01a pengine: info: short_print: Masters: [ 001db01b ]
Feb 16 14:06:39 [3912] 001db01a pengine: info: short_print: Slaves: [ 001db01a ]
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print: p_fs_clust01 (ocf::heartbeat:Filesystem): Started 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print: p_fs_clust02 (ocf::heartbeat:Filesystem): Started 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print: p_vip_clust02 (ocf::heartbeat:IPaddr2): Started 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print: p_mysql_001 (lsb:mysql_001): Started 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print: p_mysql_000 (lsb:mysql_000): Started 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print: p_mysql_002 (lsb:mysql_002): FAILED 001db01a (disabled, blocked)
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print: p_mysql_003 (lsb:mysql_003): Started 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print: p_mysql_004 (lsb:mysql_004): Started 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print: p_mysql_005 (lsb:mysql_005): Started 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print: p_mysql_006 (lsb:mysql_006): Started 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print: p_mysql_007 (lsb:mysql_007): Started 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print: p_mysql_008 (lsb:mysql_008): Started 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: common_print: p_mysql_622 (lsb:mysql_622): Started 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: pe_get_failcount: p_mysql_002 has failed INFINITY times on 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: warning: check_migration_threshold: Forcing p_mysql_002 away from 001db01a after 1000000 failures (max=1000000)
Feb 16 14:06:39 [3912] 001db01a pengine: info: rsc_merge_weights: p_drbd0:1: Rolling back scores from p_mysql_002
Feb 16 14:06:39 [3912] 001db01a pengine: info: master_color: Promoting p_drbd0:0 (Slave 001db01b)
Feb 16 14:06:39 [3912] 001db01a pengine: info: master_color: ms_drbd0: Promoted 1 instances of a possible 1 to master
Feb 16 14:06:39 [3912] 001db01a pengine: info: master_color: Promoting p_drbd1:0 (Master 001db01b)
Feb 16 14:06:39 [3912] 001db01a pengine: info: master_color: ms_drbd1: Promoted 1 instances of a possible 1 to master
Feb 16 14:06:39 [3912] 001db01a pengine: info: native_color: Unmanaged resource p_mysql_002 allocated to 001db01a: failed
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp: Start recurring monitor (30s) for p_vip_clust01 on 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp: Cancelling action p_drbd0:0_monitor_60000 (Slave vs. Master)
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp: Start recurring monitor (60s) for p_drbd0:1 on 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp: Cancelling action p_drbd0:0_monitor_60000 (Slave vs. Master)
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp: Start recurring monitor (60s) for p_drbd0:1 on 001db01a
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp: Start recurring monitor (20s) for p_fs_clust01 on 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp: Start recurring monitor (15s) for p_mysql_001 on 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp: Start recurring monitor (15s) for p_mysql_000 on 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp: Start recurring monitor (15s) for p_mysql_003 on 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp: Start recurring monitor (15s) for p_mysql_004 on 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp: Start recurring monitor (15s) for p_mysql_005 on 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: info: RecurringOp: Start recurring monitor (15s) for p_mysql_622 on 001db01b
Feb 16 14:06:39 [3912] 001db01a pengine: notice: LogAction: * Stop p_vip_clust01 ( 001db01a ) blocked
Feb 16 14:06:39 [3912] 001db01a pengine: notice: LogAction: * Promote p_drbd0:0 ( Slave -> Master 001db01b )
Feb 16 14:06:39 [3912] 001db01a pengine: notice: LogAction: * Demote p_drbd0:1 ( Master -> Slave 001db01a )
Feb 16 14:06:39 [3912] 001db01a pengine: info: LogActions: Leave p_drbd1:0 (Master 001db01b)
Feb 16 14:06:39 [3912] 001db01a pengine: info: LogActions: Leave p_drbd1:1 (Slave 001db01a)
Feb 16 14:06:39 [3912] 001db01a pengine: notice: LogAction: * Move p_fs_clust01 ( 001db01a -> 001db01b )
Feb 16 14:06:39 [3912] 001db01a pengine: info: LogActions: Leave p_fs_clust02 (Started 001db01b)
Feb 16 14:06:39 [3912] 001db01a pengine: info: LogActions: Leave p_vip_clust02 (Started 001db01b)
Feb 16 14:06:39 [3912] 001db01a pengine: notice: LogAction: * Stop p_mysql_001 ( 001db01a ) due to colocation with p_vip_clust01
Feb 16 14:06:39 [3912] 001db01a pengine: notice: LogAction: * Stop p_mysql_000 ( 001db01a ) due to colocation with p_vip_clust01
Feb 16 14:06:39 [3912] 001db01a pengine: info: LogActions: Leave p_mysql_002 (Started unmanaged)
Feb 16 14:06:39 [3912] 001db01a pengine: notice: LogAction: * Stop p_mysql_003 ( 001db01a ) due to colocation with p_vip_clust01
Feb 16 14:06:39 [3912] 001db01a pengine: notice: LogAction: * Stop p_mysql_004 ( 001db01a ) due to colocation with p_vip_clust01
Feb 16 14:06:39 [3912] 001db01a pengine: notice: LogAction: * Stop p_mysql_005 ( 001db01a ) due to colocation with p_vip_clust01
Feb 16 14:06:39 [3912] 001db01a pengine: info: LogActions: Leave p_mysql_006 (Started 001db01b)
Feb 16 14:06:39 [3912] 001db01a pengine: info: LogActions: Leave p_mysql_007 (Started 001db01b)
Feb 16 14:06:39 [3912] 001db01a pengine: info: LogActions: Leave p_mysql_008 (Started 001db01b)
Feb 16 14:06:39 [3912] 001db01a pengine: notice: LogAction: * Stop p_mysql_622 ( 001db01a ) due to colocation with p_vip_clust01
Feb 16 14:06:39 [3912] 001db01a pengine: notice: process_pe_message: Calculated transition 17791, saving inputs in /var/lib/pacemaker/pengine/pe-input-482.bz2
Feb 16 14:06:39 [3913] 001db01a crmd: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
Feb 16 14:06:39 [3913] 001db01a crmd: info: do_te_invoke: Processing graph 17791 (ref=pe_calc-dc-1550343999-19548) derived from /var/lib/pacemaker/pengine/pe-input-482.bz2
Feb 16 14:06:39 [3913] 001db01a crmd: notice: te_rsc_command: Initiating cancel operation p_drbd0_monitor_60000 on 001db01b | action 6
Feb 16 14:06:39 [3913] 001db01a crmd: notice: te_rsc_command: Initiating stop operation p_mysql_001_stop_0 locally on 001db01a | action 89
Feb 16 14:06:39 [3910] 001db01a lrmd: info: cancel_recurring_action: Cancelling lsb operation p_mysql_001_status_15000
Feb 16 14:06:39 [3913] 001db01a crmd: info: do_lrm_rsc_op: Performing key=89:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5 op=p_mysql_001_stop_0
Feb 16 14:06:39 [3910] 001db01a lrmd: info: log_execute: executing - rsc:p_mysql_001 action:stop call_id:1096
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/20022)
Feb 16 14:06:39 [3913] 001db01a crmd: notice: te_rsc_command: Initiating stop operation p_mysql_000_stop_0 locally on 001db01a | action 92
Feb 16 14:06:39 [3910] 001db01a lrmd: info: cancel_recurring_action: Cancelling lsb operation p_mysql_000_status_15000
Feb 16 14:06:39 [3913] 001db01a crmd: info: do_lrm_rsc_op: Performing key=92:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5 op=p_mysql_000_stop_0
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff: --- 0.346.4 2
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff: +++ 0.346.5 (null)
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: + /cib: @num_updates=5
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_001']/lrm_rsc_op[@id='p_mysql_001_last_0']: @operation_key=p_mysql_001_stop_0, @operation=stop, @transition-key=89:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @transition-magic=-1:193;89:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @call-id=-1, @rc-code=193, @op-status=-1, @last-run=1550343999, @last-rc-change=1550343999, @exec-time=0
Feb 16 14:06:39 [3910] 001db01a lrmd: info: log_execute: executing - rsc:p_mysql_000 action:stop call_id:1098
Feb 16 14:06:39 [3913] 001db01a crmd: notice: te_rsc_command: Initiating stop operation p_mysql_003_stop_0 locally on 001db01a | action 96
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=001db01a/crmd/20022, version=0.346.5)
Feb 16 14:06:39 [3910] 001db01a lrmd: info: cancel_recurring_action: Cancelling lsb operation p_mysql_003_status_15000
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff: --- 0.346.5 2
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff: +++ 0.346.6 (null)
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='p_drbd0']/lrm_rsc_op[@id='p_drbd0_monitor_60000']
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: + /cib: @num_updates=6
Feb 16 14:06:39 [3913] 001db01a crmd: info: do_lrm_rsc_op: Performing key=96:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5 op=p_mysql_003_stop_0
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_process_request: Completed cib_delete operation for section status: OK (rc=0, origin=001db01b/crmd/561, version=0.346.6)
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/20023)
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/20024)
Feb 16 14:06:39 [3910] 001db01a lrmd: info: log_execute: executing - rsc:p_mysql_003 action:stop call_id:1100
Feb 16 14:06:39 [3913] 001db01a crmd: notice: te_rsc_command: Initiating stop operation p_mysql_004_stop_0 locally on 001db01a | action 99
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff: --- 0.346.6 2
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff: +++ 0.346.7 (null)
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: + /cib: @num_updates=7
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_000']/lrm_rsc_op[@id='p_mysql_000_last_0']: @operation_key=p_mysql_000_stop_0, @operation=stop, @transition-key=92:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @transition-magic=-1:193;92:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @call-id=-1, @rc-code=193, @op-status=-1, @last-run=1550343999, @last-rc-change=1550343999, @exec-time=0
Feb 16 14:06:39 [3910] 001db01a lrmd: info: cancel_recurring_action: Cancelling lsb operation p_mysql_004_status_15000
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=001db01a/crmd/20023, version=0.346.7)
Feb 16 14:06:39 [3913] 001db01a crmd: info: do_lrm_rsc_op: Performing key=99:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5 op=p_mysql_004_stop_0
Feb 16 14:06:39 [3910] 001db01a lrmd: info: log_execute: executing - rsc:p_mysql_004 action:stop call_id:1102
Feb 16 14:06:39 [3913] 001db01a crmd: notice: te_rsc_command: Initiating stop operation p_mysql_005_stop_0 locally on 001db01a | action 102
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff: --- 0.346.7 2
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff: +++ 0.346.8 (null)
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: + /cib: @num_updates=8
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_003']/lrm_rsc_op[@id='p_mysql_003_last_0']: @operation_key=p_mysql_003_stop_0, @operation=stop, @transition-key=96:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @transition-magic=-1:193;96:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @call-id=-1, @rc-code=193, @op-status=-1, @last-run=1550343999, @last-rc-change=1550343999, @exec-time=0
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=001db01a/crmd/20024, version=0.346.8)
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/20025)
Feb 16 14:06:39 [3910] 001db01a lrmd: info: cancel_recurring_action: Cancelling lsb operation p_mysql_005_status_15000
Feb 16 14:06:39 [3913] 001db01a crmd: info: do_lrm_rsc_op: Performing key=102:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5 op=p_mysql_005_stop_0
Feb 16 14:06:39 [3910] 001db01a lrmd: info: log_execute: executing - rsc:p_mysql_005 action:stop call_id:1104
Feb 16 14:06:39 [3913] 001db01a crmd: notice: te_rsc_command: Initiating stop operation p_mysql_622_stop_0 locally on 001db01a | action 111
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff: --- 0.346.8 2
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff: +++ 0.346.9 (null)
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: + /cib: @num_updates=9
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_004']/lrm_rsc_op[@id='p_mysql_004_last_0']: @operation_key=p_mysql_004_stop_0, @operation=stop, @transition-key=99:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @transition-magic=-1:193;99:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @call-id=-1, @rc-code=193, @op-status=-1, @last-run=1550343999, @last-rc-change=1550343999, @exec-time=0
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=001db01a/crmd/20025, version=0.346.9)
Feb 16 14:06:39 [3910] 001db01a lrmd: info: cancel_recurring_action: Cancelling lsb operation p_mysql_622_status_15000
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/20026)
Feb 16 14:06:39 [3913] 001db01a crmd: info: do_lrm_rsc_op: Performing key=111:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5 op=p_mysql_622_stop_0
Feb 16 14:06:39 [3910] 001db01a lrmd: info: log_execute: executing - rsc:p_mysql_622 action:stop call_id:1106
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff: --- 0.346.9 2
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff: +++ 0.346.10 (null)
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: + /cib: @num_updates=10
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_005']/lrm_rsc_op[@id='p_mysql_005_last_0']: @operation_key=p_mysql_005_stop_0, @operation=stop, @transition-key=102:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @transition-magic=-1:193;102:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @call-id=-1, @rc-code=193, @op-status=-1, @last-run=1550343999, @last-rc-change=1550343999, @exec-time=0
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=001db01a/crmd/20026, version=0.346.10)
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/20027)
Feb 16 14:06:39 [3913] 001db01a crmd: info: te_update_diff: Cancellation of p_drbd0_monitor_60000 on 2 confirmed (6)
Feb 16 14:06:39 [3913] 001db01a crmd: info: process_lrm_event: Result of monitor operation for p_mysql_001 on 001db01a: Cancelled | call=1086 key=p_mysql_001_monitor_15000 confirmed=true
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff: --- 0.346.10 2
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff: +++ 0.346.11 (null)
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: + /cib: @num_updates=11
Feb 16 14:06:39 [3913] 001db01a crmd: info: process_lrm_event: Result of monitor operation for p_mysql_000 on 001db01a: Cancelled | call=1087 key=p_mysql_000_monitor_15000 confirmed=true
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_622']/lrm_rsc_op[@id='p_mysql_622_last_0']: @operation_key=p_mysql_622_stop_0, @operation=stop, @transition-key=111:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @transition-magic=-1:193;111:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @call-id=-1, @rc-code=193, @op-status=-1, @last-run=1550343999, @last-rc-change=1550343999, @exec-time=0
Feb 16 14:06:39 [3913] 001db01a crmd: info: process_lrm_event: Result of monitor operation for p_mysql_003 on 001db01a: Cancelled | call=1089 key=p_mysql_003_monitor_15000 confirmed=true
Feb 16 14:06:39 [3913] 001db01a crmd: info: process_lrm_event: Result of monitor operation for p_mysql_004 on 001db01a: Cancelled | call=1090 key=p_mysql_004_monitor_15000 confirmed=true
Feb 16 14:06:39 [3913] 001db01a crmd: info: process_lrm_event: Result of monitor operation for p_mysql_005 on 001db01a: Cancelled | call=1091 key=p_mysql_005_monitor_15000 confirmed=true
Feb 16 14:06:39 [3913] 001db01a crmd: info: process_lrm_event: Result of monitor operation for p_mysql_622 on 001db01a: Cancelled | call=1092 key=p_mysql_622_monitor_15000 confirmed=true
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=001db01a/crmd/20027, version=0.346.11)
Feb 16 14:06:39 [3913] 001db01a crmd: notice: te_rsc_command: Initiating notify operation p_drbd0_pre_notify_demote_0 on 001db01b | action 125
Feb 16 14:06:39 [3913] 001db01a crmd: notice: te_rsc_command: Initiating notify operation p_drbd0_pre_notify_demote_0 locally on 001db01a | action 127
Feb 16 14:06:39 [3913] 001db01a crmd: info: do_lrm_rsc_op: Performing key=127:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5 op=p_drbd0_notify_0
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/20028)
Feb 16 14:06:39 [3910] 001db01a lrmd: info: log_execute: executing - rsc:p_drbd0 action:notify call_id:1107
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff: --- 0.346.11 2
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff: +++ 0.346.12 (null)
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: + /cib: @num_updates=12
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_drbd0']/lrm_rsc_op[@id='p_drbd0_pre_notify_demote_0']: @transition-key=127:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @transition-magic=-1:193;127:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @last-run=1550343999, @last-rc-change=1550343999
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=001db01a/crmd/20028, version=0.346.12)
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff: --- 0.346.12 2
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: Diff: +++ 0.346.13 (null)
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: + /cib: @num_updates=13
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_perform_op: + /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='p_drbd0']/lrm_rsc_op[@id='p_drbd0_pre_notify_demote_0']: @transition-key=125:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @transition-magic=-1:193;125:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @last-run=1550343999, @last-rc-change=1550343999
Feb 16 14:06:39 [3908] 001db01a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=001db01b/crmd/562, version=0.346.13)
Feb 16 14:06:39 [3913] 001db01a crmd: info: match_graph_event: Action p_drbd0_notify_0 (125) confirmed on 001db01b (rc=0)
Feb 16 14:06:39 [3910] 001db01a lrmd: info: log_finished: finished - rsc:p_drbd0 action:notify call_id:1107 pid:36727 exit-code:0 exec-time:53ms queue-time:0ms
Feb 16 14:06:39 [3913] 001db01a crmd: info: match_graph_event: Action p_drbd0_notify_0 (127) confirmed on 001db01a (rc=0)
Feb 16 14:06:39 [3913] 001db01a crmd: notice: process_lrm_event: Result of notify operation for p_drbd0 on 001db01a: 0 (ok) | call=1107 key=p_drbd0_notify_0 confirmed=true cib-update=0
Feb 16 14:06:44 [3908] 001db01a cib: info: cib_process_ping: Reporting our current digest to 001db01a: 10a59a3eddb032581416a9dc95da7ade for 0.346.13 (0x55ac23ea4ef0 0)
Feb 16 14:06:44 [3910] 001db01a lrmd: info: log_finished: finished - rsc:p_mysql_622 action:stop call_id:1106 pid:36712 exit-code:0 exec-time:5082ms queue-time:0ms
Feb 16 14:06:44 [3908] 001db01a cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/20029)
Feb 16 14:06:44 [3913] 001db01a crmd: notice: process_lrm_event: Result of stop operation for p_mysql_622 on 001db01a: 0 (ok) | call=1106 key=p_mysql_622_stop_0 confirmed=true cib-update=20029
Feb 16 14:06:44 [3908] 001db01a cib: info: cib_perform_op: Diff: --- 0.346.13 2
Feb 16 14:06:44 [3908] 001db01a cib: info: cib_perform_op: Diff: +++ 0.346.14 (null)
Feb 16 14:06:44 [3908] 001db01a cib: info: cib_perform_op: + /cib: @num_updates=14
Feb 16 14:06:44 [3908] 001db01a cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_622']/lrm_rsc_op[@id='p_mysql_622_last_0']: @transition-magic=0:0;111:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @call-id=1106, @rc-code=0, @op-status=0, @exec-time=5082
Feb 16 14:06:44 [3908] 001db01a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=001db01a/crmd/20029, version=0.346.14)
Feb 16 14:06:44 [3913] 001db01a crmd: info: match_graph_event: Action p_mysql_622_stop_0 (111) confirmed on 001db01a (rc=0)
Feb 16 14:06:44 [3910] 001db01a lrmd: info: log_finished: finished - rsc:p_mysql_000 action:stop call_id:1098 pid:36695 exit-code:0 exec-time:5093ms queue-time:0ms
Feb 16 14:06:44 [3908] 001db01a cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/20030)
Feb 16 14:06:44 [3913] 001db01a crmd: notice: process_lrm_event: Result of stop operation for p_mysql_000 on 001db01a: 0 (ok) | call=1098 key=p_mysql_000_stop_0 confirmed=true cib-update=20030
Feb 16 14:06:44 [3908] 001db01a cib: info: cib_perform_op: Diff: --- 0.346.14 2
Feb 16 14:06:44 [3908] 001db01a cib: info: cib_perform_op: Diff: +++ 0.346.15 (null)
Feb 16 14:06:44 [3908] 001db01a cib: info: cib_perform_op: + /cib: @num_updates=15
Feb 16 14:06:44 [3908] 001db01a cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_000']/lrm_rsc_op[@id='p_mysql_000_last_0']: @transition-magic=0:0;92:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @call-id=1098, @rc-code=0, @op-status=0, @exec-time=5093
Feb 16 14:06:44 [3908] 001db01a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=001db01a/crmd/20030, version=0.346.15)
Feb 16 14:06:44 [3913] 001db01a crmd: info: match_graph_event: Action p_mysql_000_stop_0 (92) confirmed on 001db01a (rc=0)
Feb 16 14:06:49 [3908] 001db01a cib: info: cib_process_ping: Reporting our current digest to 001db01a: 1bc996395d5a4f64ed405bf122c8e890 for 0.346.15 (0x55ac23ea4ef0 0)
Feb 16 14:06:51 [3910] 001db01a lrmd: info: log_finished: finished - rsc:p_mysql_003 action:stop call_id:1100 pid:36696 exit-code:0 exec-time:12102ms queue-time:0ms
Feb 16 14:06:51 [3910] 001db01a lrmd: info: log_finished: finished - rsc:p_mysql_004 action:stop call_id:1102 pid:36701 exit-code:0 exec-time:12101ms queue-time:0ms
Feb 16 14:06:51 [3910] 001db01a lrmd: info: log_finished: finished - rsc:p_mysql_005 action:stop call_id:1104 pid:36705 exit-code:0 exec-time:12100ms queue-time:0ms
Feb 16 14:06:51 [3910] 001db01a lrmd: info: log_finished: finished - rsc:p_mysql_001 action:stop call_id:1096 pid:36694 exit-code:0 exec-time:12107ms queue-time:0ms
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/20031)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: process_lrm_event: Result of stop operation for p_mysql_003 on 001db01a: 0 (ok) | call=1100 key=p_mysql_003_stop_0 confirmed=true cib-update=20031
Feb 16 14:06:51 [3913] 001db01a crmd: notice: process_lrm_event: Result of stop operation for p_mysql_004 on 001db01a: 0 (ok) | call=1102 key=p_mysql_004_stop_0 confirmed=true cib-update=20032
Feb 16 14:06:51 [3913] 001db01a crmd: notice: process_lrm_event: Result of stop operation for p_mysql_005 on 001db01a: 0 (ok) | call=1104 key=p_mysql_005_stop_0 confirmed=true cib-update=20033
Feb 16 14:06:51 [3913] 001db01a crmd: notice: process_lrm_event: Result of stop operation for p_mysql_001 on 001db01a: 0 (ok) | call=1096 key=p_mysql_001_stop_0 confirmed=true cib-update=20034
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_perform_op: Diff: --- 0.346.15 2
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_perform_op: Diff: +++ 0.346.16 (null)
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_perform_op: + /cib: @num_updates=16
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_003']/lrm_rsc_op[@id='p_mysql_003_last_0']: @transition-magic=0:0;96:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @call-id=1100, @rc-code=0, @op-status=0, @exec-time=12102
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=001db01a/crmd/20031, version=0.346.16)
Feb 16 14:06:51 [3913] 001db01a crmd: info: match_graph_event: Action p_mysql_003_stop_0 (96) confirmed on 001db01a (rc=0)
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/20032)
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/20033)
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/20034)
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_perform_op: Diff: --- 0.346.16 2
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_perform_op: Diff: +++ 0.346.17 (null)
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_perform_op: + /cib: @num_updates=17
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_004']/lrm_rsc_op[@id='p_mysql_004_last_0']: @transition-magic=0:0;99:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @call-id=1102, @rc-code=0, @op-status=0, @exec-time=12101
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=001db01a/crmd/20032, version=0.346.17)
Feb 16 14:06:51 [3913] 001db01a crmd: info: match_graph_event: Action p_mysql_004_stop_0 (99) confirmed on 001db01a (rc=0)
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_perform_op: Diff: --- 0.346.17 2
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_perform_op: Diff: +++ 0.346.18 (null)
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_perform_op: + /cib: @num_updates=18
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_005']/lrm_rsc_op[@id='p_mysql_005_last_0']: @transition-magic=0:0;102:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @call-id=1104, @rc-code=0, @op-status=0, @exec-time=12100
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=001db01a/crmd/20033, version=0.346.18)
Feb 16 14:06:51 [3913] 001db01a crmd: info: match_graph_event: Action p_mysql_005_stop_0 (102) confirmed on 001db01a (rc=0)
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_perform_op: Diff: --- 0.346.18 2
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_perform_op: Diff: +++ 0.346.19 (null)
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_perform_op: + /cib: @num_updates=19
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_perform_op: + /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_001']/lrm_rsc_op[@id='p_mysql_001_last_0']: @transition-magic=0:0;89:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @call-id=1096, @rc-code=0, @op-status=0, @exec-time=12107
Feb 16 14:06:51 [3908] 001db01a cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=001db01a/crmd/20034, version=0.346.19)
Feb 16 14:06:51 [3913] 001db01a crmd: info: match_graph_event: Action p_mysql_001_stop_0 (89) confirmed on 001db01a (rc=0)
Feb 16 14:06:51 [3913] 001db01a crmd: warning: run_graph: Transition 17791 (Complete=11, Pending=0, Fired=0, Skipped=0, Incomplete=28, Source=/var/lib/pacemaker/pengine/pe-input-482.bz2): Terminated
Feb 16 14:06:51 [3913] 001db01a crmd: warning: te_graph_trigger: Transition failed: terminated
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_graph: Graph 17791 with 39 actions: batch-limit=0 jobs, network-delay=60000ms
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action 126]: Pending rsc op p_drbd0_post_notify_demote_0 on 001db01b (priority: 1000000, waiting: 50)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action 125]: Completed rsc op p_drbd0_pre_notify_demote_0 on 001db01b (priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action 122]: Pending rsc op p_drbd0_post_notify_promote_0 on 001db01b (priority: 1000000, waiting: 44)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action 121]: Pending rsc op p_drbd0_pre_notify_promote_0 on 001db01b (priority: 0, waiting: 42)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action 23]: Pending rsc op p_drbd0_promote_0 on 001db01b (priority: 0, waiting: 40)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action 6]: Completed rsc op p_drbd0_monitor_60000 on 001db01b (priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action 128]: Pending rsc op p_drbd0_post_notify_demote_0 on 001db01a (priority: 1000000, waiting: 50)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action 127]: Completed rsc op p_drbd0_pre_notify_demote_0 on 001db01a (priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action 124]: Pending rsc op p_drbd0_post_notify_promote_0 on 001db01a (priority: 1000000, waiting: 44)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action 123]: Pending rsc op p_drbd0_pre_notify_promote_0 on 001db01a (priority: 0, waiting: 42)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action 27]: Pending rsc op p_drbd0_monitor_60000 on 001db01a (priority: 0, waiting: 25 45 51)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action 25]: Pending rsc op p_drbd0_demote_0 on 001db01a (priority: 0, waiting: 46)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action 51]: Pending pseudo op ms_drbd0_confirmed-post_notify_demoted_0 on N/A (priority: 1000000, waiting: 50 126 128)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action 50]: Pending pseudo op ms_drbd0_post_notify_demoted_0 on N/A (priority: 1000000, waiting: 47)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action 49]: Completed pseudo op ms_drbd0_confirmed-pre_notify_demote_0 on N/A (priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action 48]: Completed pseudo op ms_drbd0_pre_notify_demote_0 on N/A (priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action 47]: Pending pseudo op ms_drbd0_demoted_0 on N/A (priority: 1000000, waiting: 25 46)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action 46]: Pending pseudo op ms_drbd0_demote_0 on N/A (priority: 0, waiting: 82)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action 45]: Pending pseudo op ms_drbd0_confirmed-post_notify_promoted_0 on N/A (priority: 1000000, waiting: 44 122 124)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action 44]: Pending pseudo op ms_drbd0_post_notify_promoted_0 on N/A (priority: 1000000, waiting: 41 43)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action 43]: Pending pseudo op ms_drbd0_confirmed-pre_notify_promote_0 on N/A (priority: 0, waiting: 42 121 123)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action 42]: Pending pseudo op ms_drbd0_pre_notify_promote_0 on N/A (priority: 0, waiting: 51)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action 41]: Pending pseudo op ms_drbd0_promoted_0 on N/A (priority: 1000000, waiting: 23)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action 40]: Pending pseudo op ms_drbd0_promote_0 on N/A (priority: 0, waiting: 43 47)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action 84]: Pending rsc op p_fs_clust01_monitor_20000 on 001db01b (priority: 0, waiting: 83)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action 83]: Pending rsc op p_fs_clust01_start_0 on 001db01b (priority: 0, waiting: 45 82)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action 82]: Pending rsc op p_fs_clust01_stop_0 on 001db01a (priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: * [Input 18]: Unresolved dependency rsc op p_vip_clust01_stop_0 on 001db01a
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action 91]: Pending rsc op p_mysql_001_monitor_15000 on 001db01b (priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: * [Input 90]: Unresolved dependency rsc op p_mysql_001_start_0 on 001db01b
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action 89]: Completed rsc op p_mysql_001_stop_0 on 001db01a (priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action 94]: Pending rsc op p_mysql_000_monitor_15000 on 001db01b (priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: * [Input 93]: Unresolved dependency rsc op p_mysql_000_start_0 on 001db01b
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action 92]: Completed rsc op p_mysql_000_stop_0 on 001db01a (priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action 98]: Pending rsc op p_mysql_003_monitor_15000 on 001db01b (priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: * [Input 97]: Unresolved dependency rsc op p_mysql_003_start_0 on 001db01b
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action 96]: Completed rsc op p_mysql_003_stop_0 on 001db01a (priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action 101]: Pending rsc op p_mysql_004_monitor_15000 on 001db01b (priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: * [Input 100]: Unresolved dependency rsc op p_mysql_004_start_0 on 001db01b
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action 99]: Completed rsc op p_mysql_004_stop_0 on 001db01a (priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action 104]: Pending rsc op p_mysql_005_monitor_15000 on 001db01b (priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: * [Input 103]: Unresolved dependency rsc op p_mysql_005_start_0 on 001db01b
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action 102]: Completed rsc op p_mysql_005_stop_0 on 001db01a (priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action 113]: Pending rsc op p_mysql_622_monitor_15000 on 001db01b (priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: * [Input 112]: Unresolved dependency rsc op p_mysql_622_start_0 on 001db01b
Feb 16 14:06:51 [3913] 001db01a crmd: notice: print_synapse: [Action 111]: Completed rsc op p_mysql_622_stop_0 on 001db01a (priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a crmd: info: do_log: Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd
Feb 16 14:06:51 [3913] 001db01a crmd: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd
Feb 16 14:06:56 [3908] 001db01a cib: info: cib_process_ping: Reporting our current digest to 001db01a: cf6fae5210d45a88d51e78fe8acb1178 for 0.346.19 (0x55ac23ea4ef0 0)
> -----Original Message-----
> From: Users <users-bounces at clusterlabs.org> On Behalf Of Eric Robinson
> Sent: Saturday, February 16, 2019 12:51 PM
> To: Cluster Labs - All topics related to open-source clustering welcomed
> <users at clusterlabs.org>
> Subject: Re: [ClusterLabs] Why Do All The Services Go Down When Just One
> Fails?
>
> Hi Valentin --
>
> Which logs? You mean /var/log/cluster/corosync.log?
>
> But even if the stop action is resulting in an error, why would the cluster also
> try to stop the other services which are not dependent?
>
> > -----Original Message-----
> > From: Users <users-bounces at clusterlabs.org> On Behalf Of Valentin
> > Vidic
> > Sent: Saturday, February 16, 2019 12:44 PM
> > To: users at clusterlabs.org
> > Subject: Re: [ClusterLabs] Why Do All The Services Go Down When Just
> > One Fails?
> >
> > On Sat, Feb 16, 2019 at 08:34:21PM +0000, Eric Robinson wrote:
> > > Why is it that when one of the resources that start with p_mysql_*
> > > goes into a FAILED state, all the other MySQL services also stop?
> >
> > Perhaps stop is not working correctly for these lsb services, so for
> > example stopping lsb:mysql_004 also stops the other lsb:mysql_nnn.
> >
> > You would need to send the logs from the event to confirm this.
> >
> > --
> > Valentin
> > _______________________________________________
> > Users mailing list: Users at clusterlabs.org
> > https://lists.clusterlabs.org/mailman/listinfo/users
> >
> > Project Home: http://www.clusterlabs.org Getting started:
> > http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > Bugs: http://bugs.clusterlabs.org
> _______________________________________________
> Users mailing list: Users at clusterlabs.org
> https://lists.clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org Getting started:
> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
More information about the Users
mailing list