[ClusterLabs] Why Do All The Services Go Down When Just One Fails?

Eric Robinson eric.robinson at psmnv.com
Sat Feb 16 16:03:43 EST 2019


Here are the relevant corosync logs.

It appears that the stop action for resource p_mysql_002 failed, and that caused a cascading series of service changes. However, I don't understand why, since no other resources are dependent on p_mysql_002.

[root at 001db01a cluster]# cat corosync_filtered.log
Feb 16 14:06:24 [3908] 001db01a        cib:     info: cib_process_request:      Forwarding cib_apply_diff operation for section 'all' to all (origin=local/cibadmin/2)
Feb 16 14:06:24 [3908] 001db01a        cib:     info: cib_perform_op:   Diff: --- 0.345.30 2
Feb 16 14:06:24 [3908] 001db01a        cib:     info: cib_perform_op:   Diff: +++ 0.346.0 cc0da1b030418ec8b7c72db1115e2af1
Feb 16 14:06:24 [3908] 001db01a        cib:     info: cib_perform_op:   +  /cib:  @epoch=346, @num_updates=0
Feb 16 14:06:24 [3908] 001db01a        cib:     info: cib_perform_op:   ++ /cib/configuration/resources/primitive[@id='p_mysql_002']:  <meta_attributes id="p_mysql_002-meta_attributes"/>
Feb 16 14:06:24 [3908] 001db01a        cib:     info: cib_perform_op:   ++                                                               <nvpair id="p_mysql_002-meta_attributes-target-role" name="target-role" value="Stopped"/>
Feb 16 14:06:24 [3908] 001db01a        cib:     info: cib_perform_op:   ++                                                             </meta_attributes>
Feb 16 14:06:24 [3908] 001db01a        cib:     info: cib_process_request:      Completed cib_apply_diff operation for section 'all': OK (rc=0, origin=001db01a/cibadmin/2, version=0.346.0)
Feb 16 14:06:24 [3913] 001db01a       crmd:     info: abort_transition_graph:   Transition aborted by meta_attributes.p_mysql_002-meta_attributes 'create': Configuration change | cib=0.346.0 source=te_update_diff:456 path=/cib/configuration/resources/primitive[@id='p_mysql_002'] complete=true
Feb 16 14:06:24 [3913] 001db01a       crmd:   notice: do_state_transition:      State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph
Feb 16 14:06:24 [3912] 001db01a    pengine:   notice: unpack_config:    On loss of CCM Quorum: Ignore
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: determine_online_status:  Node 001db01b is online
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: determine_online_status:  Node 001db01a is online
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: determine_op_status:      Operation monitor found resource p_drbd0:0 active in master mode on 001db01b
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: determine_op_status:      Operation monitor found resource p_drbd1:0 active on 001db01b
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: determine_op_status:      Operation monitor found resource p_mysql_004 active on 001db01a
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: determine_op_status:      Operation monitor found resource p_mysql_005 active on 001db01a
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: determine_op_status:      Operation monitor found resource p_drbd0:1 active on 001db01a
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: determine_op_status:      Operation monitor found resource p_drbd1:1 active on 001db01a
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: determine_op_status:      Operation monitor found resource p_mysql_001 active on 001db01a
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: determine_op_status:      Operation monitor found resource p_mysql_002 active on 001db01a
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: determine_op_status:      Operation monitor found resource p_mysql_002 active on 001db01a
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: unpack_node_loop: Node 2 is already processed
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: unpack_node_loop: Node 1 is already processed
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: unpack_node_loop: Node 2 is already processed
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: unpack_node_loop: Node 1 is already processed
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: common_print:     p_vip_clust01   (ocf::heartbeat:IPaddr2):       Started 001db01a
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: clone_print:       Master/Slave Set: ms_drbd0 [p_drbd0]
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: short_print:           Masters: [ 001db01a ]
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: short_print:           Slaves: [ 001db01b ]
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: clone_print:       Master/Slave Set: ms_drbd1 [p_drbd1]
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: short_print:           Masters: [ 001db01b ]
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: short_print:           Slaves: [ 001db01a ]
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: common_print:     p_fs_clust01    (ocf::heartbeat:Filesystem):    Started 001db01a
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: common_print:     p_fs_clust02    (ocf::heartbeat:Filesystem):    Started 001db01b
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: common_print:     p_vip_clust02   (ocf::heartbeat:IPaddr2):       Started 001db01b
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: common_print:     p_mysql_001     (lsb:mysql_001):        Started 001db01a
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: common_print:     p_mysql_000     (lsb:mysql_000):        Started 001db01a
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: common_print:     p_mysql_002     (lsb:mysql_002):        Started 001db01a (disabled)
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: common_print:     p_mysql_003     (lsb:mysql_003):        Started 001db01a
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: common_print:     p_mysql_004     (lsb:mysql_004):        Started 001db01a
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: common_print:     p_mysql_005     (lsb:mysql_005):        Started 001db01a
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: common_print:     p_mysql_006     (lsb:mysql_006):        Started 001db01b
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: common_print:     p_mysql_007     (lsb:mysql_007):        Started 001db01b
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: common_print:     p_mysql_008     (lsb:mysql_008):        Started 001db01b
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: common_print:     p_mysql_622     (lsb:mysql_622):        Started 001db01a
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: master_color:     Promoting p_drbd0:1 (Master 001db01a)
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: master_color:     ms_drbd0: Promoted 1 instances of a possible 1 to master
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: master_color:     Promoting p_drbd1:0 (Master 001db01b)
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: master_color:     ms_drbd1: Promoted 1 instances of a possible 1 to master
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: native_color:     Resource p_mysql_002 cannot run anywhere
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: LogActions:       Leave   p_vip_clust01   (Started 001db01a)
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: LogActions:       Leave   p_drbd0:0       (Slave 001db01b)
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: LogActions:       Leave   p_drbd0:1       (Master 001db01a)
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: LogActions:       Leave   p_drbd1:0       (Master 001db01b)
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: LogActions:       Leave   p_drbd1:1       (Slave 001db01a)
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: LogActions:       Leave   p_fs_clust01    (Started 001db01a)
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: LogActions:       Leave   p_fs_clust02    (Started 001db01b)
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: LogActions:       Leave   p_vip_clust02   (Started 001db01b)
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: LogActions:       Leave   p_mysql_001     (Started 001db01a)
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: LogActions:       Leave   p_mysql_000     (Started 001db01a)
Feb 16 14:06:24 [3912] 001db01a    pengine:   notice: LogAction:         * Stop       p_mysql_002       (                   001db01a )   due to node availability
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: LogActions:       Leave   p_mysql_003     (Started 001db01a)
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: LogActions:       Leave   p_mysql_004     (Started 001db01a)
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: LogActions:       Leave   p_mysql_005     (Started 001db01a)
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: LogActions:       Leave   p_mysql_006     (Started 001db01b)
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: LogActions:       Leave   p_mysql_007     (Started 001db01b)
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: LogActions:       Leave   p_mysql_008     (Started 001db01b)
Feb 16 14:06:24 [3912] 001db01a    pengine:     info: LogActions:       Leave   p_mysql_622     (Started 001db01a)
Feb 16 14:06:24 [3912] 001db01a    pengine:   notice: process_pe_message:       Calculated transition 17789, saving inputs in /var/lib/pacemaker/pengine/pe-input-480.bz2
Feb 16 14:06:24 [3913] 001db01a       crmd:     info: do_state_transition:      State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
Feb 16 14:06:24 [3913] 001db01a       crmd:     info: do_te_invoke:     Processing graph 17789 (ref=pe_calc-dc-1550343984-19545) derived from /var/lib/pacemaker/pengine/pe-input-480.bz2
Feb 16 14:06:24 [3913] 001db01a       crmd:   notice: te_rsc_command:   Initiating stop operation p_mysql_002_stop_0 locally on 001db01a | action 90
Feb 16 14:06:24 [3910] 001db01a       lrmd:     info: cancel_recurring_action:  Cancelling lsb operation p_mysql_002_status_15000
Feb 16 14:06:24 [3913] 001db01a       crmd:     info: do_lrm_rsc_op:    Performing key=90:17789:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5 op=p_mysql_002_stop_0
Feb 16 14:06:24 [3910] 001db01a       lrmd:     info: log_execute:      executing - rsc:p_mysql_002 action:stop call_id:1094
Feb 16 14:06:24 [3913] 001db01a       crmd:     info: process_lrm_event:        Result of monitor operation for p_mysql_002 on 001db01a: Cancelled | call=1088 key=p_mysql_002_monitor_15000 confirmed=true
Feb 16 14:06:24 [3908] 001db01a        cib:     info: cib_process_request:      Forwarding cib_modify operation for section status to all (origin=local/crmd/20017)
Feb 16 14:06:24 [3908] 001db01a        cib:     info: cib_perform_op:   Diff: --- 0.346.0 2
Feb 16 14:06:24 [3908] 001db01a        cib:     info: cib_perform_op:   Diff: +++ 0.346.1 (null)
Feb 16 14:06:24 [3908] 001db01a        cib:     info: cib_perform_op:   +  /cib:  @num_updates=1
Feb 16 14:06:24 [3908] 001db01a        cib:     info: cib_perform_op:   +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_002']/lrm_rsc_op[@id='p_mysql_002_last_0']:  @operation_key=p_mysql_002_stop_0, @operation=stop, @transition-key=90:17789:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @transition-magic=-1:193;90:17789:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @call-id=-1, @rc-code=193, @op-status=-1, @last-run=1550343984, @last-rc-change=1550343984, @exec-time=0
Feb 16 14:06:24 [3908] 001db01a        cib:     info: cib_process_request:      Completed cib_modify operation for section status: OK (rc=0, origin=001db01a/crmd/20017, version=0.346.1)
Feb 16 14:06:24 [3908] 001db01a        cib:     info: cib_file_backup:  Archived previous version as /var/lib/pacemaker/cib/cib-72.raw
Feb 16 14:06:24 [3908] 001db01a        cib:     info: cib_file_write_with_digest:       Wrote version 0.346.0 of the CIB to disk (digest: 5efac46096f70afefaf0a677cf4ca823)
Feb 16 14:06:24 [3908] 001db01a        cib:     info: cib_file_write_with_digest:       Reading cluster configuration file /var/lib/pacemaker/cib/cib.cp2XNp (digest: /var/lib/pacemaker/cib/cib.5hNGXZ)
Feb 16 14:06:29 [3908] 001db01a        cib:     info: cib_process_ping: Reporting our current digest to 001db01a: 1064b8ca173ab45545a221e13e4716d8 for 0.346.1 (0x55ac23ea4ef0 0)
Feb 16 14:06:39 [3910] 001db01a       lrmd:  warning: child_timeout_callback:   p_mysql_002_stop_0 process (PID 36067) timed out
Feb 16 14:06:39 [3910] 001db01a       lrmd:  warning: operation_finished:       p_mysql_002_stop_0:36067 - timed out after 15000ms
Feb 16 14:06:39 [3910] 001db01a       lrmd:     info: log_finished:     finished - rsc:p_mysql_002 action:stop call_id:1094 pid:36067 exit-code:1 exec-time:15004ms queue-time:0ms
Feb 16 14:06:39 [3913] 001db01a       crmd:    error: process_lrm_event:        Result of stop operation for p_mysql_002 on 001db01a: Timed Out | call=1094 key=p_mysql_002_stop_0 timeout=15000ms
Feb 16 14:06:39 [3913] 001db01a       crmd:   notice: process_lrm_event:        001db01a-p_mysql_002_stop_0:1094 [ killing mysqld with pid 69229\n.............. ]
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_process_request:      Forwarding cib_modify operation for section status to all (origin=local/crmd/20018)
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   Diff: --- 0.346.1 2
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   Diff: +++ 0.346.2 (null)
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   +  /cib:  @num_updates=2
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_002']/lrm_rsc_op[@id='p_mysql_002_last_0']:  @transition-magic=2:1;90:17789:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @call-id=1094, @rc-code=1, @op-status=2, @exec-time=15004
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_002']/lrm_rsc_op[@id='p_mysql_002_last_failure_0']:  @operation_key=p_mysql_002_stop_0, @operation=stop, @transition-key=90:17789:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @transition-magic=2:1;90:17789:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @call-id=1094, @rc-code=1, @op-status=2, @last-run=1550343984, @last-rc-change=1550343984, @exec-time=15004
Feb 16 14:06:39 [3913] 001db01a       crmd:  warning: status_from_rc:   Action 90 (p_mysql_002_stop_0) on 001db01a failed (target: 0 vs. rc: 1): Error
Feb 16 14:06:39 [3913] 001db01a       crmd:   notice: abort_transition_graph:   Transition aborted by operation p_mysql_002_stop_0 'modify' on 001db01a: Event failed | magic=2:1;90:17789:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5 cib=0.346.2 source=match_graph_event:310 complete=false
Feb 16 14:06:39 [3913] 001db01a       crmd:     info: match_graph_event:        Action p_mysql_002_stop_0 (90) confirmed on 001db01a (rc=1)
Feb 16 14:06:39 [3913] 001db01a       crmd:     info: update_failcount: Updating failcount for p_mysql_002 on 001db01a after failed stop: rc=1 (update=INFINITY, time=1550343999)
Feb 16 14:06:39 [3913] 001db01a       crmd:     info: process_graph_event:      Detected action (17789.90) p_mysql_002_stop_0.1094=unknown error: failed
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_process_request:      Completed cib_modify operation for section status: OK (rc=0, origin=001db01a/crmd/20018, version=0.346.2)
Feb 16 14:06:39 [3913] 001db01a       crmd:  warning: status_from_rc:   Action 90 (p_mysql_002_stop_0) on 001db01a failed (target: 0 vs. rc: 1): Error
Feb 16 14:06:39 [3913] 001db01a       crmd:     info: abort_transition_graph:   Transition aborted by operation p_mysql_002_stop_0 'modify' on 001db01a: Event failed | magic=2:1;90:17789:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5 cib=0.346.2 source=match_graph_event:310 complete=false
Feb 16 14:06:39 [3913] 001db01a       crmd:     info: match_graph_event:        Action p_mysql_002_stop_0 (90) confirmed on 001db01a (rc=1)
Feb 16 14:06:39 [3913] 001db01a       crmd:     info: update_failcount: Updating failcount for p_mysql_002 on 001db01a after failed stop: rc=1 (update=INFINITY, time=1550343999)
Feb 16 14:06:39 [3911] 001db01a      attrd:     info: attrd_peer_update:        Setting fail-count-p_mysql_002#stop_0[001db01a]: (null) -> INFINITY from 001db01a
Feb 16 14:06:39 [3911] 001db01a      attrd:     info: write_attribute:  Sent update 666 with 2 changes for fail-count-p_mysql_002#stop_0, id=<n/a>, set=(null)
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_process_request:      Forwarding cib_modify operation for section status to all (origin=local/attrd/666)
Feb 16 14:06:39 [3911] 001db01a      attrd:     info: attrd_peer_update:        Setting last-failure-p_mysql_002#stop_0[001db01a]: (null) -> 1550343999 from 001db01a
Feb 16 14:06:39 [3913] 001db01a       crmd:     info: process_graph_event:      Detected action (17789.90) p_mysql_002_stop_0.1094=unknown error: failed
Feb 16 14:06:39 [3913] 001db01a       crmd:   notice: run_graph:        Transition 17789 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=1, Source=/var/lib/pacemaker/pengine/pe-input-480.bz2): Complete
Feb 16 14:06:39 [3913] 001db01a       crmd:     info: do_state_transition:      State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd
Feb 16 14:06:39 [3911] 001db01a      attrd:     info: write_attribute:  Sent update 667 with 2 changes for last-failure-p_mysql_002#stop_0, id=<n/a>, set=(null)
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   Diff: --- 0.346.2 2
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   Diff: +++ 0.346.3 (null)
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   +  /cib:  @num_updates=3
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']:  <nvpair id="status-1-fail-count-p_mysql_002.stop_0" name="fail-count-p_mysql_002#stop_0" value="INFINITY"/>
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_process_request:      Completed cib_modify operation for section status: OK (rc=0, origin=001db01a/attrd/666, version=0.346.3)
Feb 16 14:06:39 [3913] 001db01a       crmd:     info: abort_transition_graph:   Transition aborted by status-1-fail-count-p_mysql_002.stop_0 doing create fail-count-p_mysql_002#stop_0=INFINITY: Transient attribute change | cib=0.346.3 source=abort_unless_down:343 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true
Feb 16 14:06:39 [3911] 001db01a      attrd:     info: attrd_cib_callback:       Update 666 for fail-count-p_mysql_002#stop_0: OK (0)
Feb 16 14:06:39 [3911] 001db01a      attrd:     info: attrd_cib_callback:       Update 666 for fail-count-p_mysql_002#stop_0[001db01a]=INFINITY: OK (0)
Feb 16 14:06:39 [3911] 001db01a      attrd:     info: attrd_cib_callback:       Update 666 for fail-count-p_mysql_002#stop_0[001db01b]=(null): OK (0)
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_process_request:      Forwarding cib_modify operation for section status to all (origin=local/attrd/667)
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   Diff: --- 0.346.3 2
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   Diff: +++ 0.346.4 (null)
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   +  /cib:  @num_updates=4
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   ++ /cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1']:  <nvpair id="status-1-last-failure-p_mysql_002.stop_0" name="last-failure-p_mysql_002#stop_0" value="1550343999"/>
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_process_request:      Completed cib_modify operation for section status: OK (rc=0, origin=001db01a/attrd/667, version=0.346.4)
Feb 16 14:06:39 [3911] 001db01a      attrd:     info: attrd_cib_callback:       Update 667 for last-failure-p_mysql_002#stop_0: OK (0)
Feb 16 14:06:39 [3911] 001db01a      attrd:     info: attrd_cib_callback:       Update 667 for last-failure-p_mysql_002#stop_0[001db01a]=1550343999: OK (0)
Feb 16 14:06:39 [3911] 001db01a      attrd:     info: attrd_cib_callback:       Update 667 for last-failure-p_mysql_002#stop_0[001db01b]=(null): OK (0)
Feb 16 14:06:39 [3913] 001db01a       crmd:     info: abort_transition_graph:   Transition aborted by status-1-last-failure-p_mysql_002.stop_0 doing create last-failure-p_mysql_002#stop_0=1550343999: Transient attribute change | cib=0.346.4 source=abort_unless_down:343 path=/cib/status/node_state[@id='1']/transient_attributes[@id='1']/instance_attributes[@id='status-1'] complete=true
Feb 16 14:06:39 [3912] 001db01a    pengine:   notice: unpack_config:    On loss of CCM Quorum: Ignore
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: determine_online_status:  Node 001db01b is online
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: determine_online_status:  Node 001db01a is online
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: determine_op_status:      Operation monitor found resource p_drbd0:0 active in master mode on 001db01b
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: determine_op_status:      Operation monitor found resource p_drbd1:0 active on 001db01b
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: determine_op_status:      Operation monitor found resource p_mysql_004 active on 001db01a
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: determine_op_status:      Operation monitor found resource p_mysql_005 active on 001db01a
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: determine_op_status:      Operation monitor found resource p_drbd0:1 active on 001db01a
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: determine_op_status:      Operation monitor found resource p_drbd1:1 active on 001db01a
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: determine_op_status:      Operation monitor found resource p_mysql_001 active on 001db01a
Feb 16 14:06:39 [3912] 001db01a    pengine:  warning: unpack_rsc_op_failure:    Processing failed op stop for p_mysql_002 on 001db01a: unknown error (1)
Feb 16 14:06:39 [3912] 001db01a    pengine:  warning: unpack_rsc_op_failure:    Processing failed op stop for p_mysql_002 on 001db01a: unknown error (1)
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: native_add_running:       resource p_mysql_002 isn't managed
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: unpack_node_loop: Node 2 is already processed
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: unpack_node_loop: Node 1 is already processed
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: unpack_node_loop: Node 2 is already processed
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: unpack_node_loop: Node 1 is already processed
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: common_print:     p_vip_clust01   (ocf::heartbeat:IPaddr2):       Started 001db01a
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: clone_print:       Master/Slave Set: ms_drbd0 [p_drbd0]
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: short_print:           Masters: [ 001db01a ]
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: short_print:           Slaves: [ 001db01b ]
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: clone_print:       Master/Slave Set: ms_drbd1 [p_drbd1]
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: short_print:           Masters: [ 001db01b ]
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: short_print:           Slaves: [ 001db01a ]
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: common_print:     p_fs_clust01    (ocf::heartbeat:Filesystem):    Started 001db01a
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: common_print:     p_fs_clust02    (ocf::heartbeat:Filesystem):    Started 001db01b
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: common_print:     p_vip_clust02   (ocf::heartbeat:IPaddr2):       Started 001db01b
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: common_print:     p_mysql_001     (lsb:mysql_001):        Started 001db01a
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: common_print:     p_mysql_000     (lsb:mysql_000):        Started 001db01a
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: common_print:     p_mysql_002     (lsb:mysql_002):        FAILED 001db01a (disabled, blocked)
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: common_print:     p_mysql_003     (lsb:mysql_003):        Started 001db01a
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: common_print:     p_mysql_004     (lsb:mysql_004):        Started 001db01a
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: common_print:     p_mysql_005     (lsb:mysql_005):        Started 001db01a
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: common_print:     p_mysql_006     (lsb:mysql_006):        Started 001db01b
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: common_print:     p_mysql_007     (lsb:mysql_007):        Started 001db01b
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: common_print:     p_mysql_008     (lsb:mysql_008):        Started 001db01b
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: common_print:     p_mysql_622     (lsb:mysql_622):        Started 001db01a
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: pe_get_failcount: p_mysql_002 has failed INFINITY times on 001db01a
Feb 16 14:06:39 [3912] 001db01a    pengine:  warning: check_migration_threshold:        Forcing p_mysql_002 away from 001db01a after 1000000 failures (max=1000000)
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: rsc_merge_weights:        p_drbd0:1: Rolling back scores from p_mysql_002
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: master_color:     Promoting p_drbd0:0 (Slave 001db01b)
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: master_color:     ms_drbd0: Promoted 1 instances of a possible 1 to master
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: master_color:     Promoting p_drbd1:0 (Master 001db01b)
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: master_color:     ms_drbd1: Promoted 1 instances of a possible 1 to master
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: native_color:     Unmanaged resource p_mysql_002 allocated to 001db01a: failed
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: RecurringOp:       Start recurring monitor (30s) for p_vip_clust01 on 001db01b
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: RecurringOp:      Cancelling action p_drbd0:0_monitor_60000 (Slave vs. Master)
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: RecurringOp:       Start recurring monitor (60s) for p_drbd0:1 on 001db01a
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: RecurringOp:      Cancelling action p_drbd0:0_monitor_60000 (Slave vs. Master)
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: RecurringOp:       Start recurring monitor (60s) for p_drbd0:1 on 001db01a
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: RecurringOp:       Start recurring monitor (20s) for p_fs_clust01 on 001db01b
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: RecurringOp:       Start recurring monitor (15s) for p_mysql_001 on 001db01b
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: RecurringOp:       Start recurring monitor (15s) for p_mysql_000 on 001db01b
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: RecurringOp:       Start recurring monitor (15s) for p_mysql_003 on 001db01b
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: RecurringOp:       Start recurring monitor (15s) for p_mysql_004 on 001db01b
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: RecurringOp:       Start recurring monitor (15s) for p_mysql_005 on 001db01b
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: RecurringOp:       Start recurring monitor (15s) for p_mysql_622 on 001db01b
Feb 16 14:06:39 [3912] 001db01a    pengine:   notice: LogAction:         * Stop       p_vip_clust01     (                   001db01a )   blocked
Feb 16 14:06:39 [3912] 001db01a    pengine:   notice: LogAction:         * Promote    p_drbd0:0         (   Slave -> Master 001db01b )
Feb 16 14:06:39 [3912] 001db01a    pengine:   notice: LogAction:         * Demote     p_drbd0:1         (   Master -> Slave 001db01a )
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: LogActions:       Leave   p_drbd1:0       (Master 001db01b)
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: LogActions:       Leave   p_drbd1:1       (Slave 001db01a)
Feb 16 14:06:39 [3912] 001db01a    pengine:   notice: LogAction:         * Move       p_fs_clust01      (       001db01a -> 001db01b )
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: LogActions:       Leave   p_fs_clust02    (Started 001db01b)
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: LogActions:       Leave   p_vip_clust02   (Started 001db01b)
Feb 16 14:06:39 [3912] 001db01a    pengine:   notice: LogAction:         * Stop       p_mysql_001       (                   001db01a )   due to colocation with p_vip_clust01
Feb 16 14:06:39 [3912] 001db01a    pengine:   notice: LogAction:         * Stop       p_mysql_000       (                   001db01a )   due to colocation with p_vip_clust01
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: LogActions:       Leave   p_mysql_002     (Started unmanaged)
Feb 16 14:06:39 [3912] 001db01a    pengine:   notice: LogAction:         * Stop       p_mysql_003       (                   001db01a )   due to colocation with p_vip_clust01
Feb 16 14:06:39 [3912] 001db01a    pengine:   notice: LogAction:         * Stop       p_mysql_004       (                   001db01a )   due to colocation with p_vip_clust01
Feb 16 14:06:39 [3912] 001db01a    pengine:   notice: LogAction:         * Stop       p_mysql_005       (                   001db01a )   due to colocation with p_vip_clust01
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: LogActions:       Leave   p_mysql_006     (Started 001db01b)
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: LogActions:       Leave   p_mysql_007     (Started 001db01b)
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: LogActions:       Leave   p_mysql_008     (Started 001db01b)
Feb 16 14:06:39 [3912] 001db01a    pengine:   notice: LogAction:         * Stop       p_mysql_622       (                   001db01a )   due to colocation with p_vip_clust01
Feb 16 14:06:39 [3912] 001db01a    pengine:   notice: process_pe_message:       Calculated transition 17790, saving inputs in /var/lib/pacemaker/pengine/pe-input-481.bz2
Feb 16 14:06:39 [3913] 001db01a       crmd:     info: handle_response:  pe_calc calculation pe_calc-dc-1550343999-19547 is obsolete
Feb 16 14:06:39 [3912] 001db01a    pengine:   notice: unpack_config:    On loss of CCM Quorum: Ignore
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: determine_online_status:  Node 001db01b is online
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: determine_online_status:  Node 001db01a is online
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: determine_op_status:      Operation monitor found resource p_drbd0:0 active in master mode on 001db01b
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: determine_op_status:      Operation monitor found resource p_drbd1:0 active on 001db01b
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: determine_op_status:      Operation monitor found resource p_mysql_004 active on 001db01a
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: determine_op_status:      Operation monitor found resource p_mysql_005 active on 001db01a
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: determine_op_status:      Operation monitor found resource p_drbd0:1 active on 001db01a
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: determine_op_status:      Operation monitor found resource p_drbd1:1 active on 001db01a
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: determine_op_status:      Operation monitor found resource p_mysql_001 active on 001db01a
Feb 16 14:06:39 [3912] 001db01a    pengine:  warning: unpack_rsc_op_failure:    Processing failed op stop for p_mysql_002 on 001db01a: unknown error (1)
Feb 16 14:06:39 [3912] 001db01a    pengine:  warning: unpack_rsc_op_failure:    Processing failed op stop for p_mysql_002 on 001db01a: unknown error (1)
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: native_add_running:       resource p_mysql_002 isn't managed
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: unpack_node_loop: Node 2 is already processed
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: unpack_node_loop: Node 1 is already processed
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: unpack_node_loop: Node 2 is already processed
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: unpack_node_loop: Node 1 is already processed
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: common_print:     p_vip_clust01   (ocf::heartbeat:IPaddr2):       Started 001db01a
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: clone_print:       Master/Slave Set: ms_drbd0 [p_drbd0]
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: short_print:           Masters: [ 001db01a ]
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: short_print:           Slaves: [ 001db01b ]
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: clone_print:       Master/Slave Set: ms_drbd1 [p_drbd1]
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: short_print:           Masters: [ 001db01b ]
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: short_print:           Slaves: [ 001db01a ]
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: common_print:     p_fs_clust01    (ocf::heartbeat:Filesystem):    Started 001db01a
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: common_print:     p_fs_clust02    (ocf::heartbeat:Filesystem):    Started 001db01b
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: common_print:     p_vip_clust02   (ocf::heartbeat:IPaddr2):       Started 001db01b
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: common_print:     p_mysql_001     (lsb:mysql_001):        Started 001db01a
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: common_print:     p_mysql_000     (lsb:mysql_000):        Started 001db01a
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: common_print:     p_mysql_002     (lsb:mysql_002):        FAILED 001db01a (disabled, blocked)
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: common_print:     p_mysql_003     (lsb:mysql_003):        Started 001db01a
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: common_print:     p_mysql_004     (lsb:mysql_004):        Started 001db01a
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: common_print:     p_mysql_005     (lsb:mysql_005):        Started 001db01a
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: common_print:     p_mysql_006     (lsb:mysql_006):        Started 001db01b
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: common_print:     p_mysql_007     (lsb:mysql_007):        Started 001db01b
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: common_print:     p_mysql_008     (lsb:mysql_008):        Started 001db01b
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: common_print:     p_mysql_622     (lsb:mysql_622):        Started 001db01a
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: pe_get_failcount: p_mysql_002 has failed INFINITY times on 001db01a
Feb 16 14:06:39 [3912] 001db01a    pengine:  warning: check_migration_threshold:        Forcing p_mysql_002 away from 001db01a after 1000000 failures (max=1000000)
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: rsc_merge_weights:        p_drbd0:1: Rolling back scores from p_mysql_002
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: master_color:     Promoting p_drbd0:0 (Slave 001db01b)
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: master_color:     ms_drbd0: Promoted 1 instances of a possible 1 to master
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: master_color:     Promoting p_drbd1:0 (Master 001db01b)
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: master_color:     ms_drbd1: Promoted 1 instances of a possible 1 to master
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: native_color:     Unmanaged resource p_mysql_002 allocated to 001db01a: failed
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: RecurringOp:       Start recurring monitor (30s) for p_vip_clust01 on 001db01b
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: RecurringOp:      Cancelling action p_drbd0:0_monitor_60000 (Slave vs. Master)
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: RecurringOp:       Start recurring monitor (60s) for p_drbd0:1 on 001db01a
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: RecurringOp:      Cancelling action p_drbd0:0_monitor_60000 (Slave vs. Master)
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: RecurringOp:       Start recurring monitor (60s) for p_drbd0:1 on 001db01a
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: RecurringOp:       Start recurring monitor (20s) for p_fs_clust01 on 001db01b
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: RecurringOp:       Start recurring monitor (15s) for p_mysql_001 on 001db01b
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: RecurringOp:       Start recurring monitor (15s) for p_mysql_000 on 001db01b
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: RecurringOp:       Start recurring monitor (15s) for p_mysql_003 on 001db01b
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: RecurringOp:       Start recurring monitor (15s) for p_mysql_004 on 001db01b
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: RecurringOp:       Start recurring monitor (15s) for p_mysql_005 on 001db01b
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: RecurringOp:       Start recurring monitor (15s) for p_mysql_622 on 001db01b
Feb 16 14:06:39 [3912] 001db01a    pengine:   notice: LogAction:         * Stop       p_vip_clust01     (                   001db01a )   blocked
Feb 16 14:06:39 [3912] 001db01a    pengine:   notice: LogAction:         * Promote    p_drbd0:0         (   Slave -> Master 001db01b )
Feb 16 14:06:39 [3912] 001db01a    pengine:   notice: LogAction:         * Demote     p_drbd0:1         (   Master -> Slave 001db01a )
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: LogActions:       Leave   p_drbd1:0       (Master 001db01b)
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: LogActions:       Leave   p_drbd1:1       (Slave 001db01a)
Feb 16 14:06:39 [3912] 001db01a    pengine:   notice: LogAction:         * Move       p_fs_clust01      (       001db01a -> 001db01b )
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: LogActions:       Leave   p_fs_clust02    (Started 001db01b)
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: LogActions:       Leave   p_vip_clust02   (Started 001db01b)
Feb 16 14:06:39 [3912] 001db01a    pengine:   notice: LogAction:         * Stop       p_mysql_001       (                   001db01a )   due to colocation with p_vip_clust01
Feb 16 14:06:39 [3912] 001db01a    pengine:   notice: LogAction:         * Stop       p_mysql_000       (                   001db01a )   due to colocation with p_vip_clust01
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: LogActions:       Leave   p_mysql_002     (Started unmanaged)
Feb 16 14:06:39 [3912] 001db01a    pengine:   notice: LogAction:         * Stop       p_mysql_003       (                   001db01a )   due to colocation with p_vip_clust01
Feb 16 14:06:39 [3912] 001db01a    pengine:   notice: LogAction:         * Stop       p_mysql_004       (                   001db01a )   due to colocation with p_vip_clust01
Feb 16 14:06:39 [3912] 001db01a    pengine:   notice: LogAction:         * Stop       p_mysql_005       (                   001db01a )   due to colocation with p_vip_clust01
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: LogActions:       Leave   p_mysql_006     (Started 001db01b)
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: LogActions:       Leave   p_mysql_007     (Started 001db01b)
Feb 16 14:06:39 [3912] 001db01a    pengine:     info: LogActions:       Leave   p_mysql_008     (Started 001db01b)
Feb 16 14:06:39 [3912] 001db01a    pengine:   notice: LogAction:         * Stop       p_mysql_622       (                   001db01a )   due to colocation with p_vip_clust01
Feb 16 14:06:39 [3912] 001db01a    pengine:   notice: process_pe_message:       Calculated transition 17791, saving inputs in /var/lib/pacemaker/pengine/pe-input-482.bz2
Feb 16 14:06:39 [3913] 001db01a       crmd:     info: do_state_transition:      State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
Feb 16 14:06:39 [3913] 001db01a       crmd:     info: do_te_invoke:     Processing graph 17791 (ref=pe_calc-dc-1550343999-19548) derived from /var/lib/pacemaker/pengine/pe-input-482.bz2
Feb 16 14:06:39 [3913] 001db01a       crmd:   notice: te_rsc_command:   Initiating cancel operation p_drbd0_monitor_60000 on 001db01b | action 6
Feb 16 14:06:39 [3913] 001db01a       crmd:   notice: te_rsc_command:   Initiating stop operation p_mysql_001_stop_0 locally on 001db01a | action 89
Feb 16 14:06:39 [3910] 001db01a       lrmd:     info: cancel_recurring_action:  Cancelling lsb operation p_mysql_001_status_15000
Feb 16 14:06:39 [3913] 001db01a       crmd:     info: do_lrm_rsc_op:    Performing key=89:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5 op=p_mysql_001_stop_0
Feb 16 14:06:39 [3910] 001db01a       lrmd:     info: log_execute:      executing - rsc:p_mysql_001 action:stop call_id:1096
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_process_request:      Forwarding cib_modify operation for section status to all (origin=local/crmd/20022)
Feb 16 14:06:39 [3913] 001db01a       crmd:   notice: te_rsc_command:   Initiating stop operation p_mysql_000_stop_0 locally on 001db01a | action 92
Feb 16 14:06:39 [3910] 001db01a       lrmd:     info: cancel_recurring_action:  Cancelling lsb operation p_mysql_000_status_15000
Feb 16 14:06:39 [3913] 001db01a       crmd:     info: do_lrm_rsc_op:    Performing key=92:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5 op=p_mysql_000_stop_0
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   Diff: --- 0.346.4 2
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   Diff: +++ 0.346.5 (null)
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   +  /cib:  @num_updates=5
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_001']/lrm_rsc_op[@id='p_mysql_001_last_0']:  @operation_key=p_mysql_001_stop_0, @operation=stop, @transition-key=89:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @transition-magic=-1:193;89:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @call-id=-1, @rc-code=193, @op-status=-1, @last-run=1550343999, @last-rc-change=1550343999, @exec-time=0
Feb 16 14:06:39 [3910] 001db01a       lrmd:     info: log_execute:      executing - rsc:p_mysql_000 action:stop call_id:1098
Feb 16 14:06:39 [3913] 001db01a       crmd:   notice: te_rsc_command:   Initiating stop operation p_mysql_003_stop_0 locally on 001db01a | action 96
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_process_request:      Completed cib_modify operation for section status: OK (rc=0, origin=001db01a/crmd/20022, version=0.346.5)
Feb 16 14:06:39 [3910] 001db01a       lrmd:     info: cancel_recurring_action:  Cancelling lsb operation p_mysql_003_status_15000
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   Diff: --- 0.346.5 2
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   Diff: +++ 0.346.6 (null)
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   -- /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='p_drbd0']/lrm_rsc_op[@id='p_drbd0_monitor_60000']
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   +  /cib:  @num_updates=6
Feb 16 14:06:39 [3913] 001db01a       crmd:     info: do_lrm_rsc_op:    Performing key=96:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5 op=p_mysql_003_stop_0
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_process_request:      Completed cib_delete operation for section status: OK (rc=0, origin=001db01b/crmd/561, version=0.346.6)
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_process_request:      Forwarding cib_modify operation for section status to all (origin=local/crmd/20023)
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_process_request:      Forwarding cib_modify operation for section status to all (origin=local/crmd/20024)
Feb 16 14:06:39 [3910] 001db01a       lrmd:     info: log_execute:      executing - rsc:p_mysql_003 action:stop call_id:1100
Feb 16 14:06:39 [3913] 001db01a       crmd:   notice: te_rsc_command:   Initiating stop operation p_mysql_004_stop_0 locally on 001db01a | action 99
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   Diff: --- 0.346.6 2
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   Diff: +++ 0.346.7 (null)
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   +  /cib:  @num_updates=7
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_000']/lrm_rsc_op[@id='p_mysql_000_last_0']:  @operation_key=p_mysql_000_stop_0, @operation=stop, @transition-key=92:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @transition-magic=-1:193;92:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @call-id=-1, @rc-code=193, @op-status=-1, @last-run=1550343999, @last-rc-change=1550343999, @exec-time=0
Feb 16 14:06:39 [3910] 001db01a       lrmd:     info: cancel_recurring_action:  Cancelling lsb operation p_mysql_004_status_15000
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_process_request:      Completed cib_modify operation for section status: OK (rc=0, origin=001db01a/crmd/20023, version=0.346.7)
Feb 16 14:06:39 [3913] 001db01a       crmd:     info: do_lrm_rsc_op:    Performing key=99:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5 op=p_mysql_004_stop_0
Feb 16 14:06:39 [3910] 001db01a       lrmd:     info: log_execute:      executing - rsc:p_mysql_004 action:stop call_id:1102
Feb 16 14:06:39 [3913] 001db01a       crmd:   notice: te_rsc_command:   Initiating stop operation p_mysql_005_stop_0 locally on 001db01a | action 102
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   Diff: --- 0.346.7 2
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   Diff: +++ 0.346.8 (null)
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   +  /cib:  @num_updates=8
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_003']/lrm_rsc_op[@id='p_mysql_003_last_0']:  @operation_key=p_mysql_003_stop_0, @operation=stop, @transition-key=96:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @transition-magic=-1:193;96:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @call-id=-1, @rc-code=193, @op-status=-1, @last-run=1550343999, @last-rc-change=1550343999, @exec-time=0
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_process_request:      Completed cib_modify operation for section status: OK (rc=0, origin=001db01a/crmd/20024, version=0.346.8)
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_process_request:      Forwarding cib_modify operation for section status to all (origin=local/crmd/20025)
Feb 16 14:06:39 [3910] 001db01a       lrmd:     info: cancel_recurring_action:  Cancelling lsb operation p_mysql_005_status_15000
Feb 16 14:06:39 [3913] 001db01a       crmd:     info: do_lrm_rsc_op:    Performing key=102:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5 op=p_mysql_005_stop_0
Feb 16 14:06:39 [3910] 001db01a       lrmd:     info: log_execute:      executing - rsc:p_mysql_005 action:stop call_id:1104
Feb 16 14:06:39 [3913] 001db01a       crmd:   notice: te_rsc_command:   Initiating stop operation p_mysql_622_stop_0 locally on 001db01a | action 111
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   Diff: --- 0.346.8 2
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   Diff: +++ 0.346.9 (null)
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   +  /cib:  @num_updates=9
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_004']/lrm_rsc_op[@id='p_mysql_004_last_0']:  @operation_key=p_mysql_004_stop_0, @operation=stop, @transition-key=99:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @transition-magic=-1:193;99:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @call-id=-1, @rc-code=193, @op-status=-1, @last-run=1550343999, @last-rc-change=1550343999, @exec-time=0
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_process_request:      Completed cib_modify operation for section status: OK (rc=0, origin=001db01a/crmd/20025, version=0.346.9)
Feb 16 14:06:39 [3910] 001db01a       lrmd:     info: cancel_recurring_action:  Cancelling lsb operation p_mysql_622_status_15000
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_process_request:      Forwarding cib_modify operation for section status to all (origin=local/crmd/20026)
Feb 16 14:06:39 [3913] 001db01a       crmd:     info: do_lrm_rsc_op:    Performing key=111:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5 op=p_mysql_622_stop_0
Feb 16 14:06:39 [3910] 001db01a       lrmd:     info: log_execute:      executing - rsc:p_mysql_622 action:stop call_id:1106
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   Diff: --- 0.346.9 2
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   Diff: +++ 0.346.10 (null)
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   +  /cib:  @num_updates=10
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_005']/lrm_rsc_op[@id='p_mysql_005_last_0']:  @operation_key=p_mysql_005_stop_0, @operation=stop, @transition-key=102:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @transition-magic=-1:193;102:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @call-id=-1, @rc-code=193, @op-status=-1, @last-run=1550343999, @last-rc-change=1550343999, @exec-time=0
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_process_request:      Completed cib_modify operation for section status: OK (rc=0, origin=001db01a/crmd/20026, version=0.346.10)
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_process_request:      Forwarding cib_modify operation for section status to all (origin=local/crmd/20027)
Feb 16 14:06:39 [3913] 001db01a       crmd:     info: te_update_diff:   Cancellation of p_drbd0_monitor_60000 on 2 confirmed (6)
Feb 16 14:06:39 [3913] 001db01a       crmd:     info: process_lrm_event:        Result of monitor operation for p_mysql_001 on 001db01a: Cancelled | call=1086 key=p_mysql_001_monitor_15000 confirmed=true
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   Diff: --- 0.346.10 2
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   Diff: +++ 0.346.11 (null)
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   +  /cib:  @num_updates=11
Feb 16 14:06:39 [3913] 001db01a       crmd:     info: process_lrm_event:        Result of monitor operation for p_mysql_000 on 001db01a: Cancelled | call=1087 key=p_mysql_000_monitor_15000 confirmed=true
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_622']/lrm_rsc_op[@id='p_mysql_622_last_0']:  @operation_key=p_mysql_622_stop_0, @operation=stop, @transition-key=111:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @transition-magic=-1:193;111:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @call-id=-1, @rc-code=193, @op-status=-1, @last-run=1550343999, @last-rc-change=1550343999, @exec-time=0
Feb 16 14:06:39 [3913] 001db01a       crmd:     info: process_lrm_event:        Result of monitor operation for p_mysql_003 on 001db01a: Cancelled | call=1089 key=p_mysql_003_monitor_15000 confirmed=true
Feb 16 14:06:39 [3913] 001db01a       crmd:     info: process_lrm_event:        Result of monitor operation for p_mysql_004 on 001db01a: Cancelled | call=1090 key=p_mysql_004_monitor_15000 confirmed=true
Feb 16 14:06:39 [3913] 001db01a       crmd:     info: process_lrm_event:        Result of monitor operation for p_mysql_005 on 001db01a: Cancelled | call=1091 key=p_mysql_005_monitor_15000 confirmed=true
Feb 16 14:06:39 [3913] 001db01a       crmd:     info: process_lrm_event:        Result of monitor operation for p_mysql_622 on 001db01a: Cancelled | call=1092 key=p_mysql_622_monitor_15000 confirmed=true
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_process_request:      Completed cib_modify operation for section status: OK (rc=0, origin=001db01a/crmd/20027, version=0.346.11)
Feb 16 14:06:39 [3913] 001db01a       crmd:   notice: te_rsc_command:   Initiating notify operation p_drbd0_pre_notify_demote_0 on 001db01b | action 125
Feb 16 14:06:39 [3913] 001db01a       crmd:   notice: te_rsc_command:   Initiating notify operation p_drbd0_pre_notify_demote_0 locally on 001db01a | action 127
Feb 16 14:06:39 [3913] 001db01a       crmd:     info: do_lrm_rsc_op:    Performing key=127:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5 op=p_drbd0_notify_0
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_process_request:      Forwarding cib_modify operation for section status to all (origin=local/crmd/20028)
Feb 16 14:06:39 [3910] 001db01a       lrmd:     info: log_execute:      executing - rsc:p_drbd0 action:notify call_id:1107
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   Diff: --- 0.346.11 2
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   Diff: +++ 0.346.12 (null)
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   +  /cib:  @num_updates=12
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_drbd0']/lrm_rsc_op[@id='p_drbd0_pre_notify_demote_0']:  @transition-key=127:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @transition-magic=-1:193;127:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @last-run=1550343999, @last-rc-change=1550343999
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_process_request:      Completed cib_modify operation for section status: OK (rc=0, origin=001db01a/crmd/20028, version=0.346.12)
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   Diff: --- 0.346.12 2
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   Diff: +++ 0.346.13 (null)
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   +  /cib:  @num_updates=13
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_perform_op:   +  /cib/status/node_state[@id='2']/lrm[@id='2']/lrm_resources/lrm_resource[@id='p_drbd0']/lrm_rsc_op[@id='p_drbd0_pre_notify_demote_0']:  @transition-key=125:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @transition-magic=-1:193;125:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @last-run=1550343999, @last-rc-change=1550343999
Feb 16 14:06:39 [3908] 001db01a        cib:     info: cib_process_request:      Completed cib_modify operation for section status: OK (rc=0, origin=001db01b/crmd/562, version=0.346.13)
Feb 16 14:06:39 [3913] 001db01a       crmd:     info: match_graph_event:        Action p_drbd0_notify_0 (125) confirmed on 001db01b (rc=0)
Feb 16 14:06:39 [3910] 001db01a       lrmd:     info: log_finished:     finished - rsc:p_drbd0 action:notify call_id:1107 pid:36727 exit-code:0 exec-time:53ms queue-time:0ms
Feb 16 14:06:39 [3913] 001db01a       crmd:     info: match_graph_event:        Action p_drbd0_notify_0 (127) confirmed on 001db01a (rc=0)
Feb 16 14:06:39 [3913] 001db01a       crmd:   notice: process_lrm_event:        Result of notify operation for p_drbd0 on 001db01a: 0 (ok) | call=1107 key=p_drbd0_notify_0 confirmed=true cib-update=0
Feb 16 14:06:44 [3908] 001db01a        cib:     info: cib_process_ping: Reporting our current digest to 001db01a: 10a59a3eddb032581416a9dc95da7ade for 0.346.13 (0x55ac23ea4ef0 0)
Feb 16 14:06:44 [3910] 001db01a       lrmd:     info: log_finished:     finished - rsc:p_mysql_622 action:stop call_id:1106 pid:36712 exit-code:0 exec-time:5082ms queue-time:0ms
Feb 16 14:06:44 [3908] 001db01a        cib:     info: cib_process_request:      Forwarding cib_modify operation for section status to all (origin=local/crmd/20029)
Feb 16 14:06:44 [3913] 001db01a       crmd:   notice: process_lrm_event:        Result of stop operation for p_mysql_622 on 001db01a: 0 (ok) | call=1106 key=p_mysql_622_stop_0 confirmed=true cib-update=20029
Feb 16 14:06:44 [3908] 001db01a        cib:     info: cib_perform_op:   Diff: --- 0.346.13 2
Feb 16 14:06:44 [3908] 001db01a        cib:     info: cib_perform_op:   Diff: +++ 0.346.14 (null)
Feb 16 14:06:44 [3908] 001db01a        cib:     info: cib_perform_op:   +  /cib:  @num_updates=14
Feb 16 14:06:44 [3908] 001db01a        cib:     info: cib_perform_op:   +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_622']/lrm_rsc_op[@id='p_mysql_622_last_0']:  @transition-magic=0:0;111:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @call-id=1106, @rc-code=0, @op-status=0, @exec-time=5082
Feb 16 14:06:44 [3908] 001db01a        cib:     info: cib_process_request:      Completed cib_modify operation for section status: OK (rc=0, origin=001db01a/crmd/20029, version=0.346.14)
Feb 16 14:06:44 [3913] 001db01a       crmd:     info: match_graph_event:        Action p_mysql_622_stop_0 (111) confirmed on 001db01a (rc=0)
Feb 16 14:06:44 [3910] 001db01a       lrmd:     info: log_finished:     finished - rsc:p_mysql_000 action:stop call_id:1098 pid:36695 exit-code:0 exec-time:5093ms queue-time:0ms
Feb 16 14:06:44 [3908] 001db01a        cib:     info: cib_process_request:      Forwarding cib_modify operation for section status to all (origin=local/crmd/20030)
Feb 16 14:06:44 [3913] 001db01a       crmd:   notice: process_lrm_event:        Result of stop operation for p_mysql_000 on 001db01a: 0 (ok) | call=1098 key=p_mysql_000_stop_0 confirmed=true cib-update=20030
Feb 16 14:06:44 [3908] 001db01a        cib:     info: cib_perform_op:   Diff: --- 0.346.14 2
Feb 16 14:06:44 [3908] 001db01a        cib:     info: cib_perform_op:   Diff: +++ 0.346.15 (null)
Feb 16 14:06:44 [3908] 001db01a        cib:     info: cib_perform_op:   +  /cib:  @num_updates=15
Feb 16 14:06:44 [3908] 001db01a        cib:     info: cib_perform_op:   +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_000']/lrm_rsc_op[@id='p_mysql_000_last_0']:  @transition-magic=0:0;92:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @call-id=1098, @rc-code=0, @op-status=0, @exec-time=5093
Feb 16 14:06:44 [3908] 001db01a        cib:     info: cib_process_request:      Completed cib_modify operation for section status: OK (rc=0, origin=001db01a/crmd/20030, version=0.346.15)
Feb 16 14:06:44 [3913] 001db01a       crmd:     info: match_graph_event:        Action p_mysql_000_stop_0 (92) confirmed on 001db01a (rc=0)
Feb 16 14:06:49 [3908] 001db01a        cib:     info: cib_process_ping: Reporting our current digest to 001db01a: 1bc996395d5a4f64ed405bf122c8e890 for 0.346.15 (0x55ac23ea4ef0 0)
Feb 16 14:06:51 [3910] 001db01a       lrmd:     info: log_finished:     finished - rsc:p_mysql_003 action:stop call_id:1100 pid:36696 exit-code:0 exec-time:12102ms queue-time:0ms
Feb 16 14:06:51 [3910] 001db01a       lrmd:     info: log_finished:     finished - rsc:p_mysql_004 action:stop call_id:1102 pid:36701 exit-code:0 exec-time:12101ms queue-time:0ms
Feb 16 14:06:51 [3910] 001db01a       lrmd:     info: log_finished:     finished - rsc:p_mysql_005 action:stop call_id:1104 pid:36705 exit-code:0 exec-time:12100ms queue-time:0ms
Feb 16 14:06:51 [3910] 001db01a       lrmd:     info: log_finished:     finished - rsc:p_mysql_001 action:stop call_id:1096 pid:36694 exit-code:0 exec-time:12107ms queue-time:0ms
Feb 16 14:06:51 [3908] 001db01a        cib:     info: cib_process_request:      Forwarding cib_modify operation for section status to all (origin=local/crmd/20031)
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: process_lrm_event:        Result of stop operation for p_mysql_003 on 001db01a: 0 (ok) | call=1100 key=p_mysql_003_stop_0 confirmed=true cib-update=20031
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: process_lrm_event:        Result of stop operation for p_mysql_004 on 001db01a: 0 (ok) | call=1102 key=p_mysql_004_stop_0 confirmed=true cib-update=20032
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: process_lrm_event:        Result of stop operation for p_mysql_005 on 001db01a: 0 (ok) | call=1104 key=p_mysql_005_stop_0 confirmed=true cib-update=20033
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: process_lrm_event:        Result of stop operation for p_mysql_001 on 001db01a: 0 (ok) | call=1096 key=p_mysql_001_stop_0 confirmed=true cib-update=20034
Feb 16 14:06:51 [3908] 001db01a        cib:     info: cib_perform_op:   Diff: --- 0.346.15 2
Feb 16 14:06:51 [3908] 001db01a        cib:     info: cib_perform_op:   Diff: +++ 0.346.16 (null)
Feb 16 14:06:51 [3908] 001db01a        cib:     info: cib_perform_op:   +  /cib:  @num_updates=16
Feb 16 14:06:51 [3908] 001db01a        cib:     info: cib_perform_op:   +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_003']/lrm_rsc_op[@id='p_mysql_003_last_0']:  @transition-magic=0:0;96:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @call-id=1100, @rc-code=0, @op-status=0, @exec-time=12102
Feb 16 14:06:51 [3908] 001db01a        cib:     info: cib_process_request:      Completed cib_modify operation for section status: OK (rc=0, origin=001db01a/crmd/20031, version=0.346.16)
Feb 16 14:06:51 [3913] 001db01a       crmd:     info: match_graph_event:        Action p_mysql_003_stop_0 (96) confirmed on 001db01a (rc=0)
Feb 16 14:06:51 [3908] 001db01a        cib:     info: cib_process_request:      Forwarding cib_modify operation for section status to all (origin=local/crmd/20032)
Feb 16 14:06:51 [3908] 001db01a        cib:     info: cib_process_request:      Forwarding cib_modify operation for section status to all (origin=local/crmd/20033)
Feb 16 14:06:51 [3908] 001db01a        cib:     info: cib_process_request:      Forwarding cib_modify operation for section status to all (origin=local/crmd/20034)
Feb 16 14:06:51 [3908] 001db01a        cib:     info: cib_perform_op:   Diff: --- 0.346.16 2
Feb 16 14:06:51 [3908] 001db01a        cib:     info: cib_perform_op:   Diff: +++ 0.346.17 (null)
Feb 16 14:06:51 [3908] 001db01a        cib:     info: cib_perform_op:   +  /cib:  @num_updates=17
Feb 16 14:06:51 [3908] 001db01a        cib:     info: cib_perform_op:   +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_004']/lrm_rsc_op[@id='p_mysql_004_last_0']:  @transition-magic=0:0;99:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @call-id=1102, @rc-code=0, @op-status=0, @exec-time=12101
Feb 16 14:06:51 [3908] 001db01a        cib:     info: cib_process_request:      Completed cib_modify operation for section status: OK (rc=0, origin=001db01a/crmd/20032, version=0.346.17)
Feb 16 14:06:51 [3913] 001db01a       crmd:     info: match_graph_event:        Action p_mysql_004_stop_0 (99) confirmed on 001db01a (rc=0)
Feb 16 14:06:51 [3908] 001db01a        cib:     info: cib_perform_op:   Diff: --- 0.346.17 2
Feb 16 14:06:51 [3908] 001db01a        cib:     info: cib_perform_op:   Diff: +++ 0.346.18 (null)
Feb 16 14:06:51 [3908] 001db01a        cib:     info: cib_perform_op:   +  /cib:  @num_updates=18
Feb 16 14:06:51 [3908] 001db01a        cib:     info: cib_perform_op:   +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_005']/lrm_rsc_op[@id='p_mysql_005_last_0']:  @transition-magic=0:0;102:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @call-id=1104, @rc-code=0, @op-status=0, @exec-time=12100
Feb 16 14:06:51 [3908] 001db01a        cib:     info: cib_process_request:      Completed cib_modify operation for section status: OK (rc=0, origin=001db01a/crmd/20033, version=0.346.18)
Feb 16 14:06:51 [3913] 001db01a       crmd:     info: match_graph_event:        Action p_mysql_005_stop_0 (102) confirmed on 001db01a (rc=0)
Feb 16 14:06:51 [3908] 001db01a        cib:     info: cib_perform_op:   Diff: --- 0.346.18 2
Feb 16 14:06:51 [3908] 001db01a        cib:     info: cib_perform_op:   Diff: +++ 0.346.19 (null)
Feb 16 14:06:51 [3908] 001db01a        cib:     info: cib_perform_op:   +  /cib:  @num_updates=19
Feb 16 14:06:51 [3908] 001db01a        cib:     info: cib_perform_op:   +  /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='p_mysql_001']/lrm_rsc_op[@id='p_mysql_001_last_0']:  @transition-magic=0:0;89:17791:0:06cb68d1-6d39-4d7a-b772-ac943ab8cdb5, @call-id=1096, @rc-code=0, @op-status=0, @exec-time=12107
Feb 16 14:06:51 [3908] 001db01a        cib:     info: cib_process_request:      Completed cib_modify operation for section status: OK (rc=0, origin=001db01a/crmd/20034, version=0.346.19)
Feb 16 14:06:51 [3913] 001db01a       crmd:     info: match_graph_event:        Action p_mysql_001_stop_0 (89) confirmed on 001db01a (rc=0)
Feb 16 14:06:51 [3913] 001db01a       crmd:  warning: run_graph:        Transition 17791 (Complete=11, Pending=0, Fired=0, Skipped=0, Incomplete=28, Source=/var/lib/pacemaker/pengine/pe-input-482.bz2): Terminated
Feb 16 14:06:51 [3913] 001db01a       crmd:  warning: te_graph_trigger: Transition failed: terminated
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_graph:      Graph 17791 with 39 actions: batch-limit=0 jobs, network-delay=60000ms
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:    [Action  126]: Pending rsc op p_drbd0_post_notify_demote_0        on 001db01b (priority: 1000000, waiting:  50)
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:    [Action  125]: Completed rsc op p_drbd0_pre_notify_demote_0       on 001db01b (priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:    [Action  122]: Pending rsc op p_drbd0_post_notify_promote_0       on 001db01b (priority: 1000000, waiting:  44)
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:    [Action  121]: Pending rsc op p_drbd0_pre_notify_promote_0        on 001db01b (priority: 0, waiting:  42)
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:    [Action   23]: Pending rsc op p_drbd0_promote_0                   on 001db01b (priority: 0, waiting:  40)
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:    [Action    6]: Completed rsc op p_drbd0_monitor_60000             on 001db01b (priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:    [Action  128]: Pending rsc op p_drbd0_post_notify_demote_0        on 001db01a (priority: 1000000, waiting:  50)
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:    [Action  127]: Completed rsc op p_drbd0_pre_notify_demote_0       on 001db01a (priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:    [Action  124]: Pending rsc op p_drbd0_post_notify_promote_0       on 001db01a (priority: 1000000, waiting:  44)
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:    [Action  123]: Pending rsc op p_drbd0_pre_notify_promote_0        on 001db01a (priority: 0, waiting:  42)
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:    [Action   27]: Pending rsc op p_drbd0_monitor_60000               on 001db01a (priority: 0, waiting:  25 45 51)
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:    [Action   25]: Pending rsc op p_drbd0_demote_0                    on 001db01a (priority: 0, waiting:  46)
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:    [Action   51]: Pending pseudo op ms_drbd0_confirmed-post_notify_demoted_0 on N/A (priority: 1000000, waiting:  50 126 128)
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:    [Action   50]: Pending pseudo op ms_drbd0_post_notify_demoted_0   on N/A (priority: 1000000, waiting:  47)
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:    [Action   49]: Completed pseudo op ms_drbd0_confirmed-pre_notify_demote_0 on N/A (priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:    [Action   48]: Completed pseudo op ms_drbd0_pre_notify_demote_0   on N/A (priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:    [Action   47]: Pending pseudo op ms_drbd0_demoted_0               on N/A (priority: 1000000, waiting:  25 46)
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:    [Action   46]: Pending pseudo op ms_drbd0_demote_0                on N/A (priority: 0, waiting:  82)
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:    [Action   45]: Pending pseudo op ms_drbd0_confirmed-post_notify_promoted_0 on N/A (priority: 1000000, waiting:  44 122 124)
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:    [Action   44]: Pending pseudo op ms_drbd0_post_notify_promoted_0  on N/A (priority: 1000000, waiting:  41 43)
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:    [Action   43]: Pending pseudo op ms_drbd0_confirmed-pre_notify_promote_0 on N/A (priority: 0, waiting:  42 121 123)
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:    [Action   42]: Pending pseudo op ms_drbd0_pre_notify_promote_0    on N/A (priority: 0, waiting:  51)
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:    [Action   41]: Pending pseudo op ms_drbd0_promoted_0              on N/A (priority: 1000000, waiting:  23)
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:    [Action   40]: Pending pseudo op ms_drbd0_promote_0               on N/A (priority: 0, waiting:  43 47)
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:    [Action   84]: Pending rsc op p_fs_clust01_monitor_20000          on 001db01b (priority: 0, waiting:  83)
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:    [Action   83]: Pending rsc op p_fs_clust01_start_0                on 001db01b (priority: 0, waiting:  45 82)
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:    [Action   82]: Pending rsc op p_fs_clust01_stop_0                 on 001db01a (priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:     * [Input 18]: Unresolved dependency rsc op p_vip_clust01_stop_0 on 001db01a
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:    [Action   91]: Pending rsc op p_mysql_001_monitor_15000           on 001db01b (priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:     * [Input 90]: Unresolved dependency rsc op p_mysql_001_start_0 on 001db01b
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:    [Action   89]: Completed rsc op p_mysql_001_stop_0                on 001db01a (priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:    [Action   94]: Pending rsc op p_mysql_000_monitor_15000           on 001db01b (priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:     * [Input 93]: Unresolved dependency rsc op p_mysql_000_start_0 on 001db01b
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:    [Action   92]: Completed rsc op p_mysql_000_stop_0                on 001db01a (priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:    [Action   98]: Pending rsc op p_mysql_003_monitor_15000           on 001db01b (priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:     * [Input 97]: Unresolved dependency rsc op p_mysql_003_start_0 on 001db01b
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:    [Action   96]: Completed rsc op p_mysql_003_stop_0                on 001db01a (priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:    [Action  101]: Pending rsc op p_mysql_004_monitor_15000           on 001db01b (priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:     * [Input 100]: Unresolved dependency rsc op p_mysql_004_start_0 on 001db01b
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:    [Action   99]: Completed rsc op p_mysql_004_stop_0                on 001db01a (priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:    [Action  104]: Pending rsc op p_mysql_005_monitor_15000           on 001db01b (priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:     * [Input 103]: Unresolved dependency rsc op p_mysql_005_start_0 on 001db01b
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:    [Action  102]: Completed rsc op p_mysql_005_stop_0                on 001db01a (priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:    [Action  113]: Pending rsc op p_mysql_622_monitor_15000           on 001db01b (priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:     * [Input 112]: Unresolved dependency rsc op p_mysql_622_start_0 on 001db01b
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: print_synapse:    [Action  111]: Completed rsc op p_mysql_622_stop_0                on 001db01a (priority: 0, waiting: none)
Feb 16 14:06:51 [3913] 001db01a       crmd:     info: do_log:   Input I_TE_SUCCESS received in state S_TRANSITION_ENGINE from notify_crmd
Feb 16 14:06:51 [3913] 001db01a       crmd:   notice: do_state_transition:      State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd
Feb 16 14:06:56 [3908] 001db01a        cib:     info: cib_process_ping: Reporting our current digest to 001db01a: cf6fae5210d45a88d51e78fe8acb1178 for 0.346.19 (0x55ac23ea4ef0 0)



> -----Original Message-----
> From: Users <users-bounces at clusterlabs.org> On Behalf Of Eric Robinson
> Sent: Saturday, February 16, 2019 12:51 PM
> To: Cluster Labs - All topics related to open-source clustering welcomed
> <users at clusterlabs.org>
> Subject: Re: [ClusterLabs] Why Do All The Services Go Down When Just One
> Fails?
> 
> Hi Valentin --
> 
> Which logs? You mean /var/log/cluster/corosync.log?
> 
> But even if the stop action is resulting in an error, why would the cluster also
> try to stop the other services which are not dependent?
> 
> > -----Original Message-----
> > From: Users <users-bounces at clusterlabs.org> On Behalf Of Valentin
> > Vidic
> > Sent: Saturday, February 16, 2019 12:44 PM
> > To: users at clusterlabs.org
> > Subject: Re: [ClusterLabs] Why Do All The Services Go Down When Just
> > One Fails?
> >
> > On Sat, Feb 16, 2019 at 08:34:21PM +0000, Eric Robinson wrote:
> > > Why is it that when one of the resources that start with p_mysql_*
> > > goes into a FAILED state, all the other MySQL services also stop?
> >
> > Perhaps stop is not working correctly for these lsb services, so for
> > example stopping lsb:mysql_004 also stops the other lsb:mysql_nnn.
> >
> > You would need to send the logs from the event to confirm this.
> >
> > --
> > Valentin
> > _______________________________________________
> > Users mailing list: Users at clusterlabs.org
> > https://lists.clusterlabs.org/mailman/listinfo/users
> >
> > Project Home: http://www.clusterlabs.org Getting started:
> > http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > Bugs: http://bugs.clusterlabs.org
> _______________________________________________
> Users mailing list: Users at clusterlabs.org
> https://lists.clusterlabs.org/mailman/listinfo/users
> 
> Project Home: http://www.clusterlabs.org Getting started:
> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org



More information about the Users mailing list