[ClusterLabs] DC marks itself as OFFLINE, continues orchestrating the other nodes
Lars Ellenberg
lars.ellenberg at linbit.com
Wed Sep 14 06:40:04 EDT 2022
On Thu, Sep 08, 2022 at 10:11:46AM -0500, Ken Gaillot wrote:
> On Thu, 2022-09-08 at 15:01 +0200, Lars Ellenberg wrote:
> > Scenario:
> > three nodes, no fencing (I know)
> > break network, isolating nodes
> > unbreak network, see how cluster partitions rejoin and resume service
>
> I'm guessing the CIB changed during the break, with more changes in one
> of the other partitions than mqhavm24 ...
quite likely.
> Reconciling CIB differences in different partitions is inherently
> lossy. Basically we gotta pick one side to win, and the current
> algorithm just looks at the number of changes. (An "admin epoch" can
> also be bumped manually to override that.)
Yes.
> > I have full crm_reports and some context knowledge about the setup.
> >
> > For now I'd like to know: has anyone seen this before,
> > is that a known bug in corner cases/races during re-join,
> > has it even been fixed meanwhile?
>
> No, yes, no
Thank you.
That's what I thought :-|
> It does seem we could handle the specific case of the local node's
> state being overwritten a little better. We can't just override the
> join state if the other nodes think it is different, but we could
> release DC and restart the join process. How did it handle the
> situation in this case?
I think these are the most interesting lines:
-----------------
Aug 11 12:32:45 mqhavm24 corosync[13296]: [QUORUM] Members[1]: 1
stopping stuff
Aug 11 12:33:36 mqhavm24 corosync[13296]: [QUORUM] Members[3]: 1 3 2
Aug 11 12:33:36 [13310] mqhavm24 crmd: warning: crmd_ha_msg_filter: Another DC detected: mqhavm37 (op=noop)
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: update_dc: Set DC to mqhavm24 (3.0.14)
Aug 11 12:33:36 [13308] mqhavm24 attrd: notice: attrd_check_for_new_writer: Detected another attribute writer (mqhavm37), starting new election
Aug 11 12:33:36 [13308] mqhavm24 attrd: notice: attrd_declare_winner: Recorded local node as attribute writer (was unset)
plan to start stuff on all three nodes
Aug 11 12:33:36 [13309] mqhavm24 pengine: notice: process_pe_message: Calculated transition 161, saving inputs in /var/lib/pacemaker/pengine/pe-input-688.bz2
but then
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_cib_replaced, @join=down
and we now keep stuff stopped locally, but continue to manage the other two nodes.
-----------------
commented log of the most intersting node below,
starting at the point when communication goes down.
maybe you see something that gives you an idea how to handle this better.
If it helps, I have the full crm_report of all nodes,
should you feel the urge to have a look.
Aug 11 12:32:45 mqhavm24 corosync[13296]: [TOTEM ] Failed to receive the leave message. failed: 3 2
Aug 11 12:32:45 mqhavm24 corosync[13296]: [QUORUM] This node is within the non-primary component and will NOT provide any services.
Aug 11 12:32:45 mqhavm24 corosync[13296]: [QUORUM] Members[1]: 1
Aug 11 12:32:45 mqhavm24 corosync[13296]: [MAIN ] Completed service synchronization, ready to provide service.
[stripping most info level for now]
Aug 11 12:32:45 [13306] mqhavm24 stonith-ng: notice: crm_update_peer_state_iter: Node mqhavm37 state is now lost | nodeid=2 previous=member source=crm_update_peer_proc
Aug 11 12:32:45 [13306] mqhavm24 stonith-ng: notice: reap_crm_member: Purged 1 peer with id=2 and/or uname=mqhavm37 from the membership cache
Aug 11 12:32:45 [13306] mqhavm24 stonith-ng: notice: crm_update_peer_state_iter: Node mqhavm34 state is now lost | nodeid=3 previous=member source=crm_update_peer_proc
Aug 11 12:32:45 [13306] mqhavm24 stonith-ng: notice: reap_crm_member: Purged 1 peer with id=3 and/or uname=mqhavm34 from the membership cache
Aug 11 12:32:45 [13303] mqhavm24 pacemakerd: warning: pcmk_quorum_notification: Quorum lost | membership=3112546 members=1
Aug 11 12:32:45 [13303] mqhavm24 pacemakerd: notice: crm_update_peer_state_iter: Node mqhavm34 state is now lost | nodeid=3 previous=member source=crm_reap_unseen_nodes
Aug 11 12:32:45 [13303] mqhavm24 pacemakerd: notice: crm_update_peer_state_iter: Node mqhavm37 state is now lost | nodeid=2 previous=member source=crm_reap_unseen_nodes
Aug 11 12:32:45 [13310] mqhavm24 crmd: warning: pcmk_quorum_notification: Quorum lost | membership=3112546 members=1
Aug 11 12:32:45 [13310] mqhavm24 crmd: notice: crm_update_peer_state_iter: Node mqhavm34 state is now lost | nodeid=3 previous=member source=crm_reap_unseen_nodes
Aug 11 12:32:45 [13308] mqhavm24 attrd: notice: crm_update_peer_state_iter: Node mqhavm37 state is now lost | nodeid=2 previous=member source=crm_update_peer_proc
Aug 11 12:32:45 [13305] mqhavm24 cib: notice: crm_update_peer_state_iter: Node mqhavm37 state is now lost | nodeid=2 previous=member source=crm_update_peer_proc
Aug 11 12:32:45 [13308] mqhavm24 attrd: notice: attrd_peer_remove: Removing all mqhavm37 attributes for peer loss
Aug 11 12:32:45 [13305] mqhavm24 cib: notice: reap_crm_member: Purged 1 peer with id=2 and/or uname=mqhavm37 from the membership cache
Aug 11 12:32:45 [13308] mqhavm24 attrd: notice: reap_crm_member: Purged 1 peer with id=2 and/or uname=mqhavm37 from the membership cache
Aug 11 12:32:45 [13305] mqhavm24 cib: notice: crm_update_peer_state_iter: Node mqhavm34 state is now lost | nodeid=3 previous=member source=crm_update_peer_proc
Aug 11 12:32:45 [13308] mqhavm24 attrd: notice: crm_update_peer_state_iter: Node mqhavm34 state is now lost | nodeid=3 previous=member source=crm_update_peer_proc
Aug 11 12:32:45 [13305] mqhavm24 cib: notice: reap_crm_member: Purged 1 peer with id=3 and/or uname=mqhavm34 from the membership cache
Aug 11 12:32:45 [13308] mqhavm24 attrd: notice: attrd_peer_remove: Removing all mqhavm34 attributes for peer loss
Aug 11 12:32:45 [13308] mqhavm24 attrd: notice: reap_crm_member: Purged 1 peer with id=3 and/or uname=mqhavm34 from the membership cache
Aug 11 12:32:45 [13310] mqhavm24 crmd: warning: match_down_event: No reason to expect node 3 to be down
Aug 11 12:32:45 [13310] mqhavm24 crmd: notice: peer_update_callback: Stonith/shutdown of mqhavm34 not matched
Aug 11 12:32:45 [13310] mqhavm24 crmd: notice: crm_update_peer_state_iter: Node mqhavm37 state is now lost | nodeid=2 previous=member source=crm_reap_unseen_nodes
Aug 11 12:32:45 [13310] mqhavm24 crmd: warning: match_down_event: No reason to expect node 2 to be down
Aug 11 12:32:45 [13310] mqhavm24 crmd: notice: peer_update_callback: Stonith/shutdown of mqhavm37 not matched
Aug 11 12:32:45 [13310] mqhavm24 crmd: notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph
Aug 11 12:32:45 [13310] mqhavm24 crmd: warning: match_down_event: No reason to expect node 2 to be down
Aug 11 12:32:45 [13310] mqhavm24 crmd: notice: peer_update_callback: Stonith/shutdown of mqhavm37 not matched
Aug 11 12:32:45 [13310] mqhavm24 crmd: warning: match_down_event: No reason to expect node 3 to be down
Aug 11 12:32:45 [13310] mqhavm24 crmd: notice: peer_update_callback: Stonith/shutdown of mqhavm34 not matched
Aug 11 12:32:45 [13309] mqhavm24 pengine: warning: cluster_status: Fencing and resource management disabled due to lack of quorum
Aug 11 12:32:45 [13309] mqhavm24 pengine: notice: LogAction: * Stop drgxrde_rdqma ( mqhavm24 ) due to no quorum
Aug 11 12:32:45 [13309] mqhavm24 pengine: notice: LogAction: * Stop p_fs_drgxrde_rdqma ( mqhavm24 ) due to no quorum
Aug 11 12:32:45 [13309] mqhavm24 pengine: notice: LogAction: * Stop p_rdqmx_drgxrde_rdqma ( mqhavm24 ) due to no quorum
Aug 11 12:32:45 [13309] mqhavm24 pengine: notice: LogAction: * Stop p_drbd_dr_drgxrde_rdqma:0 ( Master mqhavm24 ) due to no quorum
Aug 11 12:32:45 [13309] mqhavm24 pengine: notice: LogAction: * Stop p_drbd_drgxrde_rdqma:0 ( Master mqhavm24 ) due to no quorum
Aug 11 12:32:45 [13309] mqhavm24 pengine: notice: LogAction: * Stop drgxrde_rdqmb ( mqhavm24 ) due to no quorum
Aug 11 12:32:45 [13309] mqhavm24 pengine: notice: LogAction: * Stop p_fs_drgxrde_rdqmb ( mqhavm24 ) due to no quorum
Aug 11 12:32:45 [13309] mqhavm24 pengine: notice: LogAction: * Stop p_rdqmx_drgxrde_rdqmb ( mqhavm24 ) due to no quorum
Aug 11 12:32:45 [13309] mqhavm24 pengine: notice: LogAction: * Stop p_drbd_dr_drgxrde_rdqmb:0 ( Master mqhavm24 ) due to no quorum
Aug 11 12:32:45 [13309] mqhavm24 pengine: notice: LogAction: * Stop p_drbd_drgxrde_rdqmb:0 ( Master mqhavm24 ) due to no quorum
Aug 11 12:32:45 [13309] mqhavm24 pengine: notice: LogAction: * Stop p_ip_drgxrde_rdqma ( mqhavm24 ) due to no quorum
Aug 11 12:32:45 [13309] mqhavm24 pengine: notice: LogAction: * Stop p_ip_drgxrde_rdqmb ( mqhavm24 ) due to no quorum
Aug 11 12:32:45 [13309] mqhavm24 pengine: notice: process_pe_message: Calculated transition 154, saving inputs in /var/lib/pacemaker/pengine/pe-input-682.bz2
Aug 11 12:32:45 [13310] mqhavm24 crmd: notice: te_rsc_command: Initiating stop operation drgxrde_rdqma_stop_0 locally ...
boring stopping stuff stripped ...
Aug 11 12:32:48 [13310] mqhavm24 crmd: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd
Aug 11 12:33:02 [13310] mqhavm24 crmd: notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph
Aug 11 12:33:02 [13309] mqhavm24 pengine: warning: cluster_status: Fencing and resource management disabled due to lack of quorum
Aug 11 12:33:02 [13309] mqhavm24 pengine: notice: LogAction: * Start p_drbd_drgxrde_rdqma:0 ( mqhavm24 ) due to no quorum (blocked)
Aug 11 12:33:02 [13309] mqhavm24 pengine: notice: LogAction: * Start p_drbd_drgxrde_rdqmb:0 ( mqhavm24 ) due to no quorum (blocked)
Aug 11 12:33:02 [13309] mqhavm24 pengine: notice: process_pe_message: Calculated transition 158, saving inputs in /var/lib/pacemaker/pengine/pe-input-686.bz2
Aug 11 12:33:02 [13310] mqhavm24 crmd: notice: run_graph: Transition 158 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-686.bz2): Complete
Aug 11 12:33:02 [13310] mqhavm24 crmd: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd
Aug 11 12:33:02 [13305] mqhavm24 cib: info: cib_process_request: Forwarding cib_modify operation for section resources to all (origin=local/crm_resource/6)
Aug 11 12:33:02 [13305] mqhavm24 cib: info: cib_perform_op: Diff: --- 0.171.0 2
Aug 11 12:33:02 [13305] mqhavm24 cib: info: cib_perform_op: Diff: +++ 0.172.0 (null)
Aug 11 12:33:02 [13305] mqhavm24 cib: info: cib_perform_op: + /cib: @epoch=172
Aug 11 12:33:02 [13305] mqhavm24 cib: info: cib_perform_op: + /cib/configuration/resources/master[@id='ms_drbd_dr_drgxrde_rdqmb']/meta_attributes[@id='ms_drbd_dr_drgxrde_rdqmb-meta_attributes']/nvpair[@id='ms_drbd_dr_drgxrde_rdqmb-meta_attributes-target-role']: @value=Slave
Aug 11 12:33:02 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_modify operation for section resources: OK (rc=0, origin=mqhavm24/crm_resource/6, version=0.172.0)
Aug 11 12:33:02 [13310] mqhavm24 crmd: info: abort_transition_graph: Transition aborted by ms_drbd_dr_drgxrde_rdqmb-meta_attributes-target-role doing modify target-role=Slave: Configuration change | cib=0.172.0 source=te_update_diff_v2:522 path=/cib/configuration/resources/master[@id='ms_drbd_dr_drgxrde_rdqmb']/meta_attributes[@id='ms_drbd_dr_drgxrde_rdqmb-meta_attributes']/nvpair[@id='ms_drbd_dr_drgxrde_rdqmb-meta_attributes-target-role'] complete=true
Aug 11 12:33:02 [13310] mqhavm24 crmd: notice: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph
Aug 11 12:33:02 [13305] mqhavm24 cib: info: cib_file_backup: Archived previous version as /var/lib/pacemaker/cib/cib-78.raw
Aug 11 12:33:02 [13309] mqhavm24 pengine: warning: cluster_status: Fencing and resource management disabled due to lack of quorum
Aug 11 12:33:02 [13309] mqhavm24 pengine: notice: LogAction: * Start p_drbd_drgxrde_rdqma:0 ( mqhavm24 ) due to no quorum (blocked)
Aug 11 12:33:02 [13309] mqhavm24 pengine: notice: LogAction: * Start p_drbd_drgxrde_rdqmb:0 ( mqhavm24 ) due to no quorum (blocked)
Aug 11 12:33:02 [13309] mqhavm24 pengine: notice: process_pe_message: Calculated transition 159, saving inputs in /var/lib/pacemaker/pengine/pe-input-687.bz2
Aug 11 12:33:02 [13310] mqhavm24 crmd: notice: run_graph: Transition 159 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-687.bz2): Complete
Aug 11 12:33:02 [13310] mqhavm24 crmd: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd
nothing, until connectivity is back.
I leave the info level in below:
Aug 11 12:33:36 mqhavm24 corosync[13296]: [TOTEM ] A new membership (192.168.101.24:3112554) was formed. Members joined: 3 2
Aug 11 12:33:36 mqhavm24 corosync[13296]: [QUORUM] This node is within the primary component and will provide service.
Aug 11 12:33:36 mqhavm24 corosync[13296]: [QUORUM] Members[3]: 1 3 2
Aug 11 12:33:36 mqhavm24 corosync[13296]: [MAIN ] Completed service synchronization, ready to provide service.
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: pcmk_cpg_membership: Group attrd event 11: node 3 pid 2553 joined via cluster join
Aug 11 12:33:36 [13303] mqhavm24 pacemakerd: info: pcmk_cpg_membership: Group pacemakerd event 11: node 3 pid 2546 joined via cluster join
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: pcmk_cpg_membership: Group attrd event 11: mqhavm24 (node 1 pid 13308) is member
Aug 11 12:33:36 [13303] mqhavm24 pacemakerd: info: pcmk_cpg_membership: Group pacemakerd event 11: mqhavm24 (node 1 pid 13303) is member
Aug 11 12:33:36 [13303] mqhavm24 pacemakerd: info: pcmk_cpg_membership: Group pacemakerd event 11: mqhavm34 (node 3 pid 2546) is member
Aug 11 12:33:36 [13303] mqhavm24 pacemakerd: info: crm_update_peer_proc: pcmk_cpg_membership: Node mqhavm34[3] - corosync-cpg is now online
Aug 11 12:33:36 [13303] mqhavm24 pacemakerd: warning: pcmk_cpg_membership: Node 3 is member of group pacemakerd but was believed offline
Aug 11 12:33:36 [13303] mqhavm24 pacemakerd: notice: crm_update_peer_state_iter: Node mqhavm34 state is now member | nodeid=3 previous=lost source=pcmk_cpg_membership
Aug 11 12:33:36 [13303] mqhavm24 pacemakerd: info: crm_cs_flush: Sent 0 CPG messages (1 remaining, last=17): Try again (6)
Aug 11 12:33:36 [13306] mqhavm24 stonith-ng: info: pcmk_cpg_membership: Group stonith-ng event 11: node 3 pid 2551 joined via cluster join
Aug 11 12:33:36 [13306] mqhavm24 stonith-ng: info: pcmk_cpg_membership: Group stonith-ng event 11: mqhavm24 (node 1 pid 13306) is member
Aug 11 12:33:36 [13303] mqhavm24 pacemakerd: info: pcmk_cpg_membership: Group pacemakerd event 12: node 2 pid 41735 joined via cluster join
Aug 11 12:33:36 [13303] mqhavm24 pacemakerd: info: pcmk_cpg_membership: Group pacemakerd event 12: mqhavm24 (node 1 pid 13303) is member
Aug 11 12:33:36 [13303] mqhavm24 pacemakerd: info: pcmk_cpg_membership: Group pacemakerd event 12: mqhavm37 (node 2 pid 41735) is member
Aug 11 12:33:36 [13303] mqhavm24 pacemakerd: info: crm_update_peer_proc: pcmk_cpg_membership: Node mqhavm37[2] - corosync-cpg is now online
Aug 11 12:33:36 [13303] mqhavm24 pacemakerd: warning: pcmk_cpg_membership: Node 2 is member of group pacemakerd but was believed offline
Aug 11 12:33:36 [13303] mqhavm24 pacemakerd: notice: crm_update_peer_state_iter: Node mqhavm37 state is now member | nodeid=2 previous=lost source=pcmk_cpg_membership
Aug 11 12:33:36 [13303] mqhavm24 pacemakerd: info: pcmk_cpg_membership: Group pacemakerd event 12: mqhavm34 (node 3 pid 2546) is member
Aug 11 12:33:36 [13305] mqhavm24 cib: info: pcmk_cpg_membership: Group cib event 11: node 3 pid 2550 joined via cluster join
Aug 11 12:33:36 [13305] mqhavm24 cib: info: pcmk_cpg_membership: Group cib event 11: mqhavm24 (node 1 pid 13305) is member
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: pcmk_cpg_membership: Group crmd event 11: node 3 pid 2555 joined via cluster join
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: pcmk_cpg_membership: Group crmd event 11: mqhavm24 (node 1 pid 13310) is member
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: pcmk_cpg_membership: Group crmd event 11: mqhavm34 (node 3 pid 2555) is member
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: crm_update_peer_proc: pcmk_cpg_membership: Node mqhavm34[3] - corosync-cpg is now online
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: peer_update_callback: Client mqhavm34/peer now has status [online] (DC=true, changed=4000000)
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: te_trigger_stonith_history_sync: Fence history will be synchronized cluster-wide within 5 seconds
Aug 11 12:33:36 [13310] mqhavm24 crmd: warning: pcmk_cpg_membership: Node 3 is member of group crmd but was believed offline
Aug 11 12:33:36 [13310] mqhavm24 crmd: notice: crm_update_peer_state_iter: Node mqhavm34 state is now member | nodeid=3 previous=lost source=pcmk_cpg_membership
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: peer_update_callback: Cluster node mqhavm34 is now member (was lost)
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: exec_alert_list: Sending node alert via rdqm-alert to (null)
Aug 11 12:33:36 [13307] mqhavm24 lrmd: info: process_lrmd_alert_exec: Executing alert rdqm-alert for 304b95f0-bb72-4697-a1ec-45633c59f62d
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: pcmk_cpg_membership: Group crmd event 12: node 2 pid 41742 joined via cluster join
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: pcmk_cpg_membership: Group crmd event 12: mqhavm24 (node 1 pid 13310) is member
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: pcmk_cpg_membership: Group crmd event 12: mqhavm37 (node 2 pid 41742) is member
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: crm_update_peer_proc: pcmk_cpg_membership: Node mqhavm37[2] - corosync-cpg is now online
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: peer_update_callback: Client mqhavm37/peer now has status [online] (DC=true, changed=4000000)
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: te_trigger_stonith_history_sync: Fence history will be synchronized cluster-wide within 5 seconds
Aug 11 12:33:36 [13310] mqhavm24 crmd: warning: pcmk_cpg_membership: Node 2 is member of group crmd but was believed offline
Aug 11 12:33:36 [13310] mqhavm24 crmd: notice: crm_update_peer_state_iter: Node mqhavm37 state is now member | nodeid=2 previous=lost source=pcmk_cpg_membership
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: peer_update_callback: Cluster node mqhavm37 is now member (was lost)
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: exec_alert_list: Sending node alert via rdqm-alert to (null)
Aug 11 12:33:36 [13307] mqhavm24 lrmd: info: process_lrmd_alert_exec: Executing alert rdqm-alert for 304b95f0-bb72-4697-a1ec-45633c59f62d
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: pcmk_cpg_membership: Group crmd event 12: mqhavm34 (node 3 pid 2555) is member
Aug 11 12:33:36 [13303] mqhavm24 pacemakerd: notice: pcmk_quorum_notification: Quorum acquired | membership=3112554 members=3
Aug 11 12:33:36 [13310] mqhavm24 crmd: notice: pcmk_quorum_notification: Quorum acquired | membership=3112554 members=3
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: corosync_node_name: Unable to get node name for nodeid 3
Aug 11 12:33:36 [13308] mqhavm24 attrd: notice: get_node_name: Could not obtain a node name for corosync nodeid 3
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: crm_get_peer: Created entry 9ee8517a-3318-45e3-9d3a-dc93a8094e87/0x55901ed8de70 for node (null)/3 (2 total)
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: crm_get_peer: Node 3 has uuid 3
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: pcmk_cpg_membership: Group attrd event 11: peer node (node 3 pid 2553) is member
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: crm_update_peer_proc: pcmk_cpg_membership: Node (null)[3] - corosync-cpg is now online
Aug 11 12:33:36 [13308] mqhavm24 attrd: notice: crm_update_peer_state_iter: Node (null) state is now member | nodeid=3 previous=unknown source=crm_update_peer_proc
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: pcmk_cpg_membership: Group attrd event 12: node 2 pid 41740 joined via cluster join
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: pcmk_cpg_membership: Group attrd event 12: mqhavm24 (node 1 pid 13308) is member
Aug 11 12:33:36 [13305] mqhavm24 cib: info: corosync_node_name: Unable to get node name for nodeid 3
Aug 11 12:33:36 [13305] mqhavm24 cib: notice: get_node_name: Could not obtain a node name for corosync nodeid 3
Aug 11 12:33:36 [13305] mqhavm24 cib: info: crm_get_peer: Created entry f56cdbb5-3019-4ab7-9cce-4afd171fe3b9/0x55d7e0b7f6e0 for node (null)/3 (2 total)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: crm_get_peer: Node 3 has uuid 3
Aug 11 12:33:36 [13305] mqhavm24 cib: info: pcmk_cpg_membership: Group cib event 11: peer node (node 3 pid 2550) is member
Aug 11 12:33:36 [13305] mqhavm24 cib: info: crm_update_peer_proc: pcmk_cpg_membership: Node (null)[3] - corosync-cpg is now online
Aug 11 12:33:36 [13305] mqhavm24 cib: notice: crm_update_peer_state_iter: Node (null) state is now member | nodeid=3 previous=unknown source=crm_update_peer_proc
Aug 11 12:33:36 [13305] mqhavm24 cib: info: pcmk_cpg_membership: Group cib event 12: node 2 pid 41737 joined via cluster join
Aug 11 12:33:36 [13305] mqhavm24 cib: info: pcmk_cpg_membership: Group cib event 12: mqhavm24 (node 1 pid 13305) is member
Aug 11 12:33:36 [13306] mqhavm24 stonith-ng: info: corosync_node_name: Unable to get node name for nodeid 3
Aug 11 12:33:36 [13306] mqhavm24 stonith-ng: notice: get_node_name: Could not obtain a node name for corosync nodeid 3
Aug 11 12:33:36 [13306] mqhavm24 stonith-ng: info: crm_get_peer: Created entry 12e40f2a-affc-4a30-bdc1-6db3d06f20e6/0x560ff1f04310 for node (null)/3 (2 total)
Aug 11 12:33:36 [13306] mqhavm24 stonith-ng: info: crm_get_peer: Node 3 has uuid 3
Aug 11 12:33:36 [13306] mqhavm24 stonith-ng: info: pcmk_cpg_membership: Group stonith-ng event 11: peer node (node 3 pid 2551) is member
Aug 11 12:33:36 [13306] mqhavm24 stonith-ng: info: crm_update_peer_proc: pcmk_cpg_membership: Node (null)[3] - corosync-cpg is now online
Aug 11 12:33:36 [13306] mqhavm24 stonith-ng: notice: crm_update_peer_state_iter: Node (null) state is now member | nodeid=3 previous=unknown source=crm_update_peer_proc
Aug 11 12:33:36 [13306] mqhavm24 stonith-ng: info: pcmk_cpg_membership: Group stonith-ng event 12: node 2 pid 41738 joined via cluster join
Aug 11 12:33:36 [13306] mqhavm24 stonith-ng: info: pcmk_cpg_membership: Group stonith-ng event 12: mqhavm24 (node 1 pid 13306) is member
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: corosync_node_name: Unable to get node name for nodeid 2
Aug 11 12:33:36 [13308] mqhavm24 attrd: notice: get_node_name: Could not obtain a node name for corosync nodeid 2
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: crm_get_peer: Created entry d59e708d-c858-4d3d-a516-842e8f979e37/0x55901ed8dee0 for node (null)/2 (3 total)
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: crm_get_peer: Node 2 has uuid 2
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: pcmk_cpg_membership: Group attrd event 12: peer node (node 2 pid 41740) is member
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: crm_update_peer_proc: pcmk_cpg_membership: Node (null)[2] - corosync-cpg is now online
Aug 11 12:33:36 [13308] mqhavm24 attrd: notice: crm_update_peer_state_iter: Node (null) state is now member | nodeid=2 previous=unknown source=crm_update_peer_proc
Aug 11 12:33:36 [13305] mqhavm24 cib: info: corosync_node_name: Unable to get node name for nodeid 2
Aug 11 12:33:36 [13305] mqhavm24 cib: notice: get_node_name: Could not obtain a node name for corosync nodeid 2
Aug 11 12:33:36 [13305] mqhavm24 cib: info: crm_get_peer: Created entry 30120675-3a25-4fe3-91ac-a6f408ef5e30/0x55d7e0b7f750 for node (null)/2 (3 total)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: crm_get_peer: Node 2 has uuid 2
Aug 11 12:33:36 [13305] mqhavm24 cib: info: pcmk_cpg_membership: Group cib event 12: peer node (node 2 pid 41737) is member
Aug 11 12:33:36 [13305] mqhavm24 cib: info: crm_update_peer_proc: pcmk_cpg_membership: Node (null)[2] - corosync-cpg is now online
Aug 11 12:33:36 [13305] mqhavm24 cib: notice: crm_update_peer_state_iter: Node (null) state is now member | nodeid=2 previous=unknown source=crm_update_peer_proc
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: corosync_node_name: Unable to get node name for nodeid 3
Aug 11 12:33:36 [13308] mqhavm24 attrd: notice: get_node_name: Could not obtain a node name for corosync nodeid 3
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: pcmk_cpg_membership: Group attrd event 12: peer node (node 3 pid 2553) is member
Aug 11 12:33:36 [13306] mqhavm24 stonith-ng: info: corosync_node_name: Unable to get node name for nodeid 2
Aug 11 12:33:36 [13306] mqhavm24 stonith-ng: notice: get_node_name: Could not obtain a node name for corosync nodeid 2
Aug 11 12:33:36 [13306] mqhavm24 stonith-ng: info: crm_get_peer: Created entry 7cee70c9-7707-4380-9f70-2c6c425cef33/0x560ff1f04380 for node (null)/2 (3 total)
Aug 11 12:33:36 [13306] mqhavm24 stonith-ng: info: crm_get_peer: Node 2 has uuid 2
Aug 11 12:33:36 [13306] mqhavm24 stonith-ng: info: pcmk_cpg_membership: Group stonith-ng event 12: peer node (node 2 pid 41738) is member
Aug 11 12:33:36 [13306] mqhavm24 stonith-ng: info: crm_update_peer_proc: pcmk_cpg_membership: Node (null)[2] - corosync-cpg is now online
Aug 11 12:33:36 [13306] mqhavm24 stonith-ng: notice: crm_update_peer_state_iter: Node (null) state is now member | nodeid=2 previous=unknown source=crm_update_peer_proc
Aug 11 12:33:36 [13305] mqhavm24 cib: info: corosync_node_name: Unable to get node name for nodeid 3
Aug 11 12:33:36 [13305] mqhavm24 cib: notice: get_node_name: Could not obtain a node name for corosync nodeid 3
Aug 11 12:33:36 [13305] mqhavm24 cib: info: pcmk_cpg_membership: Group cib event 12: peer node (node 3 pid 2550) is member
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/1085)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/1086)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/1087)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/1088)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Forwarding cib_modify operation for section cib to all (origin=local/crmd/1089)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1093)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/1094)
Aug 11 12:33:36 [13306] mqhavm24 stonith-ng: info: corosync_node_name: Unable to get node name for nodeid 3
Aug 11 12:33:36 [13306] mqhavm24 stonith-ng: notice: get_node_name: Could not obtain a node name for corosync nodeid 3
Aug 11 12:33:36 [13306] mqhavm24 stonith-ng: info: pcmk_cpg_membership: Group stonith-ng event 12: peer node (node 3 pid 2551) is member
Aug 11 12:33:36 [13305] mqhavm24 cib: info: crm_get_peer: Node 3 is now known as mqhavm34
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: register_fsa_error_adv: Resetting the current action list
Elections:
Aug 11 12:33:36 [13310] mqhavm24 crmd: warning: crmd_ha_msg_filter: Another DC detected: mqhavm37 (op=noop)
Aug 11 12:33:36 [13310] mqhavm24 crmd: notice: do_state_transition: State transition S_IDLE -> S_ELECTION | input=I_ELECTION cause=C_FSA_INTERNAL origin=crmd_ha_msg_filter
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: update_dc: Unset DC. Was mqhavm24
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: election_count_vote: election-DC round 6 (owner node ID 2) pass: vote from mqhavm37 (Uptime)
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: election_count_vote: election-DC round 7 (owner node ID 2) pass: vote from mqhavm37 (Uptime)
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: crm_get_peer: Node 3 is now known as mqhavm34
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: attrd_peer_message: Processing sync-response from mqhavm34
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: attrd_peer_update: Setting #attrd-protocol[mqhavm34]: (null) -> 2 from mqhavm34
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: write_attribute: Processed 2 private changes for #attrd-protocol, id=n/a, set=n/a
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: attrd_peer_update: Setting rdqm-transient-attribute[mqhavm34]: (null) -> 1 from mqhavm34
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_replace: Digest matched on replace from mqhavm34: 3c345690432f9c09a722bbf58085e174
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_replace: Replacement 0.169.51 from mqhavm34 not applied to 0.172.0: current epoch is greater than the replacement
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: write_attribute: Sent CIB request 138 with 2 changes for rdqm-transient-attribute (id n/a, set n/a)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: __xml_diff_object: lrm_resource.p_fs_drgxrde_rdqma moved from 1 to 0
Aug 11 12:33:36 [13305] mqhavm24 cib: info: __xml_diff_object: lrm_resource.p_fs_drgxrde_rdqmb moved from 5 to 1
Aug 11 12:33:36 [13305] mqhavm24 cib: info: __xml_diff_object: lrm_resource.drgxrde_rdqma moved from 0 to 2
Aug 11 12:33:36 [13305] mqhavm24 cib: info: __xml_diff_object: lrm_resource.drgxrde_rdqmb moved from 4 to 2
Aug 11 12:33:36 [13305] mqhavm24 cib: info: __xml_diff_object: lrm_resource.p_drbd_drgxrde_rdqmb moved from 6 to 3
Aug 11 12:33:36 [13305] mqhavm24 cib: info: __xml_diff_object: lrm_resource.p_drbd_dr_drgxrde_rdqma moved from 2 to 4
Aug 11 12:33:36 [13305] mqhavm24 cib: info: __xml_diff_object: lrm_resource.p_drbd_dr_drgxrde_rdqmb moved from 5 to 4
Aug 11 12:33:36 [13305] mqhavm24 cib: info: __xml_diff_object: lrm_resource.p_drbd_drgxrde_rdqma moved from 3 to 5
Aug 11 12:33:36 [13305] mqhavm24 cib: info: __xml_diff_object: lrm_resource.p_rdqmx_drgxrde_rdqma moved from 1 to 6
Aug 11 12:33:36 [13305] mqhavm24 cib: info: __xml_diff_object: lrm_resource.p_rdqmx_drgxrde_rdqmb moved from 4 to 6
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: --- 0.172.0 2
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: +++ 0.172.1 (null)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib: @num_updates=1
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib/status/node_state[@id='3']: @crmd=online, @crm-debug-origin=peer_update_callback
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=mqhavm24/crmd/1085, version=0.172.1)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=mqhavm24/crmd/1086, version=0.172.1)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: --- 0.172.1 2
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: +++ 0.172.2 (null)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib: @num_updates=2
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib/status/node_state[@id='2']: @crmd=online, @crm-debug-origin=peer_update_callback
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=mqhavm24/crmd/1087, version=0.172.2)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=mqhavm24/crmd/1088, version=0.172.2)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: --- 0.172.2 2
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: +++ 0.172.3 4ee9a15c3183a0db2ce37e6fc5615a57
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib: @num_updates=3, @have-quorum=1
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_modify operation for section cib: OK (rc=0, origin=mqhavm24/crmd/1089, version=0.172.3)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=mqhavm24/crmd/1093, version=0.172.3)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: --- 0.172.3 2
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: +++ 0.172.4 (null)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib: @num_updates=4
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib/status/node_state[@id='1']: @crm-debug-origin=post_cache_update
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib/status/node_state[@id='2']: @in_ccm=true, @crm-debug-origin=post_cache_update
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib/status/node_state[@id='3']: @in_ccm=true, @crm-debug-origin=post_cache_update
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=mqhavm24/crmd/1094, version=0.172.4)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/attrd/138)
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: election_check: election-DC won by local node
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: do_log: Input I_ELECTION_DC received in state S_ELECTION from election_win_cb
Aug 11 12:33:36 [13310] mqhavm24 crmd: notice: do_state_transition: State transition S_ELECTION -> S_INTEGRATION | input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=election_win_cb
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: --- 0.172.4 2
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: +++ 0.172.5 (null)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib: @num_updates=5
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ /cib/status/node_state[@id='3']: <transient_attributes id="3"/>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <instance_attributes id="status-3">
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <nvpair id="status-3-rdqm-transient-attribute" name="rdqm-transient-attribute" value="1"/>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </instance_attributes>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </transient_attributes>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=mqhavm24/attrd/138, version=0.172.5)
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: attrd_cib_callback: CIB update 138 result for rdqm-transient-attribute: OK | rc=0
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: attrd_cib_callback: * rdqm-transient-attribute[mqhavm34]=1
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: attrd_cib_callback: * rdqm-transient-attribute[mqhavm24]=1
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: do_dc_takeover: Taking over DC status for this partition
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_master operation for section 'all': OK (rc=0, origin=local/crmd/1095, version=0.172.5)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Forwarding cib_modify operation for section cib to all (origin=local/crmd/1096)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1098)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1100)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Forwarding cib_modify operation for section crm_config to all (origin=local/crmd/1102)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_modify operation for section cib: OK (rc=0, origin=mqhavm24/crmd/1096, version=0.172.5)
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: corosync_cluster_name: Cannot get totem.cluster_name: CS_ERR_NOT_EXIST (12)
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: join_make_offer: Making join-4 offers based on membership event 3112554
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: join_make_offer: Sending join-4 offer to mqhavm34
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: join_make_offer: Sending join-4 offer to mqhavm24
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: join_make_offer: Sending join-4 offer to mqhavm37
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: do_dc_join_offer_all: Waiting on join-4 requests from 3 outstanding nodes
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_modify operation for section crm_config: OK (rc=0, origin=mqhavm24/crmd/1098, version=0.172.5)
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: update_dc: Set DC to mqhavm24 (3.0.14)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_modify operation for section crm_config: OK (rc=0, origin=mqhavm24/crmd/1100, version=0.172.5)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_modify operation for section crm_config: OK (rc=0, origin=mqhavm24/crmd/1102, version=0.172.5)
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN | input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1106)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1107)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1108)
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: controld_delete_node_state: Deleting resource history for node mqhavm24 (via CIB call 1109) | xpath=//node_state[@uname='mqhavm24']/lrm
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: controld_delete_node_state: Deleting resource history for node mqhavm37 (via CIB call 1111) | xpath=//node_state[@uname='mqhavm37']/lrm
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_replace: Digest matched on replace from mqhavm24: f257c75862a5238d3303815722d3205b
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_replace: Replaced 0.172.5 with 0.172.5 from mqhavm24
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_replace operation for section 'all': OK (rc=0, origin=mqhavm24/crmd/1105, version=0.172.5)
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: controld_delete_node_state: Deleting resource history for node mqhavm34 (via CIB call 1113) | xpath=//node_state[@uname='mqhavm34']/lrm
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=mqhavm24/crmd/1106, version=0.172.5)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=mqhavm24/crmd/1107, version=0.172.5)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=mqhavm24/crmd/1108, version=0.172.5)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Forwarding cib_delete operation for section //node_state[@uname='mqhavm24']/lrm to all (origin=local/crmd/1109)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/1110)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Forwarding cib_delete operation for section //node_state[@uname='mqhavm37']/lrm to all (origin=local/crmd/1111)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/1112)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Forwarding cib_delete operation for section //node_state[@uname='mqhavm34']/lrm to all (origin=local/crmd/1113)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: --- 0.172.5 2
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: +++ 0.172.6 (null)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: -- /cib/status/node_state[@id='1']/lrm[@id='1']
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib: @num_updates=6
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_delete operation for section //node_state[@uname='mqhavm24']/lrm: OK (rc=0, origin=mqhavm24/crmd/1109, version=0.172.6)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: --- 0.172.6 2
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: +++ 0.172.7 (null)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib: @num_updates=7
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_lrm_query_internal
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ /cib/status/node_state[@id='1']: <lrm id="1"/>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_resources>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_resource id="p_fs_drgxrde_rdqma" type="Filesystem" class="ocf" provider="heartbeat">
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_fs_drgxrde_rdqma_last_0" operation_key="p_fs_drgxrde_rdqma_stop_0" operation="stop" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="5:155:0:f0b9e946-fb53-4805-bea5-05c841b38129" transition-magic="0:0;5:155:0:f0b9e946-fb53-4805-bea5-05c841b38129" exit-reason="" on_node="mqhavm24" call-id="1095" rc-code="0" op-st
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </lrm_resource>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_resource id="p_fs_drgxrde_rdqmb" type="Filesystem" class="ocf" provider="heartbeat">
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_fs_drgxrde_rdqmb_last_0" operation_key="p_fs_drgxrde_rdqmb_stop_0" operation="stop" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="61:155:0:f0b9e946-fb53-4805-bea5-05c841b38129" transition-magic="0:0;61:155:0:f0b9e946-fb53-4805-bea5-05c841b38129" exit-reason="" on_node="mqhavm24" call-id="1096" rc-code="0" op-
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </lrm_resource>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_resource id="p_drbd_drgxrde_rdqma" type="drbd" class="ocf" provider="linbit">
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_drbd_drgxrde_rdqma_last_failure_0" operation_key="p_drbd_drgxrde_rdqma_monitor_0" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="5:14:7:f0b9e946-fb53-4805-bea5-05c841b38129" transition-magic="0:8;5:14:7:f0b9e946-fb53-4805-bea5-05c841b38129" exit-reason="" on_node="mqhavm24" call-id="24" rc-
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_drbd_drgxrde_rdqma_last_0" operation_key="p_drbd_drgxrde_rdqma_stop_0" operation="stop" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="27:157:0:f0b9e946-fb53-4805-bea5-05c841b38129" transition-magic="0:0;27:157:0:f0b9e946-fb53-4805-bea5-05c841b38129" exit-reason="" on_node="mqhavm24" call-id="1143" rc-code="0"
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </lrm_resource>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_resource id="drgxrde_rdqma" type="rdqm" class="ocf" provider="ibm">
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_rsc_op id="drgxrde_rdqma_last_0" operation_key="drgxrde_rdqma_stop_0" operation="stop" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="9:154:0:f0b9e946-fb53-4805-bea5-05c841b38129" transition-magic="0:0;9:154:0:f0b9e946-fb53-4805-bea5-05c841b38129" exit-reason="" on_node="mqhavm24" call-id="1060" rc-code="0" op-status="0" i
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </lrm_resource>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_resource id="drgxrde_rdqmb" type="rdqm" class="ocf" provider="ibm">
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_rsc_op id="drgxrde_rdqmb_last_0" operation_key="drgxrde_rdqmb_stop_0" operation="stop" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="69:154:0:f0b9e946-fb53-4805-bea5-05c841b38129" transition-magic="0:0;69:154:0:f0b9e946-fb53-4805-bea5-05c841b38129" exit-reason="" on_node="mqhavm24" call-id="1062" rc-code="0" op-status="0"
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </lrm_resource>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_resource id="p_drbd_dr_drgxrde_rdqma" type="drbd" class="ocf" provider="linbit">
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_drbd_dr_drgxrde_rdqma_last_failure_0" operation_key="p_drbd_dr_drgxrde_rdqma_monitor_0" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="4:14:7:f0b9e946-fb53-4805-bea5-05c841b38129" transition-magic="0:8;4:14:7:f0b9e946-fb53-4805-bea5-05c841b38129" exit-reason="" on_node="mqhavm24" call-id="1
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_drbd_dr_drgxrde_rdqma_last_0" operation_key="p_drbd_dr_drgxrde_rdqma_stop_0" operation="stop" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="1:156:0:f0b9e946-fb53-4805-bea5-05c841b38129" transition-magic="0:0;1:156:0:f0b9e946-fb53-4805-bea5-05c841b38129" exit-reason="" on_node="mqhavm24" call-id="1122" rc-code
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </lrm_resource>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_resource id="p_drbd_drgxrde_rdqmb" type="drbd" class="ocf" provider="linbit">
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_drbd_drgxrde_rdqmb_last_failure_0" operation_key="p_drbd_drgxrde_rdqmb_monitor_0" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="10:16:7:f0b9e946-fb53-4805-bea5-05c841b38129" transition-magic="0:8;10:16:7:f0b9e946-fb53-4805-bea5-05c841b38129" exit-reason="" on_node="mqhavm24" call-id="60" r
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_drbd_drgxrde_rdqmb_last_0" operation_key="p_drbd_drgxrde_rdqmb_stop_0" operation="stop" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="79:157:0:f0b9e946-fb53-4805-bea5-05c841b38129" transition-magic="0:0;79:157:0:f0b9e946-fb53-4805-bea5-05c841b38129" exit-reason="" on_node="mqhavm24" call-id="1144" rc-code="0"
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </lrm_resource>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_resource id="p_drbd_dr_drgxrde_rdqmb" type="drbd" class="ocf" provider="linbit">
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_drbd_dr_drgxrde_rdqmb_last_failure_0" operation_key="p_drbd_dr_drgxrde_rdqmb_monitor_0" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="9:16:7:f0b9e946-fb53-4805-bea5-05c841b38129" transition-magic="0:8;9:16:7:f0b9e946-fb53-4805-bea5-05c841b38129" exit-reason="" on_node="mqhavm24" call-id="5
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_drbd_dr_drgxrde_rdqmb_last_0" operation_key="p_drbd_dr_drgxrde_rdqmb_stop_0" operation="stop" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="54:156:0:f0b9e946-fb53-4805-bea5-05c841b38129" transition-magic="0:0;54:156:0:f0b9e946-fb53-4805-bea5-05c841b38129" exit-reason="" on_node="mqhavm24" call-id="1123" rc-co
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </lrm_resource>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_resource id="p_ip_drgxrde_rdqma" type="IPaddr2" class="ocf" provider="heartbeat">
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_ip_drgxrde_rdqma_last_0" operation_key="p_ip_drgxrde_rdqma_stop_0" operation="stop" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="129:154:0:f0b9e946-fb53-4805-bea5-05c841b38129" transition-magic="0:0;129:154:0:f0b9e946-fb53-4805-bea5-05c841b38129" exit-reason="" on_node="mqhavm24" call-id="1075" rc-code="0" o
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </lrm_resource>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_resource id="p_rdqmx_drgxrde_rdqma" type="rdqmx" class="ocf" provider="ibm">
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_rdqmx_drgxrde_rdqma_last_0" operation_key="p_rdqmx_drgxrde_rdqma_stop_0" operation="stop" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="6:155:0:f0b9e946-fb53-4805-bea5-05c841b38129" transition-magic="0:0;6:155:0:f0b9e946-fb53-4805-bea5-05c841b38129" exit-reason="" on_node="mqhavm24" call-id="1083" rc-code="0"
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </lrm_resource>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_resource id="p_rdqmx_drgxrde_rdqmb" type="rdqmx" class="ocf" provider="ibm">
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_rdqmx_drgxrde_rdqmb_last_0" operation_key="p_rdqmx_drgxrde_rdqmb_stop_0" operation="stop" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="62:155:0:f0b9e946-fb53-4805-bea5-05c841b38129" transition-magic="0:0;62:155:0:f0b9e946-fb53-4805-bea5-05c841b38129" exit-reason="" on_node="mqhavm24" call-id="1085" rc-code="
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </lrm_resource>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_resource id="p_ip_drgxrde_rdqmb" type="IPaddr2" class="ocf" provider="heartbeat">
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_ip_drgxrde_rdqmb_last_0" operation_key="p_ip_drgxrde_rdqmb_stop_0" operation="stop" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="131:154:0:f0b9e946-fb53-4805-bea5-05c841b38129" transition-magic="0:0;131:154:0:f0b9e946-fb53-4805-bea5-05c841b38129" exit-reason="" on_node="mqhavm24" call-id="1079" rc-code="0" o
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </lrm_resource>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </lrm_resources>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </lrm>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=mqhavm24/crmd/1110, version=0.172.7)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: --- 0.172.7 2
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: +++ 0.172.8 (null)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: -- /cib/status/node_state[@id='2']/lrm[@id='2']
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib: @num_updates=8
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_delete operation for section //node_state[@uname='mqhavm37']/lrm: OK (rc=0, origin=mqhavm24/crmd/1111, version=0.172.8)
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: do_state_transition: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE | input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: abort_transition_graph: Transition aborted: Peer Cancelled | source=do_te_invoke:143 complete=true
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: --- 0.172.8 2
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: +++ 0.172.9 (null)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib: @num_updates=9
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_lrm_query_internal, @join=member
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ /cib/status/node_state[@id='2']: <lrm id="2"/>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_resources>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_resource id="p_fs_drgxrde_rdqma" type="Filesystem" class="ocf" provider="heartbeat">
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_fs_drgxrde_rdqma_last_0" operation_key="p_fs_drgxrde_rdqma_monitor_0" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="12:14:7:f0b9e946-fb53-4805-bea5-05c841b38129" transition-magic="0:7;12:14:7:f0b9e946-fb53-4805-bea5-05c841b38129" exit-reason="" on_node="mqhavm37" call-id="9" rc-code="7" op
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </lrm_resource>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_resource id="p_fs_drgxrde_rdqmb" type="Filesystem" class="ocf" provider="heartbeat">
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_fs_drgxrde_rdqmb_last_0" operation_key="p_fs_drgxrde_rdqmb_monitor_0" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="17:16:7:f0b9e946-fb53-4805-bea5-05c841b38129" transition-magic="0:7;17:16:7:f0b9e946-fb53-4805-bea5-05c841b38129" exit-reason="" on_node="mqhavm37" call-id="34" rc-code="7" o
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </lrm_resource>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_resource id="p_drbd_drgxrde_rdqma" type="drbd" class="ocf" provider="linbit">
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_drbd_drgxrde_rdqma_last_failure_0" operation_key="p_drbd_drgxrde_rdqma_monitor_0" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="15:14:7:f0b9e946-fb53-4805-bea5-05c841b38129" transition-magic="0:0;15:14:7:f0b9e946-fb53-4805-bea5-05c841b38129" exit-reason="" on_node="mqhavm37" call-id="23" r
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_drbd_drgxrde_rdqma_last_0" operation_key="p_drbd_drgxrde_rdqma_stop_0" operation="stop" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="30:2:0:1816128c-a49e-4b13-b1b6-ee3672c04867" transition-magic="0:0;30:2:0:1816128c-a49e-4b13-b1b6-ee3672c04867" exit-reason="" on_node="mqhavm37" call-id="354" rc-code="0" op-s
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </lrm_resource>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_resource id="drgxrde_rdqma" type="rdqm" class="ocf" provider="ibm">
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_rsc_op id="drgxrde_rdqma_last_0" operation_key="drgxrde_rdqma_monitor_0" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="11:14:7:f0b9e946-fb53-4805-bea5-05c841b38129" transition-magic="0:7;11:14:7:f0b9e946-fb53-4805-bea5-05c841b38129" exit-reason="" on_node="mqhavm37" call-id="5" rc-code="7" op-status="0
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </lrm_resource>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_resource id="drgxrde_rdqmb" type="rdqm" class="ocf" provider="ibm">
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_rsc_op id="drgxrde_rdqmb_last_0" operation_key="drgxrde_rdqmb_monitor_0" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="16:16:7:f0b9e946-fb53-4805-bea5-05c841b38129" transition-magic="0:7;16:16:7:f0b9e946-fb53-4805-bea5-05c841b38129" exit-reason="" on_node="mqhavm37" call-id="30" rc-code="7" op-status="
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </lrm_resource>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_resource id="p_drbd_dr_drgxrde_rdqma" type="drbd" class="ocf" provider="linbit">
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_drbd_dr_drgxrde_rdqma_last_0" operation_key="p_drbd_dr_drgxrde_rdqma_monitor_0" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="14:14:7:f0b9e946-fb53-4805-bea5-05c841b38129" transition-magic="0:7;14:14:7:f0b9e946-fb53-4805-bea5-05c841b38129" exit-reason="" on_node="mqhavm37" call-id="18" rc-
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </lrm_resource>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_resource id="p_drbd_dr_drgxrde_rdqmb" type="drbd" class="ocf" provider="linbit">
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_drbd_dr_drgxrde_rdqmb_last_0" operation_key="p_drbd_dr_drgxrde_rdqmb_monitor_0" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="19:16:7:f0b9e946-fb53-4805-bea5-05c841b38129" transition-magic="0:7;19:16:7:f0b9e946-fb53-4805-bea5-05c841b38129" exit-reason="" on_node="mqhavm37" call-id="43" rc-
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </lrm_resource>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_resource id="p_drbd_drgxrde_rdqmb" type="drbd" class="ocf" provider="linbit">
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_drbd_drgxrde_rdqmb_last_failure_0" operation_key="p_drbd_drgxrde_rdqmb_monitor_0" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="20:16:7:f0b9e946-fb53-4805-bea5-05c841b38129" transition-magic="0:0;20:16:7:f0b9e946-fb53-4805-bea5-05c841b38129" exit-reason="" on_node="mqhavm37" call-id="48" r
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_drbd_drgxrde_rdqmb_last_0" operation_key="p_drbd_drgxrde_rdqmb_stop_0" operation="stop" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="83:2:0:1816128c-a49e-4b13-b1b6-ee3672c04867" transition-magic="0:0;83:2:0:1816128c-a49e-4b13-b1b6-ee3672c04867" exit-reason="" on_node="mqhavm37" call-id="356" rc-code="0" op-s
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </lrm_resource>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_resource id="p_ip_drgxrde_rdqma" type="IPaddr2" class="ocf" provider="heartbeat">
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_ip_drgxrde_rdqma_last_0" operation_key="p_ip_drgxrde_rdqma_monitor_0" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="13:35:7:f0b9e946-fb53-4805-bea5-05c841b38129" transition-magic="0:7;13:35:7:f0b9e946-fb53-4805-bea5-05c841b38129" exit-reason="" on_node="mqhavm37" call-id="55" rc-code="7" o
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </lrm_resource>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_resource id="p_rdqmx_drgxrde_rdqma" type="rdqmx" class="ocf" provider="ibm">
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_rdqmx_drgxrde_rdqma_last_0" operation_key="p_rdqmx_drgxrde_rdqma_monitor_0" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="13:14:7:f0b9e946-fb53-4805-bea5-05c841b38129" transition-magic="0:7;13:14:7:f0b9e946-fb53-4805-bea5-05c841b38129" exit-reason="" on_node="mqhavm37" call-id="13" rc-code
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </lrm_resource>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_resource id="p_rdqmx_drgxrde_rdqmb" type="rdqmx" class="ocf" provider="ibm">
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_rdqmx_drgxrde_rdqmb_last_0" operation_key="p_rdqmx_drgxrde_rdqmb_monitor_0" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="18:16:7:f0b9e946-fb53-4805-bea5-05c841b38129" transition-magic="0:7;18:16:7:f0b9e946-fb53-4805-bea5-05c841b38129" exit-reason="" on_node="mqhavm37" call-id="38" rc-code
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </lrm_resource>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_resource id="p_ip_drgxrde_rdqmb" type="IPaddr2" class="ocf" provider="heartbeat">
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_ip_drgxrde_rdqmb_last_0" operation_key="p_ip_drgxrde_rdqmb_monitor_0" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="14:39:7:f0b9e946-fb53-4805-bea5-05c841b38129" transition-magic="0:7;14:39:7:f0b9e946-fb53-4805-bea5-05c841b38129" exit-reason="" on_node="mqhavm37" call-id="59" rc-code="7" o
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </lrm_resource>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </lrm_resources>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </lrm>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=mqhavm24/crmd/1112, version=0.172.9)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: --- 0.172.9 2
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: +++ 0.172.10 (null)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: -- /cib/status/node_state[@id='3']/lrm[@id='3']
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib: @num_updates=10
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_delete operation for section //node_state[@uname='mqhavm34']/lrm: OK (rc=0, origin=mqhavm24/crmd/1113, version=0.172.10)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/1114)
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: abort_transition_graph: Transition aborted: LRM Refresh | source=process_resource_updates:294 complete=true
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Forwarding cib_modify operation for section nodes to all (origin=local/crmd/1118)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/crmd/1119)
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: abort_transition_graph: Transition aborted by deletion of lrm[@id='3']: Resource state removal | cib=0.172.10 source=abort_unless_down:370 path=/cib/status/node_state[@id='3']/lrm[@id='3'] complete=true
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Forwarding cib_modify operation for section cib to all (origin=local/crmd/1120)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: --- 0.172.10 2
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: +++ 0.172.11 (null)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib: @num_updates=11
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib/status/node_state[@id='3']: @crm-debug-origin=do_lrm_query_internal, @join=member
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ /cib/status/node_state[@id='3']: <lrm id="3"/>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_resources>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_resource id="p_fs_drgxrde_rdqma" type="Filesystem" class="ocf" provider="heartbeat">
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_fs_drgxrde_rdqma_last_0" operation_key="p_fs_drgxrde_rdqma_monitor_0" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="8:147:7:f0b9e946-fb53-4805-bea5-05c841b38129" transition-magic="0:7;8:147:7:f0b9e946-fb53-4805-bea5-05c841b38129" exit-reason="" on_node="mqhavm34" call-id="10" rc-code="7" o
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </lrm_resource>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_resource id="p_fs_drgxrde_rdqmb" type="Filesystem" class="ocf" provider="heartbeat">
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_fs_drgxrde_rdqmb_last_0" operation_key="p_fs_drgxrde_rdqmb_monitor_0" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="13:147:7:f0b9e946-fb53-4805-bea5-05c841b38129" transition-magic="0:7;13:147:7:f0b9e946-fb53-4805-bea5-05c841b38129" exit-reason="" on_node="mqhavm34" call-id="32" rc-code="7"
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </lrm_resource>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_resource id="drgxrde_rdqma" type="rdqm" class="ocf" provider="ibm">
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_rsc_op id="drgxrde_rdqma_last_0" operation_key="drgxrde_rdqma_monitor_0" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="7:147:7:f0b9e946-fb53-4805-bea5-05c841b38129" transition-magic="0:7;7:147:7:f0b9e946-fb53-4805-bea5-05c841b38129" exit-reason="" on_node="mqhavm34" call-id="6" rc-code="7" op-status="0
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </lrm_resource>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_resource id="drgxrde_rdqmb" type="rdqm" class="ocf" provider="ibm">
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_rsc_op id="drgxrde_rdqmb_last_0" operation_key="drgxrde_rdqmb_monitor_0" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="12:147:7:f0b9e946-fb53-4805-bea5-05c841b38129" transition-magic="0:7;12:147:7:f0b9e946-fb53-4805-bea5-05c841b38129" exit-reason="" on_node="mqhavm34" call-id="28" rc-code="7" op-status
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </lrm_resource>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_resource id="p_drbd_drgxrde_rdqmb" type="drbd" class="ocf" provider="linbit">
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_drbd_drgxrde_rdqmb_last_0" operation_key="p_drbd_drgxrde_rdqmb_stop_0" operation="stop" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="83:0:0:18b7675e-eee2-4abb-8521-a55663441465" transition-magic="0:0;83:0:0:18b7675e-eee2-4abb-8521-a55663441465" exit-reason="" on_node="mqhavm34" call-id="77" rc-code="0" op-st
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </lrm_resource>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_resource id="p_drbd_dr_drgxrde_rdqma" type="drbd" class="ocf" provider="linbit">
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_drbd_dr_drgxrde_rdqma_last_0" operation_key="p_drbd_dr_drgxrde_rdqma_monitor_0" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="10:147:7:f0b9e946-fb53-4805-bea5-05c841b38129" transition-magic="0:7;10:147:7:f0b9e946-fb53-4805-bea5-05c841b38129" exit-reason="" on_node="mqhavm34" call-id="19" r
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </lrm_resource>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_resource id="p_drbd_dr_drgxrde_rdqmb" type="drbd" class="ocf" provider="linbit">
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_drbd_dr_drgxrde_rdqmb_last_0" operation_key="p_drbd_dr_drgxrde_rdqmb_monitor_0" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="15:147:7:f0b9e946-fb53-4805-bea5-05c841b38129" transition-magic="0:7;15:147:7:f0b9e946-fb53-4805-bea5-05c841b38129" exit-reason="" on_node="mqhavm34" call-id="41" r
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </lrm_resource>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_resource id="p_drbd_drgxrde_rdqma" type="drbd" class="ocf" provider="linbit">
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_drbd_drgxrde_rdqma_last_0" operation_key="p_drbd_drgxrde_rdqma_stop_0" operation="stop" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="30:0:0:18b7675e-eee2-4abb-8521-a55663441465" transition-magic="0:0;30:0:0:18b7675e-eee2-4abb-8521-a55663441465" exit-reason="" on_node="mqhavm34" call-id="79" rc-code="0" op-st
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </lrm_resource>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_resource id="p_ip_drgxrde_rdqma" type="IPaddr2" class="ocf" provider="heartbeat">
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_ip_drgxrde_rdqma_last_0" operation_key="p_ip_drgxrde_rdqma_monitor_0" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="17:147:7:f0b9e946-fb53-4805-bea5-05c841b38129" transition-magic="0:7;17:147:7:f0b9e946-fb53-4805-bea5-05c841b38129" exit-reason="" on_node="mqhavm34" call-id="50" rc-code="7"
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </lrm_resource>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_resource id="p_rdqmx_drgxrde_rdqma" type="rdqmx" class="ocf" provider="ibm">
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_rdqmx_drgxrde_rdqma_last_0" operation_key="p_rdqmx_drgxrde_rdqma_monitor_0" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="9:147:7:f0b9e946-fb53-4805-bea5-05c841b38129" transition-magic="0:7;9:147:7:f0b9e946-fb53-4805-bea5-05c841b38129" exit-reason="" on_node="mqhavm34" call-id="14" rc-code
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </lrm_resource>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_resource id="p_rdqmx_drgxrde_rdqmb" type="rdqmx" class="ocf" provider="ibm">
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_rdqmx_drgxrde_rdqmb_last_0" operation_key="p_rdqmx_drgxrde_rdqmb_monitor_0" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="14:147:7:f0b9e946-fb53-4805-bea5-05c841b38129" transition-magic="0:7;14:147:7:f0b9e946-fb53-4805-bea5-05c841b38129" exit-reason="" on_node="mqhavm34" call-id="36" rc-co
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </lrm_resource>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_resource id="p_ip_drgxrde_rdqmb" type="IPaddr2" class="ocf" provider="heartbeat">
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <lrm_rsc_op id="p_ip_drgxrde_rdqmb_last_0" operation_key="p_ip_drgxrde_rdqmb_monitor_0" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.14" transition-key="18:147:7:f0b9e946-fb53-4805-bea5-05c841b38129" transition-magic="0:7;18:147:7:f0b9e946-fb53-4805-bea5-05c841b38129" exit-reason="" on_node="mqhavm34" call-id="54" rc-code="7"
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </lrm_resource>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </lrm_resources>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </lrm>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=mqhavm24/crmd/1114, version=0.172.11)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=mqhavm24/crmd/1118, version=0.172.11)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: --- 0.172.11 2
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: +++ 0.172.12 (null)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib: @num_updates=12
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib/status/node_state[@id='3']: @crm-debug-origin=do_state_transition
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=mqhavm24/crmd/1119, version=0.172.12)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_modify operation for section cib: OK (rc=0, origin=mqhavm24/crmd/1120, version=0.172.12)
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: abort_transition_graph: Transition aborted: LRM Refresh | source=process_resource_updates:294 complete=true
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_file_backup: Archived previous version as /var/lib/pacemaker/cib/cib-79.raw
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_file_write_with_digest: Wrote version 0.172.0 of the CIB to disk (digest: e499c1e040d16e4a97fce7f8b0c5bf32)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_file_write_with_digest: Reading cluster configuration file /var/lib/pacemaker/cib/cib.Jrr9mZ (digest: /var/lib/pacemaker/cib/cib.7vPm5x)
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: determine_online_status: Node mqhavm24 is online
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: determine_online_status: Node mqhavm37 is online
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: determine_online_status: Node mqhavm34 is online
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: determine_op_status: Operation monitor found resource p_drbd_drgxrde_rdqma:0 active in master mode on mqhavm24
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: determine_op_status: Operation monitor found resource p_drbd_dr_drgxrde_rdqma:0 active in master mode on mqhavm24
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: determine_op_status: Operation monitor found resource p_drbd_drgxrde_rdqmb:0 active in master mode on mqhavm24
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: determine_op_status: Operation monitor found resource p_drbd_dr_drgxrde_rdqmb:0 active in master mode on mqhavm24
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: determine_op_status: Operation monitor found resource p_drbd_drgxrde_rdqma:0 active on mqhavm37
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: determine_op_status: Operation monitor found resource p_drbd_drgxrde_rdqmb:0 active on mqhavm37
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: unpack_node_loop: Node 1 is already processed
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: unpack_node_loop: Node 2 is already processed
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: unpack_node_loop: Node 3 is already processed
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: unpack_node_loop: Node 1 is already processed
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: unpack_node_loop: Node 2 is already processed
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: unpack_node_loop: Node 3 is already processed
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: common_print: drgxrde_rdqma (ocf::ibm:rdqm): Stopped
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: common_print: p_fs_drgxrde_rdqma (ocf::heartbeat:Filesystem): Stopped
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: common_print: p_rdqmx_drgxrde_rdqma (ocf::ibm:rdqmx): Stopped
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: clone_print: Master/Slave Set: ms_drbd_dr_drgxrde_rdqma [p_drbd_dr_drgxrde_rdqma]
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: short_print: Stopped: [ mqhavm24 mqhavm34 mqhavm37 ]
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: clone_print: Master/Slave Set: ms_drbd_drgxrde_rdqma [p_drbd_drgxrde_rdqma]
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: short_print: Stopped: [ mqhavm24 mqhavm34 mqhavm37 ]
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: common_print: drgxrde_rdqmb (ocf::ibm:rdqm): Stopped
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: common_print: p_fs_drgxrde_rdqmb (ocf::heartbeat:Filesystem): Stopped
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: common_print: p_rdqmx_drgxrde_rdqmb (ocf::ibm:rdqmx): Stopped
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: clone_print: Master/Slave Set: ms_drbd_dr_drgxrde_rdqmb [p_drbd_dr_drgxrde_rdqmb]
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: short_print: Stopped: [ mqhavm24 mqhavm34 mqhavm37 ]
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: clone_print: Master/Slave Set: ms_drbd_drgxrde_rdqmb [p_drbd_drgxrde_rdqmb]
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: short_print: Stopped: [ mqhavm24 mqhavm34 mqhavm37 ]
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: common_print: p_ip_drgxrde_rdqma (ocf::heartbeat:IPaddr2): Stopped
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: common_print: p_ip_drgxrde_rdqmb (ocf::heartbeat:IPaddr2): Stopped
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__set_instance_roles: ms_drbd_drgxrde_rdqma: Promoted 0 instances of a possible 1 to master
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_merge_weights: p_drbd_dr_drgxrde_rdqma:0: Rolling back optional scores from p_fs_drgxrde_rdqma
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_allocate: Resource p_drbd_dr_drgxrde_rdqma:0 cannot run anywhere
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__set_instance_roles: ms_drbd_dr_drgxrde_rdqma: Promoted 0 instances of a possible 1 to master
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_merge_weights: p_fs_drgxrde_rdqma: Rolling back optional scores from p_rdqmx_drgxrde_rdqma
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_allocate: Resource p_fs_drgxrde_rdqma cannot run anywhere
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_merge_weights: p_rdqmx_drgxrde_rdqma: Rolling back optional scores from p_ip_drgxrde_rdqma
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_allocate: Resource p_rdqmx_drgxrde_rdqma cannot run anywhere
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_merge_weights: p_ip_drgxrde_rdqma: Rolling back optional scores from drgxrde_rdqma
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_allocate: Resource p_ip_drgxrde_rdqma cannot run anywhere
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_allocate: Resource drgxrde_rdqma cannot run anywhere
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__set_instance_roles: ms_drbd_drgxrde_rdqmb: Promoted 0 instances of a possible 1 to master
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_merge_weights: p_drbd_dr_drgxrde_rdqmb:0: Rolling back optional scores from p_fs_drgxrde_rdqmb
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_allocate: Resource p_drbd_dr_drgxrde_rdqmb:0 cannot run anywhere
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__set_instance_roles: ms_drbd_dr_drgxrde_rdqmb: Promoted 0 instances of a possible 1 to master
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_merge_weights: p_fs_drgxrde_rdqmb: Rolling back optional scores from p_rdqmx_drgxrde_rdqmb
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_allocate: Resource p_fs_drgxrde_rdqmb cannot run anywhere
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_merge_weights: p_rdqmx_drgxrde_rdqmb: Rolling back optional scores from p_ip_drgxrde_rdqmb
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_allocate: Resource p_rdqmx_drgxrde_rdqmb cannot run anywhere
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_merge_weights: p_ip_drgxrde_rdqmb: Rolling back optional scores from drgxrde_rdqmb
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_allocate: Resource p_ip_drgxrde_rdqmb cannot run anywhere
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_allocate: Resource drgxrde_rdqmb cannot run anywhere
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: RecurringOp: Start recurring monitor (20s) for p_drbd_drgxrde_rdqma:0 on mqhavm24
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: RecurringOp: Start recurring monitor (20s) for p_drbd_drgxrde_rdqma:1 on mqhavm34
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: RecurringOp: Start recurring monitor (20s) for p_drbd_drgxrde_rdqma:2 on mqhavm37
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: RecurringOp: Start recurring monitor (20s) for p_drbd_drgxrde_rdqma:0 on mqhavm24
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: RecurringOp: Start recurring monitor (20s) for p_drbd_drgxrde_rdqma:1 on mqhavm34
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: RecurringOp: Start recurring monitor (20s) for p_drbd_drgxrde_rdqma:2 on mqhavm37
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: RecurringOp: Start recurring monitor (20s) for p_drbd_drgxrde_rdqmb:0 on mqhavm24
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: RecurringOp: Start recurring monitor (20s) for p_drbd_drgxrde_rdqmb:1 on mqhavm34
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: RecurringOp: Start recurring monitor (20s) for p_drbd_drgxrde_rdqmb:2 on mqhavm37
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: RecurringOp: Start recurring monitor (20s) for p_drbd_drgxrde_rdqmb:0 on mqhavm24
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: RecurringOp: Start recurring monitor (20s) for p_drbd_drgxrde_rdqmb:1 on mqhavm34
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: RecurringOp: Start recurring monitor (20s) for p_drbd_drgxrde_rdqmb:2 on mqhavm37
Aug 11 12:33:36 [13306] mqhavm24 stonith-ng: info: crm_get_peer: Node 3 is now known as mqhavm34
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: LogActions: Leave drgxrde_rdqma (Stopped)
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: LogActions: Leave p_fs_drgxrde_rdqma (Stopped)
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: LogActions: Leave p_rdqmx_drgxrde_rdqma (Stopped)
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: LogActions: Leave p_drbd_dr_drgxrde_rdqma:0 (Stopped)
Aug 11 12:33:36 [13309] mqhavm24 pengine: notice: LogAction: * Start p_drbd_drgxrde_rdqma:0 ( mqhavm24 )
Aug 11 12:33:36 [13309] mqhavm24 pengine: notice: LogAction: * Start p_drbd_drgxrde_rdqma:1 ( mqhavm34 )
Aug 11 12:33:36 [13309] mqhavm24 pengine: notice: LogAction: * Start p_drbd_drgxrde_rdqma:2 ( mqhavm37 )
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: LogActions: Leave drgxrde_rdqmb (Stopped)
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: LogActions: Leave p_fs_drgxrde_rdqmb (Stopped)
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: LogActions: Leave p_rdqmx_drgxrde_rdqmb (Stopped)
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: LogActions: Leave p_drbd_dr_drgxrde_rdqmb:0 (Stopped)
Aug 11 12:33:36 [13309] mqhavm24 pengine: notice: LogAction: * Start p_drbd_drgxrde_rdqmb:0 ( mqhavm24 )
Aug 11 12:33:36 [13309] mqhavm24 pengine: notice: LogAction: * Start p_drbd_drgxrde_rdqmb:1 ( mqhavm34 )
Aug 11 12:33:36 [13309] mqhavm24 pengine: notice: LogAction: * Start p_drbd_drgxrde_rdqmb:2 ( mqhavm37 )
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: LogActions: Leave p_ip_drgxrde_rdqma (Stopped)
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: LogActions: Leave p_ip_drgxrde_rdqmb (Stopped)
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: crm_get_peer: Node 2 is now known as mqhavm37
Aug 11 12:33:36 [13308] mqhavm24 attrd: notice: attrd_check_for_new_writer: Detected another attribute writer (mqhavm37), starting new election
^^^ This one looks fishy.
attrd / cib writer should have followed DC
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: attrd_peer_message: Processing sync-response from mqhavm37
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: attrd_peer_update: Setting #attrd-protocol[mqhavm37]: (null) -> 2 from mqhavm37
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: attrd_peer_update: Setting rdqm-transient-attribute[mqhavm37]: (null) -> 1 from mqhavm37
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: election_count_vote: election-attrd round 10 (owner node ID 2) pass: vote from mqhavm37 (Uptime)
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: attrd_peer_message: Processing sync-response from mqhavm37
Aug 11 12:33:36 [13303] mqhavm24 pacemakerd: info: crm_cs_flush: Sent 2 CPG messages (0 remaining, last=19): OK (1)
Aug 11 12:33:36 [13309] mqhavm24 pengine: notice: process_pe_message: Calculated transition 160, saving inputs in /var/lib/pacemaker/pengine/pe-input-688.bz2
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: handle_response: pe_calc calculation pe_calc-dc-1660217616-1276 is obsolete
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: election_check: election-attrd won by local node
Aug 11 12:33:36 [13308] mqhavm24 attrd: notice: attrd_declare_winner: Recorded local node as attribute writer (was unset)
which it does now.
Aug 11 12:33:36 [13303] mqhavm24 pacemakerd: info: mcp_cpg_deliver: Ignoring process list sent by peer for local node
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: write_attribute: Sent CIB request 139 with 1 change for master-p_drbd_dr_drgxrde_rdqma (id n/a, set n/a)
Aug 11 12:33:36 [13303] mqhavm24 pacemakerd: info: mcp_cpg_deliver: Ignoring process list sent by peer for local node
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: write_attribute: Sent CIB request 140 with 1 change for master-p_drbd_dr_drgxrde_rdqmb (id n/a, set n/a)
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: write_attribute: Processed 3 private changes for #attrd-protocol, id=n/a, set=n/a
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: write_attribute: Sent CIB request 141 with 3 changes for rdqm-transient-attribute (id n/a, set n/a)
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: write_attribute: Sent CIB request 142 with 3 changes for master-p_drbd_drgxrde_rdqma (id n/a, set n/a)
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: write_attribute: Sent CIB request 143 with 3 changes for master-p_drbd_drgxrde_rdqmb (id n/a, set n/a)
Aug 11 12:33:36 [13306] mqhavm24 stonith-ng: info: crm_get_peer: Node 2 is now known as mqhavm37
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/attrd/139)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/attrd/140)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/attrd/141)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/attrd/142)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Forwarding cib_modify operation for section status to all (origin=local/attrd/143)
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: process_pe_message: Input has not changed since last time, not saving to disk
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: determine_online_status: Node mqhavm24 is online
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: determine_online_status: Node mqhavm37 is online
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: determine_online_status: Node mqhavm34 is online
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: determine_op_status: Operation monitor found resource p_drbd_drgxrde_rdqma:0 active in master mode on mqhavm24
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: determine_op_status: Operation monitor found resource p_drbd_dr_drgxrde_rdqma:0 active in master mode on mqhavm24
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: determine_op_status: Operation monitor found resource p_drbd_drgxrde_rdqmb:0 active in master mode on mqhavm24
Aug 11 12:33:36 [13305] mqhavm24 cib: info: crm_get_peer: Node 2 is now known as mqhavm37
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: determine_op_status: Operation monitor found resource p_drbd_dr_drgxrde_rdqmb:0 active in master mode on mqhavm24
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: determine_op_status: Operation monitor found resource p_drbd_drgxrde_rdqma:0 active on mqhavm37
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: determine_op_status: Operation monitor found resource p_drbd_drgxrde_rdqmb:0 active on mqhavm37
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: unpack_node_loop: Node 1 is already processed
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: unpack_node_loop: Node 2 is already processed
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: unpack_node_loop: Node 3 is already processed
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: unpack_node_loop: Node 1 is already processed
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: unpack_node_loop: Node 2 is already processed
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: unpack_node_loop: Node 3 is already processed
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: common_print: drgxrde_rdqma (ocf::ibm:rdqm): Stopped
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: common_print: p_fs_drgxrde_rdqma (ocf::heartbeat:Filesystem): Stopped
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: common_print: p_rdqmx_drgxrde_rdqma (ocf::ibm:rdqmx): Stopped
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: clone_print: Master/Slave Set: ms_drbd_dr_drgxrde_rdqma [p_drbd_dr_drgxrde_rdqma]
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: short_print: Stopped: [ mqhavm24 mqhavm34 mqhavm37 ]
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: clone_print: Master/Slave Set: ms_drbd_drgxrde_rdqma [p_drbd_drgxrde_rdqma]
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: short_print: Stopped: [ mqhavm24 mqhavm34 mqhavm37 ]
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: common_print: drgxrde_rdqmb (ocf::ibm:rdqm): Stopped
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: common_print: p_fs_drgxrde_rdqmb (ocf::heartbeat:Filesystem): Stopped
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: common_print: p_rdqmx_drgxrde_rdqmb (ocf::ibm:rdqmx): Stopped
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: clone_print: Master/Slave Set: ms_drbd_dr_drgxrde_rdqmb [p_drbd_dr_drgxrde_rdqmb]
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: short_print: Stopped: [ mqhavm24 mqhavm34 mqhavm37 ]
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: clone_print: Master/Slave Set: ms_drbd_drgxrde_rdqmb [p_drbd_drgxrde_rdqmb]
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: short_print: Stopped: [ mqhavm24 mqhavm34 mqhavm37 ]
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: common_print: p_ip_drgxrde_rdqma (ocf::heartbeat:IPaddr2): Stopped
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: common_print: p_ip_drgxrde_rdqmb (ocf::heartbeat:IPaddr2): Stopped
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: --- 0.172.12 2
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: +++ 0.172.13 (null)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib: @num_updates=13
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib/status/node_state[@id='1']: @crm-debug-origin=peer_update_callback
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=mqhavm37/crmd/294, version=0.172.13)
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__set_instance_roles: ms_drbd_drgxrde_rdqma: Promoted 0 instances of a possible 1 to master
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_merge_weights: p_drbd_dr_drgxrde_rdqma:0: Rolling back optional scores from p_fs_drgxrde_rdqma
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_allocate: Resource p_drbd_dr_drgxrde_rdqma:0 cannot run anywhere
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__set_instance_roles: ms_drbd_dr_drgxrde_rdqma: Promoted 0 instances of a possible 1 to master
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_merge_weights: p_fs_drgxrde_rdqma: Rolling back optional scores from p_rdqmx_drgxrde_rdqma
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_allocate: Resource p_fs_drgxrde_rdqma cannot run anywhere
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_merge_weights: p_rdqmx_drgxrde_rdqma: Rolling back optional scores from p_ip_drgxrde_rdqma
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_allocate: Resource p_rdqmx_drgxrde_rdqma cannot run anywhere
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_merge_weights: p_ip_drgxrde_rdqma: Rolling back optional scores from drgxrde_rdqma
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_allocate: Resource p_ip_drgxrde_rdqma cannot run anywhere
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_allocate: Resource drgxrde_rdqma cannot run anywhere
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=mqhavm37/crmd/295, version=0.172.13)
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__set_instance_roles: ms_drbd_drgxrde_rdqmb: Promoted 0 instances of a possible 1 to master
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_merge_weights: p_drbd_dr_drgxrde_rdqmb:0: Rolling back optional scores from p_fs_drgxrde_rdqmb
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_allocate: Resource p_drbd_dr_drgxrde_rdqmb:0 cannot run anywhere
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__set_instance_roles: ms_drbd_dr_drgxrde_rdqmb: Promoted 0 instances of a possible 1 to master
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_merge_weights: p_fs_drgxrde_rdqmb: Rolling back optional scores from p_rdqmx_drgxrde_rdqmb
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_allocate: Resource p_fs_drgxrde_rdqmb cannot run anywhere
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_merge_weights: p_rdqmx_drgxrde_rdqmb: Rolling back optional scores from p_ip_drgxrde_rdqmb
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_allocate: Resource p_rdqmx_drgxrde_rdqmb cannot run anywhere
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_merge_weights: p_ip_drgxrde_rdqmb: Rolling back optional scores from drgxrde_rdqmb
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_allocate: Resource p_ip_drgxrde_rdqmb cannot run anywhere
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_allocate: Resource drgxrde_rdqmb cannot run anywhere
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: RecurringOp: Start recurring monitor (20s) for p_drbd_drgxrde_rdqma:0 on mqhavm24
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: RecurringOp: Start recurring monitor (20s) for p_drbd_drgxrde_rdqma:1 on mqhavm34
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: RecurringOp: Start recurring monitor (20s) for p_drbd_drgxrde_rdqma:2 on mqhavm37
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: RecurringOp: Start recurring monitor (20s) for p_drbd_drgxrde_rdqma:0 on mqhavm24
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: RecurringOp: Start recurring monitor (20s) for p_drbd_drgxrde_rdqma:1 on mqhavm34
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: RecurringOp: Start recurring monitor (20s) for p_drbd_drgxrde_rdqma:2 on mqhavm37
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: RecurringOp: Start recurring monitor (20s) for p_drbd_drgxrde_rdqmb:0 on mqhavm24
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: RecurringOp: Start recurring monitor (20s) for p_drbd_drgxrde_rdqmb:1 on mqhavm34
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: RecurringOp: Start recurring monitor (20s) for p_drbd_drgxrde_rdqmb:2 on mqhavm37
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: RecurringOp: Start recurring monitor (20s) for p_drbd_drgxrde_rdqmb:0 on mqhavm24
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: RecurringOp: Start recurring monitor (20s) for p_drbd_drgxrde_rdqmb:1 on mqhavm34
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: RecurringOp: Start recurring monitor (20s) for p_drbd_drgxrde_rdqmb:2 on mqhavm37
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: LogActions: Leave drgxrde_rdqma (Stopped)
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: LogActions: Leave p_fs_drgxrde_rdqma (Stopped)
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: LogActions: Leave p_rdqmx_drgxrde_rdqma (Stopped)
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: LogActions: Leave p_drbd_dr_drgxrde_rdqma:0 (Stopped)
Aug 11 12:33:36 [13309] mqhavm24 pengine: notice: LogAction: * Start p_drbd_drgxrde_rdqma:0 ( mqhavm24 )
Aug 11 12:33:36 [13309] mqhavm24 pengine: notice: LogAction: * Start p_drbd_drgxrde_rdqma:1 ( mqhavm34 )
Aug 11 12:33:36 [13309] mqhavm24 pengine: notice: LogAction: * Start p_drbd_drgxrde_rdqma:2 ( mqhavm37 )
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: LogActions: Leave drgxrde_rdqmb (Stopped)
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: LogActions: Leave p_fs_drgxrde_rdqmb (Stopped)
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: LogActions: Leave p_rdqmx_drgxrde_rdqmb (Stopped)
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: LogActions: Leave p_drbd_dr_drgxrde_rdqmb:0 (Stopped)
Aug 11 12:33:36 [13309] mqhavm24 pengine: notice: LogAction: * Start p_drbd_drgxrde_rdqmb:0 ( mqhavm24 )
Aug 11 12:33:36 [13309] mqhavm24 pengine: notice: LogAction: * Start p_drbd_drgxrde_rdqmb:1 ( mqhavm34 )
Aug 11 12:33:36 [13309] mqhavm24 pengine: notice: LogAction: * Start p_drbd_drgxrde_rdqmb:2 ( mqhavm37 )
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: LogActions: Leave p_ip_drgxrde_rdqma (Stopped)
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: LogActions: Leave p_ip_drgxrde_rdqmb (Stopped)
We have a plan: starting stuff everywhere (including on "local node" aka DC aka mqhavm24)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=mqhavm37/crmd/299, version=0.172.13)
Aug 11 12:33:36 [13309] mqhavm24 pengine: notice: process_pe_message: Calculated transition 161, saving inputs in /var/lib/pacemaker/pengine/pe-input-688.bz2
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: --- 0.172.13 2
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: +++ 0.172.14 (null)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib: @num_updates=14
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib/status/node_state[@id='1']: @crm-debug-origin=post_cache_update
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib/status/node_state[@id='2']: @crm-debug-origin=post_cache_update
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib/status/node_state[@id='3']: @crm-debug-origin=post_cache_update
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=mqhavm37/crmd/300, version=0.172.14)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: --- 0.172.14 2
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: +++ 0.172.15 (null)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib: @num_updates=15
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ /cib/status/node_state[@id='2']: <transient_attributes id="2"/>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <instance_attributes id="status-2">
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ <nvpair id="status-2-rdqm-transient-attribute" name="rdqm-transient-attribute" value="1"/>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </instance_attributes>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: ++ </transient_attributes>
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=mqhavm37/attrd/19, version=0.172.15)
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: --- 0.172.15 2
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: +++ 0.172.16 (null)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib: @num_updates=16
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=mqhavm37/attrd/20, version=0.172.16)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: --- 0.172.16 2
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: +++ 0.172.17 (null)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib: @num_updates=17
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=mqhavm37/attrd/21, version=0.172.17)
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: do_te_invoke: Processing graph 161 (ref=pe_calc-dc-1660217616-1277) derived from /var/lib/pacemaker/pengine/pe-input-688.bz2
Aug 11 12:33:36 [13310] mqhavm24 crmd: notice: abort_transition_graph: Transition aborted by transient_attributes.2 'create': Transient attribute change | cib=0.172.15 source=abort_unless_down:356 path=/cib/status/node_state[@id='2'] complete=false
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=mqhavm37/crmd/304, version=0.172.17)
Aug 11 12:33:36 [13310] mqhavm24 crmd: notice: run_graph: Transition 161 (Complete=6, Pending=0, Fired=0, Skipped=6, Incomplete=24, Source=/var/lib/pacemaker/pengine/pe-input-688.bz2): Stopped
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: do_state_transition: State transition S_TRANSITION_ENGINE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=notify_crmd
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: --- 0.172.17 2
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: +++ 0.172.18 (null)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib: @num_updates=18
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_cib_replaced, @join=down
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_cib_replaced
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib/status/node_state[@id='3']: @crm-debug-origin=do_cib_replaced
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=mqhavm37/crmd/305, version=0.172.18)
But now we are "join=down" ourselves :-(
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: --- 0.172.18 2
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: +++ 0.172.19 (null)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib: @num_updates=19
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=mqhavm24/attrd/139, version=0.172.19)
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: attrd_cib_callback: CIB update 139 result for master-p_drbd_dr_drgxrde_rdqma: OK | rc=0
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: attrd_cib_callback: * master-p_drbd_dr_drgxrde_rdqma[mqhavm24]=(null)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: --- 0.172.19 2
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: +++ 0.172.20 (null)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib: @num_updates=20
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=mqhavm24/attrd/140, version=0.172.20)
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: attrd_cib_callback: CIB update 140 result for master-p_drbd_dr_drgxrde_rdqmb: OK | rc=0
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: attrd_cib_callback: * master-p_drbd_dr_drgxrde_rdqmb[mqhavm24]=(null)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=mqhavm24/attrd/141, version=0.172.20)
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: attrd_cib_callback: CIB update 141 result for rdqm-transient-attribute: OK | rc=0
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: attrd_cib_callback: * rdqm-transient-attribute[mqhavm34]=1
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: attrd_cib_callback: * rdqm-transient-attribute[mqhavm24]=1
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: attrd_cib_callback: * rdqm-transient-attribute[mqhavm37]=1
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: --- 0.172.20 2
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: +++ 0.172.21 (null)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib: @num_updates=21
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=mqhavm24/attrd/142, version=0.172.21)
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: attrd_cib_callback: CIB update 142 result for master-p_drbd_drgxrde_rdqma: OK | rc=0
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: attrd_cib_callback: * master-p_drbd_drgxrde_rdqma[mqhavm34]=(null)
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: attrd_cib_callback: * master-p_drbd_drgxrde_rdqma[mqhavm24]=(null)
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: attrd_cib_callback: * master-p_drbd_drgxrde_rdqma[mqhavm37]=(null)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: --- 0.172.21 2
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: +++ 0.172.22 (null)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib: @num_updates=22
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=mqhavm24/attrd/143, version=0.172.22)
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: attrd_cib_callback: CIB update 143 result for master-p_drbd_drgxrde_rdqmb: OK | rc=0
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: attrd_cib_callback: * master-p_drbd_drgxrde_rdqmb[mqhavm34]=(null)
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: attrd_cib_callback: * master-p_drbd_drgxrde_rdqmb[mqhavm37]=(null)
Aug 11 12:33:36 [13308] mqhavm24 attrd: info: attrd_cib_callback: * master-p_drbd_drgxrde_rdqmb[mqhavm24]=(null)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: --- 0.172.22 2
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: +++ 0.172.23 (null)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib: @num_updates=23
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=mqhavm37/attrd/24, version=0.172.23)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: --- 0.172.23 2
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: +++ 0.172.24 (null)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib: @num_updates=24
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=mqhavm37/attrd/25, version=0.172.24)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=mqhavm37/attrd/26, version=0.172.24)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: --- 0.172.24 2
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: +++ 0.172.25 (null)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib: @num_updates=25
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=mqhavm37/attrd/27, version=0.172.25)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: --- 0.172.25 2
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: Diff: +++ 0.172.26 (null)
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_perform_op: + /cib: @num_updates=26
Aug 11 12:33:36 [13305] mqhavm24 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=mqhavm37/attrd/28, version=0.172.26)
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: determine_online_status: Node mqhavm37 is online
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: determine_online_status: Node mqhavm34 is online
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: determine_op_status: Operation monitor found resource p_drbd_drgxrde_rdqma:0 active on mqhavm37
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: determine_op_status: Operation monitor found resource p_drbd_drgxrde_rdqmb:0 active on mqhavm37
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: unpack_node_loop: Node 2 is already processed
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: unpack_node_loop: Node 3 is already processed
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: unpack_node_loop: Node 2 is already processed
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: unpack_node_loop: Node 3 is already processed
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: common_print: drgxrde_rdqma (ocf::ibm:rdqm): Stopped
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: common_print: p_fs_drgxrde_rdqma (ocf::heartbeat:Filesystem): Stopped
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: common_print: p_rdqmx_drgxrde_rdqma (ocf::ibm:rdqmx): Stopped
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: clone_print: Master/Slave Set: ms_drbd_dr_drgxrde_rdqma [p_drbd_dr_drgxrde_rdqma]
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: short_print: Stopped: [ mqhavm24 mqhavm34 mqhavm37 ]
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: clone_print: Master/Slave Set: ms_drbd_drgxrde_rdqma [p_drbd_drgxrde_rdqma]
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: short_print: Stopped: [ mqhavm24 mqhavm34 mqhavm37 ]
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: common_print: drgxrde_rdqmb (ocf::ibm:rdqm): Stopped
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: common_print: p_fs_drgxrde_rdqmb (ocf::heartbeat:Filesystem): Stopped
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: common_print: p_rdqmx_drgxrde_rdqmb (ocf::ibm:rdqmx): Stopped
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: clone_print: Master/Slave Set: ms_drbd_dr_drgxrde_rdqmb [p_drbd_dr_drgxrde_rdqmb]
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: short_print: Stopped: [ mqhavm24 mqhavm34 mqhavm37 ]
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: clone_print: Master/Slave Set: ms_drbd_drgxrde_rdqmb [p_drbd_drgxrde_rdqmb]
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: short_print: Stopped: [ mqhavm24 mqhavm34 mqhavm37 ]
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: common_print: p_ip_drgxrde_rdqma (ocf::heartbeat:IPaddr2): Stopped
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: common_print: p_ip_drgxrde_rdqmb (ocf::heartbeat:IPaddr2): Stopped
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_allocate: Resource p_drbd_drgxrde_rdqma:2 cannot run anywhere
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__set_instance_roles: ms_drbd_drgxrde_rdqma: Promoted 0 instances of a possible 1 to master
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_merge_weights: p_drbd_dr_drgxrde_rdqma:0: Rolling back optional scores from p_fs_drgxrde_rdqma
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_allocate: Resource p_drbd_dr_drgxrde_rdqma:0 cannot run anywhere
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__set_instance_roles: ms_drbd_dr_drgxrde_rdqma: Promoted 0 instances of a possible 1 to master
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_merge_weights: p_fs_drgxrde_rdqma: Rolling back optional scores from p_rdqmx_drgxrde_rdqma
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_allocate: Resource p_fs_drgxrde_rdqma cannot run anywhere
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_merge_weights: p_rdqmx_drgxrde_rdqma: Rolling back optional scores from p_ip_drgxrde_rdqma
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_allocate: Resource p_rdqmx_drgxrde_rdqma cannot run anywhere
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_merge_weights: p_ip_drgxrde_rdqma: Rolling back optional scores from drgxrde_rdqma
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_allocate: Resource p_ip_drgxrde_rdqma cannot run anywhere
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_allocate: Resource drgxrde_rdqma cannot run anywhere
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_allocate: Resource p_drbd_drgxrde_rdqmb:2 cannot run anywhere
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__set_instance_roles: ms_drbd_drgxrde_rdqmb: Promoted 0 instances of a possible 1 to master
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_merge_weights: p_drbd_dr_drgxrde_rdqmb:0: Rolling back optional scores from p_fs_drgxrde_rdqmb
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_allocate: Resource p_drbd_dr_drgxrde_rdqmb:0 cannot run anywhere
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__set_instance_roles: ms_drbd_dr_drgxrde_rdqmb: Promoted 0 instances of a possible 1 to master
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_merge_weights: p_fs_drgxrde_rdqmb: Rolling back optional scores from p_rdqmx_drgxrde_rdqmb
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_allocate: Resource p_fs_drgxrde_rdqmb cannot run anywhere
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_merge_weights: p_rdqmx_drgxrde_rdqmb: Rolling back optional scores from p_ip_drgxrde_rdqmb
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_allocate: Resource p_rdqmx_drgxrde_rdqmb cannot run anywhere
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_merge_weights: p_ip_drgxrde_rdqmb: Rolling back optional scores from drgxrde_rdqmb
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_allocate: Resource p_ip_drgxrde_rdqmb cannot run anywhere
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: pcmk__native_allocate: Resource drgxrde_rdqmb cannot run anywhere
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: RecurringOp: Start recurring monitor (20s) for p_drbd_drgxrde_rdqma:0 on mqhavm34
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: RecurringOp: Start recurring monitor (20s) for p_drbd_drgxrde_rdqma:1 on mqhavm37
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: RecurringOp: Start recurring monitor (20s) for p_drbd_drgxrde_rdqma:0 on mqhavm34
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: RecurringOp: Start recurring monitor (20s) for p_drbd_drgxrde_rdqma:1 on mqhavm37
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: RecurringOp: Start recurring monitor (20s) for p_drbd_drgxrde_rdqmb:0 on mqhavm34
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: RecurringOp: Start recurring monitor (20s) for p_drbd_drgxrde_rdqmb:1 on mqhavm37
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: RecurringOp: Start recurring monitor (20s) for p_drbd_drgxrde_rdqmb:0 on mqhavm34
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: RecurringOp: Start recurring monitor (20s) for p_drbd_drgxrde_rdqmb:1 on mqhavm37
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: LogActions: Leave drgxrde_rdqma (Stopped)
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: LogActions: Leave p_fs_drgxrde_rdqma (Stopped)
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: LogActions: Leave p_rdqmx_drgxrde_rdqma (Stopped)
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: LogActions: Leave p_drbd_dr_drgxrde_rdqma:0 (Stopped)
Aug 11 12:33:36 [13309] mqhavm24 pengine: notice: LogAction: * Start p_drbd_drgxrde_rdqma:0 ( mqhavm34 )
Aug 11 12:33:36 [13309] mqhavm24 pengine: notice: LogAction: * Start p_drbd_drgxrde_rdqma:1 ( mqhavm37 )
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: LogActions: Leave p_drbd_drgxrde_rdqma:2 (Stopped)
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: LogActions: Leave drgxrde_rdqmb (Stopped)
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: LogActions: Leave p_fs_drgxrde_rdqmb (Stopped)
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: LogActions: Leave p_rdqmx_drgxrde_rdqmb (Stopped)
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: LogActions: Leave p_drbd_dr_drgxrde_rdqmb:0 (Stopped)
Aug 11 12:33:36 [13309] mqhavm24 pengine: notice: LogAction: * Start p_drbd_drgxrde_rdqmb:0 ( mqhavm34 )
Aug 11 12:33:36 [13309] mqhavm24 pengine: notice: LogAction: * Start p_drbd_drgxrde_rdqmb:1 ( mqhavm37 )
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: LogActions: Leave p_drbd_drgxrde_rdqmb:2 (Stopped)
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: LogActions: Leave p_ip_drgxrde_rdqma (Stopped)
Aug 11 12:33:36 [13309] mqhavm24 pengine: info: LogActions: Leave p_ip_drgxrde_rdqmb (Stopped)
Aug 11 12:33:36 [13309] mqhavm24 pengine: notice: process_pe_message: Calculated transition 162, saving inputs in /var/lib/pacemaker/pengine/pe-input-689.bz2
Which made us change the plan: we don't start anything locally.
Still we continue to manage the other nodes, because we continue to be DC.
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: do_state_transition: State transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE | input=I_PE_SUCCESS cause=C_IPC_MESSAGE origin=handle_response
Aug 11 12:33:36 [13310] mqhavm24 crmd: info: do_te_invoke: Processing graph 162 (ref=pe_calc-dc-1660217616-1278) derived from /var/lib/pacemaker/pengine/pe-input-689.bz2
Aug 11 12:33:36 [13310] mqhavm24 crmd: notice: te_rsc_command: Initiating start operation p_drbd_drgxrde_rdqma_start_0 on mqhavm34 | action 25
Aug 11 12:33:36 [13310] mqhavm24 crmd: notice: te_rsc_command: Initiating start operation p_drbd_drgxrde_rdqma:1_start_0 on mqhavm37 | action 27
Aug 11 12:33:36 [13310] mqhavm24 crmd: notice: te_rsc_command: Initiating start operation p_drbd_drgxrde_rdqmb_start_0 on mqhavm34 | action 77
Aug 11 12:33:36 [13310] mqhavm24 crmd: notice: te_rsc_command: Initiating start operation p_drbd_drgxrde_rdqmb:1_start_0 on mqhavm37 | action 79
Aug 11 12:33:42 [13310] mqhavm24 crmd: notice: te_rsc_command: Initiating notify operation p_drbd_drgxrde_rdqmb_post_notify_start_0 on mqhavm34 | action 133
Aug 11 12:33:42 [13310] mqhavm24 crmd: notice: te_rsc_command: Initiating notify operation p_drbd_drgxrde_rdqmb:1_post_notify_start_0 on mqhavm37 | action 134
Aug 11 12:33:42 [13310] mqhavm24 crmd: notice: te_rsc_command: Initiating notify operation p_drbd_drgxrde_rdqma_post_notify_start_0 on mqhavm34 | action 131
Aug 11 12:33:42 [13310] mqhavm24 crmd: notice: te_rsc_command: Initiating notify operation p_drbd_drgxrde_rdqma:1_post_notify_start_0 on mqhavm37 | action 132
Aug 11 12:33:42 [13310] mqhavm24 crmd: notice: te_rsc_command: Initiating monitor operation p_drbd_drgxrde_rdqma_monitor_20000 on mqhavm34 | action 26
Aug 11 12:33:42 [13310] mqhavm24 crmd: notice: te_rsc_command: Initiating monitor operation p_drbd_drgxrde_rdqma:1_monitor_20000 on mqhavm37 | action 28
Aug 11 12:33:42 [13310] mqhavm24 crmd: notice: te_rsc_command: Initiating monitor operation p_drbd_drgxrde_rdqmb_monitor_20000 on mqhavm34 | action 78
Aug 11 12:33:42 [13310] mqhavm24 crmd: notice: te_rsc_command: Initiating monitor operation p_drbd_drgxrde_rdqmb:1_monitor_20000 on mqhavm37 | action 80
Aug 11 12:33:42 [13310] mqhavm24 crmd: notice: run_graph: Transition 162 (Complete=24, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-689.bz2): Complete
Aug 11 12:33:42 [13310] mqhavm24 crmd: notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE | input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd
Cluster state remains stable, stuff is started on the other two nodes,
just not locally, because "join=down".
More information about the Users
mailing list