[Pacemaker] One node thinks everyone is online, but the other node doesn't think so
Andrew Beekhof
andrew at beekhof.net
Sun Mar 29 20:33:47 EDT 2015
> On 11 Mar 2015, at 2:21 am, Dmitry Koterov <dmitry.koterov at gmail.com> wrote:
>
> On Tue, Feb 24, 2015 at 2:07 AM, Andrew Beekhof <andrew at beekhof.net> wrote:
>
> > I have a 3-node cluster where node1 and node2 are running corosync+pacemaker and node3 is running corosync only (for quorum). Corosync 2.3.3, pacemaker 1.1.10. Everything worked fine the first couple of days.
> >
> > Once upon a time I discovered the following situation: node2 thinks that both node1 and node2 are online, but node1 thinks that node2 is down. Could you please say: how could it be? There are no connectivity problems between the nodes at the moment (maybe they were, but why the system hasn't recovered?).
>
> The logs show connectivity problems occurring, so no doubt there.
> As to why it hasn't recovered, first check corosync - if it does not have a consistent view of the world pacemaker has no hope.
> Alternatively, I recall there was a bug that could be preventing this in your version. So if corosync looks fine, perhaps try an upgrade.
>
> Thanks.
> Are you talking about this bug: https://bugs.launchpad.net/ubuntu/+source/libqb/+bug/1341496 ?
>
> I believe I reproduced the problem one more time (it's very unstable), the symptoms were the following:
>
> 1. Once upon a time node2 became down.
> 2. The last message from corosync at node1 was "Quorum lost" (I suspect there was a temporary misconnection with node3).
> 3. Then, in a couple of days, at node3 "service corosync stop" hanged (only killall -9 helps). I tried to run strace during the service is stopping, it shows:
>
> [pid 19449] futex(0x7f580b4c62e0, FUTEX_WAIT_PRIVATE, 0, NULL <unfinished ...>
> [pid 19448] --- SIGQUIT {si_signo=SIGQUIT, si_code=SI_USER, si_pid=28183, si_uid=0} ---
> [pid 19448] write(6, "\3\0\0\0", 4) = 4
> [pid 19448] rt_sigreturn() = 360
> ... <and repeats for 19448 again and again>
>
> where pstree shows:
>
> init,1
> ├─corosync,19448
> │ └─{corosync},19449
>
> 4. As well as at node1: "service corosync stop" hangs at node1 too with same symptoms, only killall -9 helps.
> 5. Restarting corosync & pacemaker at node1 and node2 solved the problem.
>
> Could you please say is it related to the above bug in libqb?
I'm no expert in libqb, but it certainly looks likely.
>
>
>
> > The "crm status" is below. What other logs should I attach for the diagnostics?
> >
> > Also, "service corosync stop" on node1 hangs forever with no additional lines in logs, so I cannot even stop the service. (But after "service corosync stop" on node1 the node node2 starts thinking that node1 is offline, although the command still hangs.)
> >
> >
> > root at node2:~# crm status
> > Current DC: node1 (1760315215) - partition with quorum
> > 2 Nodes configured
> > 6 Resources configured
> > Online: [ node1 node2 ]
> > Master/Slave Set: ms_drbd [drbd]
> > Masters: [ node2 ]
> > Slaves: [ node1 ]
> > Resource Group: server
> > fs (ocf::heartbeat:Filesystem): Started node2
> > postgresql (lsb:postgresql): Started node2
> > bind9 (lsb:bind9): Started node2
> > nginx (lsb:nginx): Started node2
> >
> >
> > root at node1:/var/log# crm status
> > Current DC: node1 (1760315215) - partition with quorum
> > 2 Nodes configured
> > 6 Resources configured
> > Online: [ node1 ]
> > OFFLINE: [ node2 ]
> > Master/Slave Set: ms_drbd [drbd]
> > Masters: [ node1 ]
> > Stopped: [ node2 ]
> > Resource Group: server
> > fs (ocf::heartbeat:Filesystem): Started node1
> > postgresql (lsb:postgresql): Started node1
> > bind9 (lsb:bind9): Started node1
> > nginx (lsb:nginx): Started node1
> > Failed actions:
> > drbd_promote_0 (node=node1, call=634, rc=1, status=Timed Out, last-rc-change=Thu Jan 22 10:30:08 2015, queued=20004ms, exec=0ms): unknown error
> >
> >
> > A part of "crm configure show":
> >
> > property $id="cib-bootstrap-options" \
> > dc-version="1.1.10-42f2063" \
> > cluster-infrastructure="corosync" \
> > stonith-enabled="false" \
> > last-lrm-refresh="1421250983"
> > rsc_defaults $id="rsc-options" \
> > resource-stickiness="100"
> >
> >
> > Also I see in logs on node1 (maybe they're related to the issue, maybe not):
> >
> > Jan 22 10:14:02 node1 pengine[2772]: warning: pe_fence_node: Node node2 is unclean because it is partially and/or un-expectedly down
> > Jan 22 10:14:02 node1 pengine[2772]: warning: determine_online_status: Node node2 is unclean
> > Jan 22 10:14:02 node1 pengine[2772]: warning: stage6: Node node2 is unclean!
> > Jan 22 10:14:02 node1 pengine[2772]: warning: stage6: YOUR RESOURCES ARE NOW LIKELY COMPROMISED
> > Jan 22 10:14:02 node1 pengine[2772]: error: stage6: ENABLE STONITH TO KEEP YOUR RESOURCES SAFE
> >
> >
> > On node2 the logs are:
> >
> > Jan 22 10:13:57 node2 corosync[32761]: [TOTEM ] A new membership (188.166.54.190:6276) was formed. Members left: 1760315215 13071578
> > Jan 22 10:13:57 node2 crmd[311]: notice: peer_update_callback: Our peer on the DC is dead
> > Jan 22 10:13:57 node2 crmd[311]: notice: do_state_transition: State transition S_NOT_DC -> S_ELECTION [ input=I_ELECTION cause=C_CRMD_STATUS_CALLBACK origin=peer_update_callback ]
> > Jan 22 10:13:57 node2 corosync[32761]: [QUORUM] This node is within the non-primary component and will NOT provide any services.
> > Jan 22 10:13:57 node2 corosync[32761]: [QUORUM] Members[1]: 1017525950
> > Jan 22 10:13:57 node2 crmd[311]: notice: pcmk_quorum_notification: Membership 6276: quorum lost (1)
> > Jan 22 10:13:57 node2 crmd[311]: notice: crm_update_peer_state: pcmk_quorum_notification: Node (null)[13071578] - state is now lost (was member)
> > Jan 22 10:13:57 node2 crmd[311]: notice: crm_update_peer_state: pcmk_quorum_notification: Node node1[1760315215] - state is now lost (was member)
> > Jan 22 10:13:57 node2 pacemakerd[302]: notice: pcmk_quorum_notification: Membership 6276: quorum lost (1)
> > Jan 22 10:13:57 node2 pacemakerd[302]: notice: crm_update_peer_state: pcmk_quorum_notification: Node node1[1760315215] - state is now lost (was member)
> > Jan 22 10:13:57 node2 pacemakerd[302]: notice: crm_update_peer_state: pcmk_quorum_notification: Node (null)[13071578] - state is now lost (was member)
> > Jan 22 10:13:57 node2 corosync[32761]: [MAIN ] Completed service synchronization, ready to provide service.
> > Jan 22 10:14:01 node2 corosync[32761]: [TOTEM ] A new membership (104.236.71.79:6288) was formed. Members joined: 1760315215 13071578
> > Jan 22 10:14:02 node2 crmd[311]: error: pcmk_cpg_membership: Node node1[1760315215] appears to be online even though we think it is dead
> > Jan 22 10:14:02 node2 crmd[311]: notice: crm_update_peer_state: pcmk_cpg_membership: Node node1[1760315215] - state is now member (was lost)
> > Jan 22 10:14:03 node2 corosync[32761]: [QUORUM] This node is within the primary component and will provide service.
> > Jan 22 10:14:03 node2 corosync[32761]: [QUORUM] Members[3]: 1760315215 13071578 1017525950
> > Jan 22 10:14:03 node2 crmd[311]: notice: pcmk_quorum_notification: Membership 6288: quorum acquired (3)
> > Jan 22 10:14:03 node2 pacemakerd[302]: notice: pcmk_quorum_notification: Membership 6288: quorum acquired (3)
> > Jan 22 10:14:03 node2 pacemakerd[302]: notice: crm_update_peer_state: pcmk_quorum_notification: Node node1[1760315215] - state is now member (was lost)
> > Jan 22 10:14:03 node2 corosync[32761]: [MAIN ] Completed service synchronization, ready to provide service.
> > Jan 22 10:14:03 node2 crmd[311]: notice: corosync_node_name: Unable to get node name for nodeid 13071578
> > Jan 22 10:14:03 node2 crmd[311]: notice: crm_update_peer_state: pcmk_quorum_notification: Node (null)[13071578] - state is now member (was lost)
> > Jan 22 10:14:03 node2 pacemakerd[302]: notice: corosync_node_name: Unable to get node name for nodeid 13071578
> > Jan 22 10:14:03 node2 pacemakerd[302]: notice: crm_update_peer_state: pcmk_quorum_notification: Node (null)[13071578] - state is now member (was lost)
> > Jan 22 10:14:03 node2 crmd[311]: warning: do_log: FSA: Input I_JOIN_OFFER from route_message() received in state S_ELECTION
> > Jan 22 10:14:04 node2 crmd[311]: notice: do_state_transition: State transition S_ELECTION -> S_PENDING [ input=I_PENDING cause=C_FSA_INTERNAL origin=do_election_count_vote ]
> > Jan 22 10:14:05 node2 attrd[310]: notice: attrd_local_callback: Sending full refresh (origin=crmd)
> > Jan 22 10:14:05 node2 attrd[310]: notice: attrd_trigger_update: Sending flush op to all hosts for: master-drbd (10000)
> > Jan 22 10:14:05 node2 attrd[310]: notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)
> > Jan 22 10:14:05 node2 crmd[311]: notice: do_state_transition: State transition S_PENDING -> S_NOT_DC [ input=I_NOT_DC cause=C_HA_MESSAGE origin=do_cl_join_finalize_respond ]
> > Jan 22 10:15:11 node2 corosync[32761]: [TOTEM ] A new membership (104.236.71.79:6296) was formed. Members left: 13071578
> > Jan 22 10:15:14 node2 corosync[32761]: [TOTEM ] A new membership (128.199.116.218:6312) was formed. Members joined: 13071578 left: 1760315215
> > Jan 22 10:15:17 node2 corosync[32761]: [TOTEM ] A new membership (104.236.71.79:6324) was formed. Members joined: 1760315215
> > Jan 22 10:15:19 node2 crmd[311]: notice: peer_update_callback: Our peer on the DC is dead
> > Jan 22 10:15:19 node2 crmd[311]: notice: do_state_transition: State transition S_NOT_DC -> S_ELECTION [ input=I_ELECTION cause=C_CRMD_STATUS_CALLBACK origin=peer_update_callback ]
> > Jan 22 10:15:20 node2 kernel: [690741.179442] block drbd0: peer( Primary -> Secondary )
> > Jan 22 10:15:20 node2 corosync[32761]: [QUORUM] Members[3]: 1760315215 13071578 1017525950
> > Jan 22 10:15:20 node2 corosync[32761]: [MAIN ] Completed service synchronization, ready to provide service.
> >
> > _______________________________________________
> > Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> > http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> >
> > Project Home: http://www.clusterlabs.org
> > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > Bugs: http://bugs.clusterlabs.org
>
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
More information about the Pacemaker
mailing list