<div dir="ltr">Hi Ken - Thanks for you response.<div><br><div>We do have seen messages in other cases like</div><div>corosync [MAIN ] Corosync main process was not scheduled for 17314.4746 ms (threshold is 8000.0000 ms). Consider token timeout increase.</div><div>corosync [TOTEM ] A processor failed, forming new configuration.</div></div><div><br></div><div>Is this the indication of a failure due to CPU load issues and will this get resolved if I upgrade to Corosync 2.x series ?</div><div><br></div><div>In any case, for the current scenario, we did not see any scheduling related messages. </div><div><br></div><div>Thanks for your help.</div><div>Prasad</div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Aug 20, 2018 at 7:57 PM, Ken Gaillot <span dir="ltr"><<a href="mailto:kgaillot@redhat.com" target="_blank">kgaillot@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div class="HOEnZb"><div class="h5">On Sun, 2018-08-19 at 17:35 +0530, Prasad Nagaraj wrote:<br>
> Hi:<br>
> <br>
> One of these days, I saw a spurious node loss on my 3-node corosync<br>
> cluster with following logged in the corosync.log of one of the<br>
> nodes.<br>
> <br>
> Aug 18 12:40:25 corosync [pcmk ] notice: pcmk_peer_update:<br>
> Transitional membership event on ring 32: memb=2, new=0, lost=1<br>
> Aug 18 12:40:25 corosync [pcmk ] info: pcmk_peer_update: memb:<br>
> vm02d780875f 67114156<br>
> Aug 18 12:40:25 corosync [pcmk ] info: pcmk_peer_update: memb:<br>
> vmfa2757171f 151000236<br>
> Aug 18 12:40:25 corosync [pcmk ] info: pcmk_peer_update: lost:<br>
> vm728316982d 201331884<br>
> Aug 18 12:40:25 corosync [pcmk ] notice: pcmk_peer_update: Stable<br>
> membership event on ring 32: memb=2, new=0, lost=0<br>
> Aug 18 12:40:25 corosync [pcmk ] info: pcmk_peer_update: MEMB:<br>
> vm02d780875f 67114156<br>
> Aug 18 12:40:25 corosync [pcmk ] info: pcmk_peer_update: MEMB:<br>
> vmfa2757171f 151000236<br>
> Aug 18 12:40:25 corosync [pcmk ] info: ais_mark_unseen_peer_dead:<br>
> Node vm728316982d was not seen in the previous transition<br>
> Aug 18 12:40:25 corosync [pcmk ] info: update_member: Node<br>
> 201331884/vm728316982d is now: lost<br>
> Aug 18 12:40:25 corosync [pcmk ] info: send_member_notification:<br>
> Sending membership update 32 to 3 children<br>
> Aug 18 12:40:25 corosync [TOTEM ] A processor joined or left the<br>
> membership and a new membership was formed.<br>
> Aug 18 12:40:25 [4544] vmfa2757171f stonith-ng: info:<br>
> plugin_handle_membership: Membership 32: quorum retained<br>
> Aug 18 12:40:25 [4544] vmfa2757171f stonith-ng: notice:<br>
> crm_update_peer_state_iter: plugin_handle_membership: Node<br>
> vm728316982d[201331884] - state is now lost (was member)<br>
> Aug 18 12:40:25 [4548] vmfa2757171f crmd: info:<br>
> plugin_handle_membership: Membership 32: quorum retained<br>
> Aug 18 12:40:25 [4548] vmfa2757171f crmd: notice:<br>
> crm_update_peer_state_iter: plugin_handle_membership: Node<br>
> vm728316982d[201331884] - state is now lost (was member)<br>
> Aug 18 12:40:25 [4548] vmfa2757171f crmd: info:<br>
> peer_update_callback: vm728316982d is now lost (was member)<br>
> Aug 18 12:40:25 [4548] vmfa2757171f crmd: warning:<br>
> match_down_event: No match for shutdown action on vm728316982d<br>
> Aug 18 12:40:25 [4548] vmfa2757171f crmd: notice:<br>
> peer_update_callback: Stonith/shutdown of vm728316982d not matched<br>
> Aug 18 12:40:25 [4548] vmfa2757171f crmd: info:<br>
> crm_update_peer_join: peer_update_callback: Node<br>
> vm728316982d[201331884] - join-6 phase 4 -> 0<br>
> Aug 18 12:40:25 [4548] vmfa2757171f crmd: info:<br>
> abort_transition_graph: Transition aborted: Node failure<br>
> (source=peer_update_callback:<wbr>240, 1)<br>
> Aug 18 12:40:25 [4543] vmfa2757171f cib: info:<br>
> plugin_handle_membership: Membership 32: quorum retained<br>
> Aug 18 12:40:25 [4543] vmfa2757171f cib: notice:<br>
> crm_update_peer_state_iter: plugin_handle_membership: Node<br>
> vm728316982d[201331884] - state is now lost (was member)<br>
> Aug 18 12:40:25 [4543] vmfa2757171f cib: notice:<br>
> crm_reap_dead_member: Removing vm728316982d/201331884 from the<br>
> membership list<br>
> Aug 18 12:40:25 [4543] vmfa2757171f cib: notice:<br>
> reap_crm_member: Purged 1 peers with id=201331884 and/or<br>
> uname=vm728316982d from the membership cache<br>
> Aug 18 12:40:25 [4544] vmfa2757171f stonith-ng: notice:<br>
> crm_reap_dead_member: Removing vm728316982d/201331884 from the<br>
> membership list<br>
> Aug 18 12:40:25 [4544] vmfa2757171f stonith-ng: notice:<br>
> reap_crm_member: Purged 1 peers with id=201331884 and/or<br>
> uname=vm728316982d from the membership cache<br>
> <br>
> However, within seconds, the node was able to join back.<br>
> <br>
> Aug 18 12:40:34 corosync [pcmk ] notice: pcmk_peer_update: Stable<br>
> membership event on ring 36: memb=3, new=1, lost=0<br>
> Aug 18 12:40:34 corosync [pcmk ] info: update_member: Node<br>
> 201331884/vm728316982d is now: member<br>
> Aug 18 12:40:34 corosync [pcmk ] info: pcmk_peer_update: NEW: <br>
> vm728316982d 201331884<br>
> <br>
> <br>
> But this was enough time for the cluster to get into split brain kind<br>
> of situation with a resource on the node vm728316982d being stopped<br>
> because of this node loss detection.<br>
> <br>
> Could anyone help whether this could happen due to any transient<br>
> network distortion or so ?<br>
> Are there any configuration settings that can be applied in<br>
> corosync.conf so that cluster is more resilient to such temporary<br>
> distortions.<br>
<br>
</div></div>Your corosync sensitivity of 10-second token timeout and 10<br>
retransimissions is already very lengthy -- likely the node was already<br>
unresponsive for more than 10 seconds before the first message above,<br>
so it was more than 18 seconds before it rejoined.<br>
<br>
It's rarely a good idea to change token_retransmits_before_loss_<wbr>const;<br>
changing token is generally enough to deal with transient network<br>
unreliability. However 18 seconds is a really long time to raise the<br>
token to, and it's uncertain from the information here whether the root<br>
cause was networking or something on the host.<br>
<br>
I notice your configuration is corosync 1 with the pacemaker plugin;<br>
that is a long-deprecated setup, and corosync 3 is about to come out,<br>
so you may want to consider upgrading to at least corosync 2 and a<br>
reasonably recent pacemaker. That would give you some reliability<br>
improvements, including real-time priority scheduling of corosync,<br>
which could have been the issue here if CPU load rather than networking<br>
was the root cause.<br>
<div><div class="h5"><br>
> <br>
> Currently my corosync.conf looks like this :<br>
> <br>
> compatibility: whitetank<br>
> totem {<br>
> version: 2<br>
> secauth: on<br>
> threads: 0<br>
> interface {<br>
> member {<br>
> memberaddr: 172.20.0.4<br>
> }<br>
> member {<br>
> memberaddr: 172.20.0.9<br>
> }<br>
> member {<br>
> memberaddr: 172.20.0.12<br>
> }<br>
> <br>
> bindnetaddr: 172.20.0.12<br>
> <br>
> ringnumber: 0<br>
> mcastport: 5405<br>
> ttl: 1<br>
> }<br>
> transport: udpu<br>
> token: 10000<br>
> token_retransmits_before_loss_<wbr>const: 10<br>
> }<br>
> <br>
> logging {<br>
> fileline: off<br>
> to_stderr: yes<br>
> to_logfile: yes<br>
> to_syslog: no<br>
> logfile: /var/log/cluster/corosync.log<br>
> timestamp: on<br>
> logger_subsys {<br>
> subsys: AMF<br>
> debug: off<br>
> }<br>
> }<br>
> service {<br>
> name: pacemaker<br>
> ver: 1<br>
> }<br>
> amf {<br>
> mode: disabled<br>
> }<br>
> <br>
> Thanks in advance for the help.<br>
> Prasad<br>
> <br>
</div></div>> ______________________________<wbr>_________________<br>
> Users mailing list: <a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a><br>
> <a href="https://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
> <br>
> Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
> Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch" rel="noreferrer" target="_blank">http://www.clusterlabs.org/<wbr>doc/Cluster_from_Scratch</a>.<br>
> pdf<br>
> Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
<span class="HOEnZb"><font color="#888888">-- <br>
Ken Gaillot <<a href="mailto:kgaillot@redhat.com">kgaillot@redhat.com</a>><br>
______________________________<wbr>_________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a><br>
<a href="https://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/<wbr>doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
</font></span></blockquote></div><br></div></div>