<div dir="ltr"><div>Hello,</div><div><br></div><div>thanks for your help.</div><div><br></div>I don't see main process was not scheduled for at all in the log, just:<div><div>grep -i main corosync.log<br></div><div>Apr 20 02:13:30 corosync [MAIN ] Completed service synchronization, ready to provide service.</div><div>Apr 20 02:21:23 corosync [MAIN ] Completed service synchronization, ready to provide service.</div><div>Apr 20 04:18:40 corosync [MAIN ] Completed service synchronization, ready to provide service.</div><div>Apr 20 04:18:40 corosync [MAIN ] Completed service synchronization, ready to provide service.</div><div>Apr 20 15:38:42 corosync [MAIN ] Completed service synchronization, ready to provide service.</div><div>Apr 20 15:38:42 corosync [MAIN ] Completed service synchronization, ready to provide service.</div><div>Apr 20 15:57:36 corosync [MAIN ] Completed service synchronization, ready to provide service.</div><div>Apr 20 15:57:38 corosync [MAIN ] Completed service synchronization, ready to provide service.</div><div>Apr 20 16:20:07 corosync [MAIN ] Completed service synchronization, ready to provide service.</div><div>Apr 20 16:20:11 corosync [MAIN ] Completed service synchronization, ready to provide service.</div></div><div><div><br></div><div>grep -i schedul corosync.log</div><div>Apr 14 02:14:31 <a href="http://host2.example.com">host2.example.com</a> pengine: [16959]: info: stage6: Scheduling Node vif5_7 for shutdown</div><div>Apr 15 02:14:26 <a href="http://host2.example.com">host2.example.com</a> pengine: [16959]: info: stage6: Scheduling Node vif5_7 for shutdown</div><div>Apr 16 02:14:45 <a href="http://host2.example.com">host2.example.com</a> pengine: [16959]: info: stage6: Scheduling Node vif5_7 for shutdown</div><div>Apr 17 02:14:33 <a href="http://host2.example.com">host2.example.com</a> pengine: [16959]: info: stage6: Scheduling Node vif5_7 for shutdown</div><div>Apr 18 02:14:34 <a href="http://host2.example.com">host2.example.com</a> pengine: [16959]: info: stage6: Scheduling Node vif5_7 for shutdown</div><div>Apr 19 02:15:11 <a href="http://host2.example.com">host2.example.com</a> pengine: [16959]: info: stage6: Scheduling Node vif5_7 for shutdown</div><div>Apr 20 02:13:29 <a href="http://host2.example.com">host2.example.com</a> pengine: [16959]: info: stage6: Scheduling Node vif5_7 for shutdown</div><div>Apr 21 02:12:14 <a href="http://host2.example.com">host2.example.com</a> pengine: [16959]: info: stage6: Scheduling Node vif5_7 for shutdown</div><div>Apr 22 02:12:13 <a href="http://host2.example.com">host2.example.com</a> pengine: [16959]: info: stage6: Scheduling Node vif5_7 for shutdown</div><div>Apr 23 02:10:52 <a href="http://host2.example.com">host2.example.com</a> pengine: [16959]: info: stage6: Scheduling Node vif5_7 for shutdown</div><div>Apr 24 02:10:18 <a href="http://host2.example.com">host2.example.com</a> pengine: [16959]: info: stage6: Scheduling Node vif5_7 for shutdown</div><div>Apr 25 02:10:35 <a href="http://host2.example.com">host2.example.com</a> pengine: [16959]: info: stage6: Scheduling Node vif5_7 for shutdown</div><div>Apr 26 02:10:35 <a href="http://host2.example.com">host2.example.com</a> pengine: [16959]: info: stage6: Scheduling Node vif5_7 for shutdown</div><div>Apr 27 02:09:36 <a href="http://host2.example.com">host2.example.com</a> pengine: [16959]: info: stage6: Scheduling Node vif5_7 for shutdown</div><div>Apr 28 02:10:37 <a href="http://host2.example.com">host2.example.com</a> pengine: [16959]: info: stage6: Scheduling Node vif5_7 for shutdown</div><div>Apr 29 02:10:40 <a href="http://host2.example.com">host2.example.com</a> pengine: [16959]: info: stage6: Scheduling Node vif5_7 for shutdown</div><div>Apr 30 02:09:30 <a href="http://host2.example.com">host2.example.com</a> pengine: [16959]: info: stage6: Scheduling Node vif5_7 for shutdown</div></div><div><br></div><div>The node vif5_7 is rebooted after backup at ~ 2 o'clock.</div><div><br></div><div><br></div><div>Firewall is ok : accepting all traffic on eth0, without any rate limitation...</div><div><br></div><div>Best regards,</div><div><br></div><div><br></div></div><div class="gmail_extra"><br clear="all"><div><div class="gmail_signature"><div dir="ltr"><table border="0" cellspacing="0" cellpadding="0" style="font-family:'Times New Roman';width:280px"><tbody><tr><td><span style="height:10px"></span><span style="color:rgb(239,124,0);font-family:'Trebuchet MS',Arial,Helvetica,sans-serif;font-size:15px;line-height:20px">Philippe CARBONNIER </span><br><span style="color:rgb(99,99,98);font-family:'Trebuchet MS',Arial,Helvetica,sans-serif;font-size:11px;line-height:15px">Pôle Recherche et Développement </span><br><span style="color:rgb(99,99,98);font-family:'Trebuchet MS',Arial,Helvetica,sans-serif;font-size:11px;line-height:15px">Tél. +33 (0)2 51 89 12 58</span></td></tr></tbody></table><table border="0" style="font-family:'Times New Roman'"><tbody><tr><td><table border="0" cellspacing="0" cellpadding="0" style="font-family:'Times New Roman';width:300px;display:block"><tbody><tr><td style="color:rgb(0,0,0);font-family:'Trebuchet MS',Arial,Helvetica,sans-serif;font-size:11px;line-height:16px;font-weight:bold;width:220px;vertical-align:top"><br></td><td style="width:80px"><br></td></tr></tbody></table><table border="0" cellspacing="0" cellpadding="0" style="font-family:'Times New Roman';width:300px;height:45px;display:block"><tbody></tbody></table></td><td><br></td><td><br></td></tr></tbody></table></div></div></div>
<br><div class="gmail_quote">2015-04-30 8:30 GMT+02:00 Jan Friesse <span dir="ltr"><<a href="mailto:jfriesse@redhat.com" target="_blank">jfriesse@redhat.com</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Philippe,<br>
<br>
Philippe Carbonnier napsal(a):<span class=""><br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Thanks for your answers.<br>
Token value was previoulsly 5000, but I already increased it to 10000,<br>
without any change. So 10 secondes before TOTEM fire the "A processor<br>
failed, forming new configuration" message, but in the log we see that in<br>
the same second the other node reappeared !<br>
</blockquote>
<br></span>
That's weird<span class=""><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Should I use an higher token value ?<br>
</blockquote>
<br></span>
I don't think so. I mean, if both nodes are running on same ESX, it shouldn't be needed.<br>
<br>
- Is corosync scheduled regularly (you would see message "Corosync main process was not scheduled for ... sec" in logs if not)?<br>
- Is firewall correctly configured?<br>
- Isn't there some kind of rate limiting for packets?<br>
<br>
Regards,<br>
Honza<div class="HOEnZb"><div class="h5"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Best regards,<br>
<br>
2015-04-29 14:17 GMT+02:00 Ulrich Windl <<a href="mailto:Ulrich.Windl@rz.uni-regensburg.de" target="_blank">Ulrich.Windl@rz.uni-regensburg.de</a>>:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Jan Friesse <<a href="mailto:jfriesse@redhat.com" target="_blank">jfriesse@redhat.com</a>> schrieb am 29.04.2015 um 13:10 in<br>
</blockquote></blockquote></blockquote>
Nachricht<br>
<<a href="mailto:5540BC0B.50409@redhat.com" target="_blank">5540BC0B.50409@redhat.com</a>>:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Philippe,<br>
<br>
Philippe Carbonnier napsal(a):<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hello,<br>
just for the guys who doesn't want to read all the logs, I put my<br>
</blockquote></blockquote>
question<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
on top (and at the end) of the post :<br>
Is there a timer that I can raise to try to give more time to each<br>
</blockquote></blockquote>
nodes to<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
see each other BEFORE TOTEM fire the "A processor failed, forming new<br>
configuration", because the 2 nodes are really up and running.<br>
</blockquote>
<br>
There are many timers, but basically almost everything depends on token<br>
timeout, so just set "token" to higher value.<br>
</blockquote>
<br>
Please correct me if I'm wrong: A token timeout is oly triggered when<br>
1) The token is lost in the network (i.e. a packet is lost and not<br>
retransmitted in time)<br>
2) The token is lost on a node (e.g. it crashes while it has the token)<br>
3) The host or the network don't respond in time (the token is not lost,<br>
but late)<br>
4) There's a major bug in the TOTEM protocol (its implementation)<br>
<br>
I really wonder whether the resaon for frequent token timeouts is 1);<br>
usually it's not 2) either. For me 3) is hard to believe also. And nobody<br>
admits it's 4).<br>
<br>
So everybody says it's 3) and suggests to increase the timeout.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
The 2 linux servers (vif5_7 and <a href="http://host2.example.com" target="_blank">host2.example.com</a>) are 2 VM on the same<br>
VMWare ESX server. May be the network is 'not working' the way corosync<br>
wants ?<br>
</blockquote></blockquote>
<br>
OK, for virtual hosts I might add:<br>
5) The virtual time is not flowing steadily, i.e. the number of usable CPU<br>
cycles per walltime unit is highly variable.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Yep. But first give a chance to token timeout increase.<br>
</blockquote>
<br>
I agree that for 5) a longer token timeout might be a workaround, but<br>
finding the root cause may be worth the time being spent doing so.<br>
<br>
<br>
Regards,<br>
Ulrich<br>
<br>
<br>
<br>
_______________________________________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
<a href="http://clusterlabs.org/mailman/listinfo/users" target="_blank">http://clusterlabs.org/mailman/listinfo/users</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a><br>
<br>
</blockquote>
<br>
<br>
<br>
_______________________________________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
<a href="http://clusterlabs.org/mailman/listinfo/users" target="_blank">http://clusterlabs.org/mailman/listinfo/users</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a><br>
<br>
</blockquote>
<br>
<br>
_______________________________________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
<a href="http://clusterlabs.org/mailman/listinfo/users" target="_blank">http://clusterlabs.org/mailman/listinfo/users</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a><br>
</div></div></blockquote></div><br></div>
<br>
<table border="0" cellspacing="0" cellpadding="0" style="color:rgb(34,34,34);font-size:13px;background-color:rgb(255,255,255);font-family:Times;width:300px;display:block"><tbody><tr><td colspan="3" style="font-family:arial,sans-serif;margin:0px"><a href="http://www.vif.fr/" style="color:rgb(17,85,204);float:left" target="_blank"><img src="https://ci3.googleusercontent.com/proxy/jNw3kk6adk01Es5Sohxe1dLczyeO-oDK-x1ilyy9UZKzBvvYS6tLJdWQccesL76IUbn7nqjdykptLsCu6Q=s0-d-e1-ft#http://exper-ia.com/images/logo_vif.png" alt="logoVif"></a></td></tr><tr><td width="148" style="font-family:'Trebuchet MS',Arial,Helvetica,sans-serif;margin:0px;vertical-align:top;line-height:16px;font-size:11px;font-weight:bold"><span style="color:rgb(77,77,77)"><font><font><font><font>L'informatique 100% Agro</font></font></font></font></span></td><td width="81" align="right" style="font-family:'Trebuchet MS',Arial,Helvetica,sans-serif;margin:0px;color:rgb(77,77,77);font-size:11px;line-height:16px;font-weight:bold;vertical-align:top"><a href="http://www.vif.fr/" style="color:rgb(77,77,77);text-decoration:none" target="_blank"><font><font><font><font>www.vif.fr</font></font></font></font></a> <br></td><td width="71" rowspan="2" style="font-family:arial,sans-serif;margin:0px"><a href="http://www.youtube.com/user/Agrovif" style="color:rgb(17,85,204);float:right" target="_blank"><img src="https://ci3.googleusercontent.com/proxy/McHI_KgbDIV1VTZNHms4RfYjSyf9dZilloQOXJFt2R_8k5gZP5ezMt8pL7m7dkrNDrVIYdSrY9JJO8CWk8AArr_x2xs=s0-d-e1-ft#http://exper-ia.com/images/ico_YouTubeMail.png" alt="VifYouTube"></a><a href="https://twitter.com/VIF_agro" style="color:rgb(17,85,204);float:right;margin-right:4px" target="_blank"><img src="https://ci5.googleusercontent.com/proxy/TQSsKHGL--n7vnMbP-E37n9cQ0gF0xKtEyEKLVaDA1XyHPqseT8z2Ocw6PnUxjTfHHg7o8H3fzrNxcs7ZmDoKGzURx4=s0-d-e1-ft#http://exper-ia.com/images/ico_twitterMail.png" alt="VifTwitter"></a></td></tr><tr><td colspan="2" align="right" style="font-family:'Trebuchet MS',Arial,Helvetica,sans-serif;margin:0px;color:rgb(239,124,0);font-size:9px;line-height:16px;width:220px;vertical-align:top"><em><font><font><font><font>Suivez l'actualité VIF sur:</font></font></font></font></em></td></tr></tbody></table><a href="http://www.agrovif.com/" target="_blank"><img src="http://www.vif.fr/wp-content/uploads/2014/02/signature-mail-agrovif-2015.jpg"></a>