<div dir="ltr"><div dir="ltr"> Hi Honza,<div><br></div><div> Thanks for the response.</div><span class="gmail-im" style="color:rgb(80,0,80)"><div><br></div><div>If you increase token timeout even higher <br></div><div>(let's say 12sec) is it still appearing or not? </div></span><div>- I will try this.</div><span class="gmail-im" style="color:rgb(80,0,80)"><div><br></div><div> If you try to run it without RT priority, does it help? </div></span><div>- Can RT priority affect the process scheduling negatively? </div><div><br></div><div>I don't see any irregular IO activity during the time when we got these errors. Also, swap usage and swap IO is not much at all, it's only in KBs. we have vm.swappiness set to 1. So, I don't think swap is causing any issue.</div><div><br></div><div>However, I see slight network activity during the issue times (What I understand is network activity should not affect the CPU jobs as long as CPU load is normal and without any blocking IO).</div><div><br></div><div>I am thinking of debugging in the following way, unless there is option to restart corosync with debugger mode. :</div><div><br></div><div>-> Run a process strace in background on the corosync process and redirect log to a output </div><div>-> Add a frequent cron job to rotate the output log (delete old ones), unless there is a flag file to keep the old log</div><div>-> Add another frequent cron job to check corosync log for the specific token timeout error and add the above mentioned flag file to not delete the strace output. </div><div><br></div><div>Don't know if the above process is safe to run on a production server, without creating much impact on the system resources. Need to check.</div><div><br></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Sep 2, 2019 at 5:50 PM Jan Friesse <<a href="mailto:jfriesse@redhat.com">jfriesse@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">Jeevan,<br>
<br>
Jeevan Patnaik napsal(a):<br>
> Hi,<br>
> <br>
> Also, both are physical machines.<br>
> <br>
> On Fri, Aug 30, 2019 at 7:23 PM Jeevan Patnaik <<a href="mailto:g1patnaik@gmail.com" target="_blank">g1patnaik@gmail.com</a>> wrote:<br>
> <br>
>> Hi,<br>
>><br>
>> We see the following messages almost everyday in our 2 node cluster and<br>
>> resources gets migrated when it happens:<br>
>><br>
>> [16187] node1 corosyncwarning [MAIN ] Corosync main process was not scheduled for 2889.8477 ms (threshold is 800.0000 ms). Consider token timeout increase.<br>
>> [16187] node1 corosyncnotice [TOTEM ] c.<br>
>> [16187] node1 corosyncnotice [TOTEM ] A new membership (<a href="http://192.168.0.1:1268" rel="noreferrer" target="_blank">192.168.0.1:1268</a>) was formed. Members joined: 2 left: 2<br>
>> [16187] node1 corosyncnotice [TOTEM ] Failed to receive the leave message. failed: 2<br>
>><br>
>><br>
>> After setting the token timeout to 6000ms, at least the "Failed to<br>
>> receive the leave message" doesn't appear anymore. But we see corosync<br>
>> timeout errors:<br>
>> [16395] node1 corosyncwarning [MAIN ] Corosync main process was not<br>
>> scheduled for 6660.9043 ms (threshold is 4800.0000 ms). Consider token<br>
>> timeout increase.<br>
>><br>
>> 1. Why is the set timeout not in effect? It's 4800ms instead of 6000ms.<br>
<br>
It is in effect. Threshold for pause detector is set as 0.8 * token timeout.<br>
<br>
>> 2. How to fix this? We have not much load on the nodes, the corosync is<br>
>> already running with RT priority.<br>
<br>
There must be something wrong. If you increase token timeout even higher <br>
(let's say 12sec) is it still appearing or not? If so, isn't the machine <br>
swapping (for example) or waiting for IO? If you try to run it without <br>
RT priority, does it help?<br>
<br>
Regards,<br>
Honza<br>
<br>
<br>
>><br>
>> The following is the details of OS and packages:<br>
>><br>
>> Kernel: 3.10.0-957.el7.x86_64<br>
>> OS: Oracle Linux Server 7.6<br>
>><br>
>> corosync-2.4.3-4.el7.x86_64<br>
>> corosynclib-2.4.3-4.el7.x86_64<br>
>><br>
>> Thanks in advance.<br>
>><br>
>> --<br>
>> Regards,<br>
>> Jeevan.<br>
>> Create your own email signature<br>
>> <<a href="https://www.wisestamp.com/signature-in-email?utm_source=promotion&utm_medium=signature&utm_campaign=create_your_own" rel="noreferrer" target="_blank">https://www.wisestamp.com/signature-in-email?utm_source=promotion&utm_medium=signature&utm_campaign=create_your_own</a>><br>
>><br>
> <br>
> <br>
> <br>
> <br>
> _______________________________________________<br>
> Manage your subscription:<br>
> <a href="https://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.clusterlabs.org/mailman/listinfo/users</a><br>
> <br>
> ClusterLabs home: <a href="https://www.clusterlabs.org/" rel="noreferrer" target="_blank">https://www.clusterlabs.org/</a><br>
> <br>
<br>
</blockquote></div><br clear="all"><div><br></div>Regards,<div dir="ltr" class="gmail_signature"><div href="http://WS_promo" style="width:auto;padding-top:2px;font-size:10px;border-top:1px solid rgb(238,238,238);margin-top:10px;display:table;direction:ltr;line-height:normal;border-spacing:initial">
</div>
</div><div>Jeevan.</div></div>