<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri",sans-serif;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-CA">I have been getting extremely strange behavior from a Corosync/Pacemaker install on OVH Public Cloud servers.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">After hours of Googling, I thought I would try posting here to see if somebody knows what to do.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">I see this in my logs very frequently:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:36 [581] fusion01-2 corosync warning [MAIN ] Corosync main process was not scheduled for 24334.5645 ms (threshold is 2400.0000 ms). Consider token timeout increase.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:36 [581] fusion01-2 corosync notice [TOTEM ] A processor failed, forming new configuration.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">I have increased token time to 10s and this still occurs regularly even though both hosts are always up.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">There are also times when the floating IP script is fired, but corosync does not seem aware. When you run crm status it will show the ip being bound the fusion01-1 when in fact the script fired and moved it to fusion01-2.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Finally, the timing of the logs seems very odd. This is what the end of my corosync log file looks like. Notice the times appear out of order in the logs. I’m ripping my hair out with these issues. Anybody have a clue
what may be going on here?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:47 [21062] fusion01-2 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=fusion01-1/crmd/245, version=0.81.123)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:47 [21062] fusion01-2 cib: info: cib_perform_op: Diff: --- 0.81.123 2<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:47 [21062] fusion01-2 cib: info: cib_perform_op: Diff: +++ 0.81.124 (null)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:47 [21062] fusion01-2 cib: info: cib_perform_op: -- /cib/status/node_state[@id='1']/lrm[@id='1']<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:47 [21062] fusion01-2 cib: info: cib_perform_op: + /cib: @num_updates=124<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:47 [21062] fusion01-2 cib: info: cib_process_request: Completed cib_delete operation for section //node_state[@uname='fusion01-1']/lrm: OK (rc=0, origin=fusion01-1/crmd/246, version=0.81.124)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:47 [21062] fusion01-2 cib: info: cib_perform_op: Diff: --- 0.81.124 2<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:47 [21062] fusion01-2 cib: info: cib_perform_op: Diff: +++ 0.81.125 (null)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:47 [21062] fusion01-2 cib: info: cib_perform_op: + /cib: @num_updates=125<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:47 [21062] fusion01-2 cib: info: cib_perform_op: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_lrm_query_internal<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:47 [21062] fusion01-2 cib: info: cib_perform_op: ++ /cib/status/node_state[@id='1']: <lrm id="1"/><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:47 [21062] fusion01-2 cib: info: cib_perform_op: ++ <lrm_resources><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:47 [21062] fusion01-2 cib: info: cib_perform_op: ++ <lrm_resource id="FloatIP" type="floatip-ocf" class="ocf" provider="ovh"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:47 [21062] fusion01-2 cib: info: cib_perform_op: ++ <lrm_rsc_op id="FloatIP_last_0" operation_key="FloatIP_start_0" operation="start" crm-debug-origin="build_active_RAs"
crm_feature_set="3.0.10" transition-key="5:31:0:08c3f481-ccde-4f75-b1a7-acf8168cd0c1" transition-magic="0:0;5:31:0:08c3f481-ccde-4f75-b1a7-acf8168cd0c1" on_node="fusion01-1" call-id="17" rc-code="0" op-status="0" interval="0" last-run="1485859189" last-rc-change="1485859189"
e<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:47 [21062] fusion01-2 cib: info: cib_perform_op: ++ </lrm_resource><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:47 [21062] fusion01-2 cib: info: cib_perform_op: ++ <lrm_resource id="FS" type="FSSofia" class="lsb"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:47 [21062] fusion01-2 cib: info: cib_perform_op: ++ <lrm_rsc_op id="FS_last_failure_0" operation_key="FS_monitor_0" operation="monitor" crm-debug-origin="build_active_RAs"
crm_feature_set="3.0.10" transition-key="4:1:7:1fe20aa3-b305-4282-99a3-b1f8190d3c2c" transition-magic="0:0;4:1:7:1fe20aa3-b305-4282-99a3-b1f8190d3c2c" on_node="fusion01-1" call-id="9" rc-code="0" op-status="0" interval="0" last-run="1485760597" last-rc-change="1485760597"
ex<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:47 [21062] fusion01-2 cib: info: cib_perform_op: ++ <lrm_rsc_op id="FS_last_0" operation_key="FS_start_0" operation="start" crm-debug-origin="build_active_RAs"
crm_feature_set="3.0.10" transition-key="7:31:0:08c3f481-ccde-4f75-b1a7-acf8168cd0c1" transition-magic="0:0;7:31:0:08c3f481-ccde-4f75-b1a7-acf8168cd0c1" on_node="fusion01-1" call-id="18" rc-code="0" op-status="0" interval="0" last-run="1485859191" last-rc-change="1485859191"
exec-time="<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:47 [21062] fusion01-2 cib: info: cib_perform_op: ++ <lrm_rsc_op id="FS_monitor_1000" operation_key="FS_monitor_1000" operation="monitor" crm-debug-origin="build_active_RAs"
crm_feature_set="3.0.10" transition-key="1:31:0:08c3f481-ccde-4f75-b1a7-acf8168cd0c1" transition-magic="0:0;1:31:0:08c3f481-ccde-4f75-b1a7-acf8168cd0c1" on_node="fusion01-1" call-id="19" rc-code="0" op-status="0" interval="1000" last-rc-change="1485859191"
exec-time="20" qu<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:47 [21062] fusion01-2 cib: info: cib_perform_op: ++ </lrm_resource><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:47 [21062] fusion01-2 cib: info: cib_perform_op: ++ </lrm_resources><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:47 [21062] fusion01-2 cib: info: cib_perform_op: ++ </lrm><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:47 [21062] fusion01-2 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=fusion01-1/crmd/247, version=0.81.125)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:47 [21062] fusion01-2 cib: info: cib_process_request: Completed cib_modify operation for section nodes: OK (rc=0, origin=fusion01-1/crmd/250, version=0.81.125)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:47 [21062] fusion01-2 cib: info: cib_perform_op: Diff: --- 0.81.125 2<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:47 [21062] fusion01-2 cib: info: cib_perform_op: Diff: +++ 0.81.126 (null)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:47 [21062] fusion01-2 cib: info: cib_perform_op: + /cib: @num_updates=126<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:47 [21062] fusion01-2 cib: info: cib_perform_op: + /cib/status/node_state[@id='2']: @crm-debug-origin=do_state_transition<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:47 [21062] fusion01-2 cib: info: cib_perform_op: + /cib/status/node_state[@id='1']: @crm-debug-origin=do_state_transition<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:47 [21062] fusion01-2 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=fusion01-1/crmd/251, version=0.81.126)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:47 [21062] fusion01-2 cib: info: cib_process_request: Completed cib_modify operation for section cib: OK (rc=0, origin=fusion01-1/crmd/252, version=0.81.126)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:47 [21062] fusion01-2 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=fusion01-1/attrd/16, version=0.81.126)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:47 [21062] fusion01-2 cib: info: cib_perform_op: Diff: --- 0.81.126 2<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:47 [21062] fusion01-2 cib: info: cib_perform_op: Diff: +++ 0.81.127 (null)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:47 [21062] fusion01-2 cib: info: cib_perform_op: + /cib: @num_updates=127<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:47 [21062] fusion01-2 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=fusion01-1/attrd/17, version=0.81.127)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:52 [21062] fusion01-2 cib: info: cib_process_ping: Reporting our current digest to fusion01-1: 8f48b8c10ce54828f87b27bea1b50d20 for 0.81.127 (0x7ff1485dadc0 0)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:32:16 [21067] fusion01-2 crmd: info: throttle_send_command: New throttle mode: 0000 (was 0010)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:36 [581] fusion01-2 corosync warning [MAIN ] Corosync main process was not scheduled for 24334.5645 ms (threshold is 2400.0000 ms). Consider token timeout increase.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:36 [581] fusion01-2 corosync notice [TOTEM ] A processor failed, forming new configuration.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:36 [581] fusion01-2 corosync warning [MAIN ] Corosync main process was not scheduled for 9222.6709 ms (threshold is 2400.0000 ms). Consider token timeout increase.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:36 [581] fusion01-2 corosync notice [TOTEM ] A new membership (192.168.128.21:6120) was formed. Members left: 1<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:36 [581] fusion01-2 corosync notice [TOTEM ] Failed to receive the leave message. failed: 1<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:36 [581] fusion01-2 corosync notice [QUORUM] Members[1]: 2<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:36 [581] fusion01-2 corosync notice [MAIN ] Completed service synchronization, ready to provide service.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:46 [581] fusion01-2 corosync notice [TOTEM ] A new membership (192.168.128.20:6128) was formed. Members joined: 1<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:46 [581] fusion01-2 corosync notice [QUORUM] Members[2]: 1 2<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA">Jan 31 10:31:46 [581] fusion01-2 corosync notice [MAIN ] Completed service synchronization, ready to provide service.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-CA"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span style="color:#666666">Cheers,<br>
Corey Moullas<br>
<br>
</span></b><b><span style="color:black">Network Administrator<br>
EMAK</span></b><b><span style="font-size:13.5pt;color:#FF6600">|</span></b><b><span style="color:black">TECH</span></b><span style="color:#1F497D"><o:p></o:p></span></p>
<p class="MsoNormal"><span style="color:#1F497D"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span lang="FR-CA" style="color:#1F497D">T:</span></b><span lang="FR-CA" style="color:#3B3838"> (514)-4000-226 x101<o:p></o:p></span></p>
<p class="MsoNormal"><b><span lang="FR-CA" style="color:#1F497D">E:</span></b><b><span lang="FR-CA" style="color:#3B3838">
</span></b><a href="mailto:cmoullas@emak.tech"><span lang="FR-CA" style="color:#0563C1">cmoullas@emak.tech</span></a><span lang="FR-CA" style="color:#3B3838"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="FR-CA" style="color:#3B3838"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span lang="FR-CA" style="font-size:10.0pt;color:red">IMPORTANT:</span></b><span lang="FR-CA" style="font-size:10.0pt;color:red"> For all support-related requests, please email</span><span lang="FR-CA">
</span><a href="mailto:support@emak.tech"><span lang="FR-CA" style="font-size:10.0pt;color:#0563C1">support@emak.tech</span></a><span lang="FR-CA" style="font-size:10.0pt"> /
<span style="color:red">Pour toute assistance technique, veuillez envoyer un courriel à</span>
</span><a href="mailto:support@emak.tech"><span lang="FR-CA" style="font-size:10.0pt;color:#0563C1">support@emak.tech</span></a><span lang="FR-CA" style="font-size:10.0pt">.</span><span lang="FR-CA"><o:p></o:p></span></p>
<p class="MsoNormal"><span lang="FR-CA"><o:p> </o:p></span></p>
</div>
</body>
</html>