<div dir="ltr"><div dir="ltr"><br></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Fri, Jan 27, 2023 at 12:23 AM Thomas CAS <<a href="mailto:tcas@ikoula.com">tcas@ikoula.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg5436713383812025265">
<div style="overflow-wrap: break-word;" lang="FR">
<div class="m_177347375138369734WordSection1">
<p class="MsoNormal"><span>Hello Reid,<u></u><u></u></span></p>
<p class="MsoNormal"><span><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Thank you so much for your answer and bug report.<u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">If it is a bug, I do not understand why the problem is present in production but not on my lab which is identical?</span></p></div></div></div></blockquote><div><br></div><div>That's a good question. I'm not sure, and I haven't worked much with this resource agent. Does the lab show interesting logs at all from the resource agent during startup? I wonder if it's hitting the same "No MySQL master present" issue but *not* the error that follows. That error is coming from mysql itself, not from the resource agent.</div><div><br></div><div>After the agent hits the "No MySQL master present" issue, it calls the unset_master() function. You can take a look at the agent script (in /usr/lib/ocf/resource.d/heartbeat/mysql by default) to see all the things that unset_master() is doing, and try to determine what's behaving differently in production vs. lab.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg5436713383812025265"><div style="overflow-wrap: break-word;" lang="FR"><div class="m_177347375138369734WordSection1"><p class="MsoNormal"><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">What does the "$OCF_RESKEY_CRM_meta_notify_master_uname" variable do? (which shell command is done through this variable)</span></p></div></div></div></blockquote><div><br></div><div>Pacemaker sets it during a resource's notify operation. It's set to some information that's used only during a notify operation, and it's unset otherwise. You can see some other similar meta variables in the mysql_notify() function. The mysql_notify() function gets called during a notify operation, so those variables might be set. They won't be set when mysql_start() is called during a start operation.<br></div><div> </div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div class="msg5436713383812025265"><div style="overflow-wrap: break-word;" lang="FR"><div class="m_177347375138369734WordSection1"><p class="MsoNormal"><span lang="EN-US"><u></u><u></u></span></p>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal"><span lang="EN-US">Best regards,<u></u><u></u></span></p>
<p class="MsoNormal"><u></u> <u></u></p>
<table style="background:white;border-collapse:collapse" cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td style="padding:0cm">
<p class="MsoNormal" style="line-height:18.75pt"><span style="font-size:13pt;color:rgb(30,25,90)">Thomas Cas</span><span style="color:rgb(60,86,106)"> </span><span style="font-size:13pt;color:rgb(255,120,40)"> | Technicien du support infogérance</span><span style="color:rgb(60,86,106)"><u></u><u></u></span></p>
</td>
</tr>
<tr>
<td style="padding:0cm">
<p class="MsoNormal" style="line-height:18.75pt"><span style="font-size:10pt;color:rgb(30,25,90)" lang="EN-US">PHONE : </span><span style="color:black" lang="EN-US">+33 3 51 25 23 26
</span><span style="font-size:10pt;color:rgb(0,32,96)" lang="EN-US">WEB :</span><span style="color:black" lang="EN-US"> </span><span style="color:black"><a href="https://www.ikoula.com/en" target="_blank"><span style="color:black" lang="EN-US">www.ikoula.com/en</span></a></span><span style="font-size:10pt;color:rgb(17,17,17)" lang="EN-US"><u></u><u></u></span></p>
</td>
</tr>
<tr>
<td style="padding:0cm">
<p class="MsoNormal" style="line-height:18.75pt"><span style="font-size:10pt;color:rgb(17,17,17)">IKOULA Data Center 34 rue Pont Assy - 51100 Reims - FRANCE</span><u></u><u></u></p>
</td>
</tr>
<tr>
<td style="padding:0cm">
<p class="MsoNormal" style="line-height:21pt"><span style="font-size:10pt;color:rgb(51,102,0)" lang="EN-US">Before printing this letter, think about the impact on the environment!</span><span style="color:rgb(60,86,106)" lang="EN-US"><u></u><u></u></span></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<table style="background:white;border-collapse:collapse" cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td style="padding:0cm" valign="bottom">
<p class="MsoNormal"><span style="color:black"><a href="https://www.ikoula.com/en" target="_blank"><span style="color:rgb(66,139,202);text-decoration:none"><img style="width: 2.7916in; height: 0.9479in;" id="m_177347375138369734Image_x0020_7" src="cid:185f25696364cff311" alt="Ikoula" width="268" height="91" border="0"></span></a></span><span style="color:black"> </span><u></u><u></u></p>
</td>
<td style="padding:0cm" valign="bottom">
<p class="MsoNormal" style="margin-bottom:22.5pt"><span style="color:black"><a href="https://twitter.com/ikoula_en" title="Twitter Ikoula" target="_blank"><span style="color:rgb(66,139,202);text-decoration:none"><img style="width: 0.25in; height: 0.25in;" id="m_177347375138369734Image_x0020_8" src="cid:185f25696375b16b22" alt="Twitter" width="24" height="24" border="0"></span></a></span><span style="color:rgb(60,86,106)"> </span><span style="color:black"><a href="https://www.linkedin.com/company/ikoula" title="Linkedin Ikoula" target="_blank"><span style="color:rgb(66,139,202);text-decoration:none"><img style="width: 0.25in; height: 0.25in;" id="m_177347375138369734Image_x0020_9" src="cid:185f2569639692e333" alt="Linkedin" width="24" height="24" border="0"></span></a></span><span style="color:rgb(60,86,106)"> </span><span style="color:black"><a href="http://www.youtube.fr/ikoulanet" title="YouTube Ikoula" target="_blank"><span style="color:rgb(66,139,202);text-decoration:none"><img style="width: 0.25in; height: 0.25in;" id="m_177347375138369734Image_x0020_10" src="cid:185f25696397745b44" alt="Youtube" width="24" height="24" border="0"></span></a></span><span style="color:rgb(60,86,106)"> </span><span style="color:black"><a href="https://pressroom.ikoula.com/" title="Pressroom Ikoula" target="_blank"><span style="color:rgb(66,139,202);text-decoration:none"><img style="width: 0.25in; height: 0.25in;" id="m_177347375138369734Image_x0020_11" src="cid:185f2569639855d355" alt="Pressroom" width="24" height="24" border="0"></span></a></span><span style="color:rgb(60,86,106)"> </span><span style="color:black"><a href="https://blog.ikoula.com/en" title="Blog Ikoula" target="_blank"><span style="color:rgb(66,139,202);text-decoration:none"><img style="width: 0.25in; height: 0.25in;" id="m_177347375138369734Image_x0020_12" src="cid:185f25696399374b66" alt="Blog" width="24" height="24" border="0"></span></a></span><span style="color:rgb(60,86,106)"> <u></u><u></u></span></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<div>
<div style="border-color:rgb(225,225,225) currentcolor currentcolor;border-style:solid none none;border-width:1pt medium medium;padding:3pt 0cm 0cm">
<p class="MsoNormal"><b><span lang="EN-US">De :</span></b><span lang="EN-US"> Reid Wahl <<a href="mailto:nwahl@redhat.com" target="_blank">nwahl@redhat.com</a>>
<br>
<b>Envoyé :</b> jeudi 26 janvier 2023 20:31<br>
<b>À :</b> Cluster Labs - All topics related to open-source clustering welcomed <<a href="mailto:users@clusterlabs.org" target="_blank">users@clusterlabs.org</a>>; Thomas CAS <<a href="mailto:tcas@ikoula.com" target="_blank">tcas@ikoula.com</a>><br>
<b>Objet :</b> Re: [ClusterLabs] Problem with MariaDB cluster<u></u><u></u></span></p>
</div>
</div>
<p class="MsoNormal"><span lang="EN-US"><u></u> <u></u></span></p>
<table style="width:100%" width="100%" cellspacing="0" cellpadding="0" border="0" align="left">
<tbody>
<tr>
<td style="background:rgb(166,166,166);padding:5.25pt 1.5pt"></td>
<td style="width:100%;background:rgb(234,234,234);padding:5.25pt 3.75pt 5.25pt 11.25pt" width="100%">
<div>
<p class="MsoNormal">
<span style="font-size:9pt;font-family:"Segoe UI",sans-serif;color:rgb(33,33,33)">Vous ne recevez pas souvent de courriers de la part de
<a href="mailto:nwahl@redhat.com" target="_blank">nwahl@redhat.com</a>. <a href="https://aka.ms/LearnAboutSenderIdentification" target="_blank">
Découvrez pourquoi cela est important</a><u></u><u></u></span></p>
</div>
</td>
<td style="width:56.25pt;background:rgb(234,234,234);padding:5.25pt 3.75pt" width="75">
</td>
</tr>
</tbody>
</table>
<div>
<div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<p class="MsoNormal"><u></u> <u></u></p>
<div>
<div>
<p class="MsoNormal">On Thu, Jan 26, 2023 at 7:39 AM Thomas CAS <<a href="mailto:tcas@ikoula.com" target="_blank">tcas@ikoula.com</a>> wrote:<u></u><u></u></p>
</div>
<blockquote style="border-color:currentcolor currentcolor currentcolor rgb(204,204,204);border-style:none none none solid;border-width:medium medium medium 1pt;padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US">Hello,</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US">I'm having trouble with a MariaDB cluster (2 nodes, master-slave) on Debian 11.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US">I don't know what to do anymore.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> </span><u></u><u></u></p>
<p class="MsoNormal"><b><span lang="EN-US">Environment:</span></b><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US">Node1:
</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> OS: Debian 11</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US">Kernel: 5.10.0-21-amd64 #1 SMP Debian 5.10.162-1 (2023-01-21)</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US">Versions: resource-agents (4.7.0-1), pacemaker (2.0.5-2), corosync (3.1.2-2), mariadb (10.5.18-0+deb11u1)</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US">Node2:
</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> OS: Debian 11</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US">Kernel: 5.10.0-21-amd64 #1 SMP Debian 5.10.162-1 (2023-01-21)</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US">Versions: resource-agents (4.7.0-1), pacemaker (2.0.5-2), corosync (3.1.2-2), mariadb (10.5.18-0+deb11u1)</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US">crm configure show as attachment.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> </span><u></u><u></u></p>
<p class="MsoNormal"><b><span lang="EN-US">Problem:
</span></b><u></u><u></u></p>
<p class="MsoNormal"><b><span lang="EN-US"> </span></b><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US">When I restart Node2 (which is a slave), it goes up correctly in the cluster:</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US">$ crm status</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US">Cluster Summary:</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> * Stack: corosync</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> * Current DC: Node1 (version 2.0.5-ba59be7122) - partition with quorum</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> * Last updated: Thu Jan 26 12:04:57 2023</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> * Last change: Thu Jan 26 11:39:58 2023 by root via cibadmin on Node2</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> * 2 nodes configured</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> * 3 resource instances configured</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US">Node List:</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> * Online: [ Node1 Node2 ]</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US">Full List of Resources:</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> * VIP (ocf::heartbeat:IPaddr2): Started Node1</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> * Clone Set: MYSQLREPLICATOR [MYSQL] (promotable):</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> * Masters: [ Node1 ]</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> * Slaves: [ Node2 ]</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US">But it does not retrieve the replication information. (SHOW SLAVE STATUS; returns nothing)</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US">In the Node2 logs, I can see this message that explains that replication is not taking place:</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> </span><u></u><u></u></p>
<table style="border-collapse:collapse" cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td style="width:469.8pt;border:1pt solid windowtext;background:black;padding:0cm 5.4pt" width="626" valign="top">
<p class="MsoNormal"><span style="color:white" lang="EN-US">Jan 25 16:29:38 mysql(MYSQL)[22862]: INFO: No MySQL master present - clearing replication state</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:white" lang="EN-US">Jan 25 16:29:39 mysql(MYSQL)[22862]: WARNING: MySQL Slave IO threads currently not running.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:white" lang="EN-US">Jan 25 16:29:39 mysql(MYSQL)[22862]: ERROR: MySQL Slave SQL threads currently not running.</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:white" lang="EN-US">Jan 25 16:29:39 mysql(MYSQL)[22862]: ERROR: See for details</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:white" lang="EN-US">Jan 25 16:29:39 mysql(MYSQL)[22862]: ERROR: ERROR 1200 (HY000) at line 1: Misconfigured slave: MASTER_HOST was not set; Fix in config
file or with CHANGE MASTER TO</span><u></u><u></u></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><span lang="EN-US"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US">From what I see in the following file, Node2 does not seem to find the master name. So it clears its replication information:</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US">/usr/lib/ocf/resource.d/heartbeat/mysql</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> </span><u></u><u></u></p>
<table style="border-collapse:collapse" cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td style="width:469.8pt;border:1pt solid windowtext;background:black;padding:0cm 5.4pt" width="626" valign="top">
<p class="MsoNormal"><span style="color:white" lang="EN-US"> master_host=`echo $OCF_RESKEY_CRM_meta_notify_master_uname|tr -d " "`</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:white" lang="EN-US"> if [ "$master_host" -a "$master_host" != ${NODENAME} ]; then</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:white" lang="EN-US"> ocf_log info "Changing MySQL configuration to replicate from $master_host."</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:white" lang="EN-US"> set_master</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:white" lang="EN-US"> start_slave</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:white" lang="EN-US"> if [ $? -ne 0 ]; then</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:white" lang="EN-US"> ocf_exit_reason "Failed to start slave"</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:white" lang="EN-US"> return $OCF_ERR_GENERIC</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:white" lang="EN-US"> fi</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:white" lang="EN-US"> else</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:white" lang="EN-US"> ocf_log info "No MySQL master present - clearing replication state"</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:white" lang="EN-US"> unset_master</span><u></u><u></u></p>
<p class="MsoNormal"><span style="color:white" lang="EN-US"> fi</span><u></u><u></u></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><span lang="EN-US"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US">As it is a production environment, I performed a bare metal restore of these machines on 2 LAB machines and I have no problem…</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US">In production, there is a lot of writing but the servers are far from being saturated.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"> </span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US">Thank you in advance for all the help you can give me.</span><u></u><u></u></p>
<p class="MsoNormal"><span lang="EN-US"><br>
Best regards,</span><u></u><u></u></p>
</div>
</div>
</div>
</blockquote>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I'm sorry you've encountered this.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I don't understand why the resource agent checks $OCF_RESKEY_CRM_meta_notify_master_uname during the start operation. That value gets set only during a notify operation. That looks like a bug in the resource agent.<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal">I've filed an issue against it here: <a href="https://fra01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2FClusterLabs%2Fresource-agents%2Fissues%2F1839&data=05%7C01%7Ctcas%40ikoula.com%7C6cd84feaf4054c3ddaf908daffd3f66c%7Ccb7a4a4ea7f747cc931f80db4a66f1c7%7C0%7C0%7C638103583110564595%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=jsASsl7V1pLz46T%2FcqrHQLmBdMvaMg6863i7xny4AhY%3D&reserved=0" target="_blank">
https://github.com/ClusterLabs/resource-agents/issues/1839</a><u></u><u></u></p>
</div>
<div>
<p class="MsoNormal"><u></u> <u></u></p>
</div>
<div>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
<blockquote style="border-color:currentcolor currentcolor currentcolor rgb(204,204,204);border-style:none none none solid;border-width:medium medium medium 1pt;padding:0cm 0cm 0cm 6pt;margin-left:4.8pt;margin-right:0cm">
<div>
<div>
<div>
<p class="MsoNormal"><span lang="EN-US"> </span><u></u><u></u></p>
<table style="background:white;border-collapse:collapse" cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td style="padding:0cm">
<p class="MsoNormal" style="line-height:18.75pt">
<span style="font-size:13pt;color:rgb(30,25,90)">Thomas Cas</span><span style="color:rgb(60,86,106)"> </span><span style="font-size:13pt;color:rgb(255,120,40)"> | Technicien du support infogérance</span><u></u><u></u></p>
</td>
</tr>
<tr>
<td style="padding:0cm">
<p class="MsoNormal" style="line-height:18.75pt">
<span style="font-size:10pt;color:rgb(30,25,90)" lang="EN-US">PHONE : </span><span style="color:black" lang="EN-US">+33 3 51 25 23 26
</span><span style="font-size:10pt;color:rgb(0,32,96)" lang="EN-US">WEB :</span><span style="color:black" lang="EN-US"> </span><span style="color:black"><a href="https://fra01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ikoula.com%2Fen&data=05%7C01%7Ctcas%40ikoula.com%7C6cd84feaf4054c3ddaf908daffd3f66c%7Ccb7a4a4ea7f747cc931f80db4a66f1c7%7C0%7C0%7C638103583110720819%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=t1b5WNDzLi7Uu3qsJrOGDln%2F%2Bi%2FJQpGj6zWWBYG1WNo%3D&reserved=0" target="_blank"><span style="color:black" lang="EN-US">www.ikoula.com/en</span></a></span><u></u><u></u></p>
</td>
</tr>
<tr>
<td style="padding:0cm">
<p class="MsoNormal" style="line-height:18.75pt">
<span style="font-size:10pt;color:rgb(17,17,17)">IKOULA Data Center 34 rue Pont Assy - 51100 Reims - FRANCE</span><u></u><u></u></p>
</td>
</tr>
<tr>
<td style="padding:0cm">
<p class="MsoNormal" style="line-height:21pt">
<span style="font-size:10pt;color:rgb(51,102,0)" lang="EN-US">Before printing this letter, think about the impact on the environment!</span><u></u><u></u></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"><span lang="EN-US"> </span><u></u><u></u></p>
<table style="background:white;border-collapse:collapse" cellspacing="0" cellpadding="0" border="0">
<tbody>
<tr>
<td style="padding:0cm" valign="bottom">
<p class="MsoNormal"><span style="color:black"><a href="https://fra01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.ikoula.com%2Fen&data=05%7C01%7Ctcas%40ikoula.com%7C6cd84feaf4054c3ddaf908daffd3f66c%7Ccb7a4a4ea7f747cc931f80db4a66f1c7%7C0%7C0%7C638103583110720819%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=t1b5WNDzLi7Uu3qsJrOGDln%2F%2Bi%2FJQpGj6zWWBYG1WNo%3D&reserved=0" target="_blank"><span style="color:rgb(66,139,202);text-decoration:none"><img style="width: 2.7916in; height: 0.9479in;" id="m_177347375138369734m_-1836375398557362329Image_x0020_1" src="cid:185f25696364cff311" alt="Ikoula" width="268" height="91" border="0"></span></a> </span><u></u><u></u></p>
</td>
<td style="padding:0cm" valign="bottom">
<p class="MsoNormal" style="margin-bottom:22.5pt"><span style="color:black"><a href="https://fra01.safelinks.protection.outlook.com/?url=https%3A%2F%2Ftwitter.com%2Fikoula_en&data=05%7C01%7Ctcas%40ikoula.com%7C6cd84feaf4054c3ddaf908daffd3f66c%7Ccb7a4a4ea7f747cc931f80db4a66f1c7%7C0%7C0%7C638103583110720819%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=2bvhY7LWr5apEQg3S1n8tqXVb8ApBH1Sj6yDH7PLqd4%3D&reserved=0" title="Twitter Ikoula" target="_blank"><span style="color:rgb(66,139,202);text-decoration:none"><img style="width: 0.25in; height: 0.25in;" id="m_177347375138369734m_-1836375398557362329Image_x0020_2" src="cid:185f25696375b16b22" alt="Twitter" width="24" height="24" border="0"></span></a></span><span style="color:rgb(60,86,106)"> </span><span style="color:black"><a href="https://fra01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.linkedin.com%2Fcompany%2Fikoula&data=05%7C01%7Ctcas%40ikoula.com%7C6cd84feaf4054c3ddaf908daffd3f66c%7Ccb7a4a4ea7f747cc931f80db4a66f1c7%7C0%7C0%7C638103583110720819%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=NsZKS4uHglEauz%2BIM7aIUuzgjReMma79mQ0s4w%2B5dM0%3D&reserved=0" title="Linkedin Ikoula" target="_blank"><span style="color:rgb(66,139,202);text-decoration:none"><img style="width: 0.25in; height: 0.25in;" id="m_177347375138369734m_-1836375398557362329Image_x0020_3" src="cid:185f2569639692e333" alt="Linkedin" width="24" height="24" border="0"></span></a></span><span style="color:rgb(60,86,106)"> </span><span style="color:black"><a href="https://fra01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fwww.youtube.fr%2Fikoulanet&data=05%7C01%7Ctcas%40ikoula.com%7C6cd84feaf4054c3ddaf908daffd3f66c%7Ccb7a4a4ea7f747cc931f80db4a66f1c7%7C0%7C0%7C638103583110720819%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=L2eGusCXT%2BRkV8Y23a5LmZAs3MdrANPFHfduHwbosX0%3D&reserved=0" title="YouTube Ikoula" target="_blank"><span style="color:rgb(66,139,202);text-decoration:none"><img style="width: 0.25in; height: 0.25in;" id="m_177347375138369734m_-1836375398557362329Image_x0020_4" src="cid:185f25696397745b44" alt="Youtube" width="24" height="24" border="0"></span></a></span><span style="color:rgb(60,86,106)"> </span><span style="color:black"><a href="https://fra01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpressroom.ikoula.com%2F&data=05%7C01%7Ctcas%40ikoula.com%7C6cd84feaf4054c3ddaf908daffd3f66c%7Ccb7a4a4ea7f747cc931f80db4a66f1c7%7C0%7C0%7C638103583110877041%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=iY3K4OajrW5k5cum6%2F9dnA%2BwJs%2FWlQoAS0IaEScg6X4%3D&reserved=0" title="Pressroom Ikoula" target="_blank"><span style="color:rgb(66,139,202);text-decoration:none"><img style="width: 0.25in; height: 0.25in;" id="m_177347375138369734m_-1836375398557362329Image_x0020_5" src="cid:185f2569639855d355" alt="Pressroom" width="24" height="24" border="0"></span></a></span><span style="color:rgb(60,86,106)"> </span><span style="color:black"><a href="https://fra01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fblog.ikoula.com%2Fen&data=05%7C01%7Ctcas%40ikoula.com%7C6cd84feaf4054c3ddaf908daffd3f66c%7Ccb7a4a4ea7f747cc931f80db4a66f1c7%7C0%7C0%7C638103583110877041%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=xU7ptTaaNo9fld7330tLxkg6GculNta3%2BuXMbkgZUBE%3D&reserved=0" title="Blog Ikoula" target="_blank"><span style="color:rgb(66,139,202);text-decoration:none"><img style="width: 0.25in; height: 0.25in;" id="m_177347375138369734m_-1836375398557362329Image_x0020_6" src="cid:185f25696399374b66" alt="Blog" width="24" height="24" border="0"></span></a></span><span style="color:rgb(60,86,106)"> </span><u></u><u></u></p>
</td>
</tr>
</tbody>
</table>
<p class="MsoNormal"> <u></u><u></u></p>
<p class="MsoNormal"> <u></u><u></u></p>
</div>
</div>
<p class="MsoNormal">_______________________________________________<br>
Manage your subscription:<br>
<a href="https://fra01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.clusterlabs.org%2Fmailman%2Flistinfo%2Fusers&data=05%7C01%7Ctcas%40ikoula.com%7C6cd84feaf4054c3ddaf908daffd3f66c%7Ccb7a4a4ea7f747cc931f80db4a66f1c7%7C0%7C0%7C638103583110877041%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3NBeknQhLMP6hGGEV1ABu%2FLyQw5B2tEqfP0D3uvCrZ0%3D&reserved=0" target="_blank">https://lists.clusterlabs.org/mailman/listinfo/users</a><br>
<br>
ClusterLabs home: <a href="https://fra01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fwww.clusterlabs.org%2F&data=05%7C01%7Ctcas%40ikoula.com%7C6cd84feaf4054c3ddaf908daffd3f66c%7Ccb7a4a4ea7f747cc931f80db4a66f1c7%7C0%7C0%7C638103583110877041%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=qAR2FaOViM5i4GMmd1sttMGHYDhnF6bkcL56DffrNKQ%3D&reserved=0" target="_blank">
https://www.clusterlabs.org/</a><u></u><u></u></p>
</div>
</blockquote>
</div>
<p class="MsoNormal"><br clear="all">
<br>
-- <u></u><u></u></p>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<div>
<p class="MsoNormal" style="margin-bottom:12pt">Regards,<u></u><u></u></p>
</div>
<p class="MsoNormal">Reid Wahl (He/Him)<u></u><u></u></p>
</div>
<div>
<p class="MsoNormal">Senior Software Engineer, Red Hat<u></u><u></u></p>
</div>
<p class="MsoNormal">RHEL High Availability - Pacemaker<u></u><u></u></p>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div>
</div></blockquote></div><br clear="all"><br>-- <br><div dir="ltr" class="gmail_signature"><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div dir="ltr"><div><div>Regards,<br><br></div>Reid Wahl (He/Him)<br></div><div>Senior Software Engineer, Red Hat<br></div>RHEL High Availability - Pacemaker<br></div></div></div></div></div></div></div></div></div></div></div></div></div></div></div>