<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body>
<p>Ciao,<br>
<br>
I'm among the people that have to deal with with the in-famous <a href="http://www.beekhof.net/blog/2018/two-node-problems" moz-do-not-send="true">
two nodes problem</a> (<a class="moz-txt-link-freetext" href="http://www.beekhof.net/blog/2018/two-node-problems">http://www.beekhof.net/blog/2018/two-node-problems</a>).</p>
<p>I am not sure if to open a bug for this.. so I'm first off reporting on the list.. in the hope to get fast feedback.<br>
</p>
<p><b>Problem statement</b><br>
<br>
I have a cluster made by <b>two</b> nodes with a DRBD shared partition which some resources (systemd services) have to stick to.
</p>
<p>Software versions<br>
</p>
<ul>
<li><font face="Courier New, Courier, monospace">corosync -v<br>
Corosync Cluster Engine, version '2.4.5'<br>
Copyright (c) 2006-2009 Red Hat, Inc.</font></li><li><font face="Courier New, Courier, monospace">pacemakerd --version<br>
Pacemaker 1.1.21-4.el7</font></li><li><font face="Courier New, Courier, monospace">drbdadm --version<br>
DRBDADM_BUILDTAG=GIT-hash:\ fb98589a8e76783d2c56155c645dbaf02ac7ece7\ build\ by\ mockbuild@\,\ 2020-04-05\ 03:21:05<br>
DRBDADM_API_VERSION=2<br>
DRBD_KERNEL_VERSION_CODE=0x090010<br>
DRBD_KERNEL_VERSION=9.0.16<br>
DRBDADM_VERSION_CODE=0x090c02<br>
DRBDADM_VERSION=9.12.2</font></li></ul>
<p>corosync.conf nodes:</p>
<p><font face="Courier New, Courier, monospace">nodelist {<br>
node {<br>
ring0_addr: 10.1.3.1<br>
nodeid: 1<br>
}<br>
node {<br>
ring0_addr: 10.1.3.2<br>
nodeid: 2<br>
}<br>
}<br>
quorum {<br>
provider: corosync_votequorum<br>
two_node: 1<br>
}</font><br>
</p>
<p>drbd nodes config:</p>
<p>resource myresource {<br>
<br>
volume 0 {<br>
device /dev/drbd0;<br>
disk /dev/mapper/vg0-res--etc;<br>
meta-disk internal;<br>
}<br>
<br>
on 123z555666y0 {<br>
node-id 0;<br>
address 10.1.3.1:7789;<br>
}<br>
<br>
on 123z555666y1 {<br>
node-id 1;<br>
address 10.1.3.2:7789;<br>
}<br>
<br>
connection {<br>
host 123z555666y0;<br>
host 123z555666y1;<br>
}<br>
<br>
handlers {<br>
before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh";<br>
after-resync-target "/usr/lib/drbd/unsnapshot-resync-target-lvm.sh";<br>
}<br>
<br>
}<br>
</p>
<p><b>I need to reconfigure the hostname of both the nodes of the cluster</b>. I've gathered some literature around<br>
</p>
<ul>
<li><a class="moz-txt-link-freetext" href="https://pacemaker.oss.clusterlabs.narkive.com/csHZkR5R/change-hostname">https://pacemaker.oss.clusterlabs.narkive.com/csHZkR5R/change-hostname</a></li><li><a class="moz-txt-link-freetext" href="https://clusterlabs.org/pacemaker/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/s-node-name.html">https://clusterlabs.org/pacemaker/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/s-node-name.html</a><br>
</li><li><a class="moz-txt-link-freetext" href="https://www.suse.com/support/kb/doc/?id=000018878">https://www.suse.com/support/kb/doc/?id=000018878</a> <- DIDN'T WORK</li><li><a class="moz-txt-link-freetext" href="https://bugs.clusterlabs.org/show_bug.cgi?id=5265">https://bugs.clusterlabs.org/show_bug.cgi?id=5265</a> <- DIDN'T WORK<br>
</li></ul>
<p>but have not yet found a way to address this (unless with simultaneous reboot of both nodes).</p>
<p>The procedure:</p>
<ul>
<li>Update the hostname on both Master and Slave nodes<br>
</li><ul>
<li>update<font face="Courier New, Courier, monospace"> /etc/hostname</font></li><li>update <font face="Courier New, Courier, monospace">/etc/hosts</font></li><li>update system with <font face="Courier New, Courier,
monospace">
hostname -F /etc/hostname</font></li></ul>
<li>Reconfigure drbd on Master and Slave nodes<br>
</li><ul>
<li>modify <font face="Courier New, Courier, monospace">drbd.01.conf</font> (attached) to reflect new hostname</li><li>invoke <font face="Courier New, Courier, monospace">drbdadm adjust all</font></li></ul>
<li><font face="Courier New, Courier, monospace">Update pacemaker config on Master node only<br>
</font></li><ul>
<li><font face="Courier New, Courier, monospace">crm configure property maintenance-mode=true</font></li><li><font face="Courier New, Courier, monospace">crm configure delete --force 1</font></li><li><font face="Courier New, Courier, monospace"><font face="Courier New, Courier, monospace">crm configure delete --force 2</font></font></li><li><font face="Courier New, Courier, monospace"><font face="Courier New, Courier, monospace">crm configure xml ' <node id="1" uname="newhostname0"><br>
<instance_attributes id="node-1"><br>
<nvpair id="node-1-standby" name="standby" value="off"/><br>
</instance_attributes><br>
</node>'</font></font></li><li><font face="Courier New, Courier, monospace"><font face="Courier New, Courier, monospace"><font face="Courier
New, Courier, monospace"><font face="Courier New,
Courier, monospace">crm configure xml ' <node id="2" uname="newhostname1"><br>
<instance_attributes id="node-2"><br>
<nvpair id="node-2-standby" name="standby" value="off"/><br>
</instance_attributes><br>
</node>'</font></font></font></font></li><li><font face="Courier New, Courier, monospace"><font face="Courier New, Courier, monospace"><font face="Courier
New, Courier, monospace"><font face="Courier New,
Courier, monospace">crm resource reprobe</font></font></font></font></li><li><font face="Courier New, Courier, monospace"><font face="Courier New, Courier, monospace"><font face="Courier
New, Courier, monospace"><font face="Courier New,
Courier, monospace">crm configure refresh</font></font></font></font></li><li><font face="Courier New, Courier, monospace"><font face="Courier New, Courier, monospace"><font face="Courier
New, Courier, monospace"><font face="Courier New,
Courier, monospace"><font face="Courier New, Courier,
monospace">crm
configure property maintenance-mode=false</font></font></font></font></font></li></ul>
</ul>
<p>Let's say for example that I migrate the hostnames like this<br>
</p>
<p><font face="Courier New, Courier, monospace">hostname10 -> hostname20<br>
hostname11 -> hostname21</font><br>
</p>
<p>After the above procedure is concluded the cluster is correctly reconfigured and when I check with
<font face="Courier New,
Courier, monospace">crm_mon</font> or <font face="Courier New,
Courier, monospace">
crm status</font> or <font face="Courier
New, Courier, monospace">crm configure show xml</font> or even by inspecting the<font face="Courier New, Courier, monospace"> cib.xml
</font>I find the proper new hostnames fetched by pacemaker/corosync (hostname20 and hostname21).<br>
</p>
<p>The <a moz-do-not-send="true" href="https://clusterlabs.org/pacemaker/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/s-node-name.html">
documentation</a> reports that pacemaker node name is taken from</p>
<ol>
<li>corosync.conf nodelist-><span class="strong strong"><strong>ring0_addr </strong>
if not an ip address: NOT MY CASE => skip</span></li><li><span class="strong strong">corosync.conf nodelist-><strong>name</strong><span class="strong strong"><strong>
</strong>if available: NOT MY CASE => skip</span></span></li><li><span class="strong strong"><span class="strong strong">uname -n [SHOULD BE IN HERE]<br>
</span></span></li></ol>
<p>Apparently case number 3 does not apply:<br>
</p>
<p><font face="Courier New, Courier, monospace">[root@hostname20 ~]# crm_node -n<br>
hostname10<br>
[root@hostname20 ~]# uname -n<br>
hostname20</font><br>
</p>
<p>This becomes evident as soon as I reboot/poweroff one of the two nodes: crm_mon which after the reconfiguration was correctly showing</p>
<p>Online: [ <font face="Courier New, Courier, monospace">hostname21</font> <font face="Courier New, Courier, monospace">
hostname20</font> ]</p>
<p>"rolls back" the configuration without any notice and starts showing the old one</p>
<p>Online: [ <font face="Courier New, Courier, monospace">hostname10</font> ]<br>
OFFLINE: [ <font face="Courier New, Courier, monospace">hostname11</font> ]</p>
<p>Do you have any idea of where on heath pacemaker is recovering the old hostnames ?
<br>
<br>
I've even checked the code and see that there are cmaps involved so I suspect there's some caching issues involved in this.</p>
<p>It looks like it is retaining the old hostnames in memory and when something .. "fails" it restores them.<br>
</p>
<p>Besides don't blame me for this use case (reconfigure hostnames in a two-nodes cluster), as I didn't make it up. I just carry the pain.</p>
<p>R<br>
</p>
<p><br>
</p>
<p><br>
<br>
</p>
<!-- START FIRMA ITA -->
<hr>
<table style="font-family: Verdana, sans-serif; font-size:12px; color:#000;" border="0" cellspacing="0" cellpadding="0">
<tbody>
<tr>
<td style="border-right:3px solid #952a2e; padding:5px 10px 5px 26px; text-align:right; line-height:18px;">
<span style="color:#952a2e; font-weight:bold;">Riccardo Manfrin</span><br>
R&D DEPARTMENT<br>
<a href="https://www.athonet.com/" target="_blank" onMouseOver="this.style.color='#000'" onMouseOut="this.style.color='#000'" style="color:#000; text-decoration:none;">Web</a> |
<a href="https://www.linkedin.com/company/athonet/" target="_blank" onMouseOver="this.style.color='#000'" onMouseOut="this.style.color='#000'" style="color:#000; text-decoration:none;">
LinkedIn</a></td>
<td style="padding:5px 10px; text-align:left; line-height:18px;"><span style="color:#952a2e;">t</span> +39 (0)444 750045<br>
<span style="color:#952a2e;">e</span> <a href="mailto:riccardo.manfrin@athonet.com" onMouseOver="this.style.color='#000'" onMouseOut="this.style.color='#000'" style="color:#000; text-decoration:none;">
riccardo.manfrin@athonet.com</a></td>
</tr>
</tbody>
</table>
<table style="font-family: Verdana, sans-serif; font-size:12px; color:#000;" border="0" cellspacing="0" cellpadding="0">
<tbody>
<tr>
<td style="padding:14px 0 0 0;"><a href="https://www.athonet.com/" target="_blank" style="color:#000; text-decoration:none;"><img src="https://www.athonet.com/signature/logo_athonet.png"></a></td>
</tr>
<tr>
<td style="padding:15px 0 0 0;">ATHONET | Via Cà del Luogo, 6/8 - 36050 Bolzano Vicentino (VI) Italy
</td>
</tr>
</tbody>
</table>
<table style="font-family: Verdana, sans-serif; font-size:12px;" border="0" cellspacing="0" cellpadding="0" width="800">
<tbody>
<tr>
<td style="color:#868686; font-size:11px; line-height:14px; padding-top: 25px;">This email and any attachments are confidential and intended solely for the use of the intended recipient. If you are not the named addressee, please be aware that you shall not
distribute, copy, use or disclose this email. If you have received this email by error, please notify us immediately and delete this email from your system. Email transmission cannot be guaranteed to be secured or error-free or not to contain viruses. Athonet
S.r.l. processes any personal data exchanged in email correspondence in accordance with EU Reg. 679/2016 (GDPR) - you may find here the privacy policy with information on such processing and your rights. Any views or opinions presented in this email are solely
those of the sender and do not necessarily represent those of Athonet S.r.l.</td>
</tr>
</tbody>
</table>
<!-- END FIRMA ITA -->
</body>
</html>