<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=utf-8"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:Aptos;}
@font-face
{font-family:Consolas;
panose-1:2 11 6 9 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:12.0pt;
font-family:"Aptos",sans-serif;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
font-size:12.0pt;
font-family:"Aptos",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:11.0pt;
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:2.0cm 42.5pt 2.0cm 3.0cm;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:98377449;
mso-list-type:hybrid;
mso-list-template-ids:1624036722 68747279 68747289 68747291 68747279 68747289 68747291 68747279 68747289 68747291;}
@list l0:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l1
{mso-list-id:187833497;
mso-list-type:hybrid;
mso-list-template-ids:-630695680 -613802892 68747289 68747291 68747279 68747289 68747291 68747279 68747289 68747291;}
@list l1:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
mso-ansi-language:EN-US;}
@list l1:level2
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l1:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level5
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
@list l1:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level8
{mso-level-number-format:alpha-lower;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;}
@list l1:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
text-indent:-9.0pt;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=RU link="#0563C1" vlink="#954F72" style='word-wrap:break-word'><div class=WordSection1><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'>Hi,<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'>Occasionally, I have pacemaker as a base layer of custom clustering solution and I have a script to rebuild the second node from the first one. I can’t share the script itself as is has a lot of solution-dependent references, but I can share the sequence to rebuild the failed node:<o:p></o:p></span></p><ol style='margin-top:0cm' start=1 type=1><li class=MsoListParagraph style='margin-left:0cm;mso-list:l1 level1 lfo1'><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'>Setup the new node with the same IP and hostname<o:p></o:p></span></li><li class=MsoListParagraph style='margin-left:0cm;mso-list:l1 level1 lfo1'><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'>(optional) setup passwordless mutual key-based SSH access. It is not necessary, but make a lot of things easy.<o:p></o:p></span></li><li class=MsoListParagraph style='margin-left:0cm;mso-list:l1 level1 lfo1'><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'>Copy files from survived host to the new one:<o:p></o:p></span></li><ol style='margin-top:0cm' start=1 type=a><li class=MsoListParagraph style='margin-left:0cm;line-height:14.25pt;mso-list:l1 level2 lfo1;background:white'><span style='font-size:10.5pt;font-family:"Courier New";color:black'>/etc/corosync/authkey</span><span style='font-size:10.5pt;font-family:"Courier New"'><o:p></o:p></span></li><li class=MsoListParagraph style='margin-left:0cm;line-height:14.25pt;mso-list:l1 level2 lfo1;background:white'><span style='font-size:10.5pt;font-family:"Courier New";color:black'>/etc/corosync/corosync.conf</span><span style='font-size:10.5pt;font-family:"Courier New"'><o:p></o:p></span></li><li class=MsoListParagraph style='margin-left:0cm;line-height:14.25pt;mso-list:l1 level2 lfo1;background:white'><span lang=EN-US style='font-size:10.5pt;font-family:Consolas;color:black'>/</span><span style='font-size:10.5pt;font-family:"Courier New";color:black'>etc/drbd.d/*.res</span><span lang=EN-US style='font-size:10.5pt;font-family:Consolas'><o:p></o:p></span></li><li class=MsoListParagraph style='margin-left:0cm;line-height:14.25pt;mso-list:l1 level2 lfo1;background:white'><span style='font-size:10.5pt;font-family:"Courier New";color:black'>/etc/pacemaker/authkey</span><span style='font-size:10.5pt;font-family:"Courier New"'><o:p></o:p></span></li></ol><li class=MsoListParagraph style='margin-left:0cm;mso-list:l1 level1 lfo1'><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'>Set <b>hacluster</b> user pass to the same as it was on the survived node.<o:p></o:p></span></li><li class=MsoListParagraph style='margin-left:0cm;mso-list:l1 level1 lfo1'><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'>Re-auth pcs nodes with command<br></span><span style='font-size:10.5pt;font-family:"Courier New"'>pcs host auth <host1_name> <host2_name> -u hacluster -p <ha_cluster_pass><o:p></o:p></span></li><li class=MsoListParagraph style='margin-left:0cm;mso-list:l1 level1 lfo1'><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'>Reboot the restored server<o:p></o:p></span></li><li class=MsoListParagraph style='margin-left:0cm;mso-list:l1 level1 lfo1'><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'>PROFIT!!!<o:p></o:p></span></li></ol><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'>If you use no arbiter (corosync-qnetd) this should be enough for your new cluster node up and running. If you use corosync-qnetd, you need also restore corosync-qdevice nssdb keys for the second host connect the arbiter node:<o:p></o:p></span></p><ol style='margin-top:0cm' start=1 type=1><li class=MsoListParagraph style='margin-left:0cm;mso-list:l0 level1 lfo2'><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'>On old host, extract your arbiter certificate from nssdb on the survived host:<br></span><span style='font-size:10.5pt;font-family:"Courier New"'>certutil -L -d /etc/corosync/qdevice/net/nssdb -n 'QNet CA' -r > /root/qnetd-cert.crt<o:p></o:p></span></li><li class=MsoListParagraph style='margin-left:0cm;mso-list:l0 level1 lfo2'><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'>Copy certificate to the new host, assume the path on the new host is the same<o:p></o:p></span></li><li class=MsoListParagraph style='margin-left:0cm;mso-list:l0 level1 lfo2'><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'>On the new host, Init new nssdb with certificate:<br></span><span style='font-size:10.5pt;font-family:"Courier New"'>corosync-qdevice-net-certutil -i -c /root/qnetd-cert.crt<o:p></o:p></span></li><li class=MsoListParagraph style='margin-left:0cm;mso-list:l0 level1 lfo2'><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'>Copy certificate and key at location </span><span style='font-size:10.5pt;font-family:"Courier New"'>/etc/corosync/qdevice/net/nssdb/qdevice-net-node.p12</span><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'> from old node to new one<o:p></o:p></span></li><li class=MsoListParagraph style='margin-left:0cm;mso-list:l0 level1 lfo2'><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'>On the new node: Import certificate and key:<br></span><span style='font-size:10.5pt;font-family:"Courier New"'>corosync-qdevice-net-certutil -m -c /etc/corosync/qdevice/net/nssdb/qdevice-net-node.p12</span><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'><o:p></o:p></span></li><li class=MsoListParagraph style='margin-left:0cm;mso-list:l0 level1 lfo2'><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'>Enable or restart corosync-qdevice:<br></span><span lang=EN-US style='font-size:10.5pt;font-family:"Courier New"'>systemctl enable –now corosync-qdevice.service<br></span><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'>or<br></span><span lang=EN-US style='font-size:10.5pt;font-family:"Courier New"'>systemctl restart corosync-qdevice.service<o:p></o:p></span></li><li class=MsoListParagraph style='margin-left:0cm;mso-list:l0 level1 lfo2'><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'>Enjoy!<o:p></o:p></span></li></ol><p class=MsoListParagraph><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoListParagraph style='margin-left:0cm'><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'>That’s what practically work for me and included in service scripts of our product, based on Pacemaker.<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:0cm'><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoListParagraph style='margin-left:0cm'><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'>Hope this could help!<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:0cm'><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoListParagraph style='margin-left:0cm'><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'>Sincerely,<o:p></o:p></span></p><p class=MsoListParagraph style='margin-left:0cm'><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoListParagraph style='margin-left:0cm'><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'>Alex<o:p></o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'><o:p> </o:p></span></p><p class=MsoNormal><span lang=EN-US style='font-size:11.0pt;font-family:"Calibri",sans-serif;mso-fareast-language:EN-US'><o:p> </o:p></span></p><div style='border:none;border-left:solid blue 1.5pt;padding:0cm 0cm 0cm 4.0pt'><div><div style='border:none;border-top:solid #E1E1E1 1.0pt;padding:3.0pt 0cm 0cm 0cm'><p class=MsoNormal><b><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'>From:</span></b><span style='font-size:11.0pt;font-family:"Calibri",sans-serif'> Users <users-bounces@clusterlabs.org> <b>On Behalf Of </b>Fabrizio Ermini<br><b>Sent:</b> Friday, May 9, 2025 5:26 PM<br><b>To:</b> users@clusterlabs.org<br><b>Subject:</b> [ClusterLabs] Rebuild of failed node<o:p></o:p></span></p></div></div><p class=MsoNormal><o:p> </o:p></p><div><div><p class=MsoNormal>Hi all! Freshmen here, just joined. <o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>I'm currently in the need to rebuild a failed node on a pacemaker2.1/corosync3.1 2-node cluster with drbd storage. <o:p></o:p></p></div><div><p class=MsoNormal>I've searched in Pacemaker docs and in the list archives, but I haven't found a clear guide on how to proceed in this task. So far, I've reinstalled a new server, configured the same IP and hostname of the failed one, and installed all the software. I've also fixed DRBD layer and started the resync of the volumes. But it's not clear to me how to proceed - I've found some hints online pointing to the need of manually copying corosync config, but they were quite old and probably obsolete. I'm using pcs as a shell and I haven't found a command designed to replace a node, only to add or remove them. <o:p></o:p></p></div><div><p class=MsoNormal>It seems really strange to me that there isn't a guide, since this should be a very basic operation and it's quite important to know how to do it - HW breaks, as a matter of fact :D<o:p></o:p></p></div><div><p class=MsoNormal>So I'll be very grateful if anyone can point me in the right direction.<o:p></o:p></p></div><div><p class=MsoNormal>Thanks in advance, and best regards<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div><div><p class=MsoNormal>Fabrizio<o:p></o:p></p></div><div><p class=MsoNormal><o:p> </o:p></p></div></div></div></div></body></html>