[ClusterLabs] [EXT] Re: Rebuild of failed node

[Fabrizio Ermini]

Hi Ulrich, thanks for your reply. I realized I didn't give you guys
feedback. At the end, that's exactly the route I choose:

1) first, reinstall the node with same configurations of old node (IPs,
hostname, /etc/hosts files, drbd resources files, fencing scripts etc.)
2) check DRBD configuration, rebuild DRBD volumes, check that both nodes
are connected (outside of pacemaker - I'm using drbd service at this stage)
3) start pcsd service on new node and execute pcs auth command on both
nodes
4) from the surviving node, remove the old broken node instance
5) add the new node with all rings addresses  (as in the original cluster
creation command)
6) start pacemaker and corosync services on the new node
7) profit! (well, sort of...)

This way all pacemaker configurations have been migrated from the surviving
node, and it appears all went well.
Actually I forgot I had also disabled fencing before starting the
procedure, and re-enabled it when all was working again.

Best regards!


Il giorno ven 6 giu 2025 alle ore 08:05 Windl, Ulrich <u.windl at ukr.de> ha
scritto:

> Fabrizio,
>
>
>
> Yes, it’s easier to configure all cluster nodes at once. Years ago I added
> an extra node to an existing cluster, and it didn’t work as smoothly as
> expected. Cleanly re-adding a node is probably:
>
>    1. Cleanly remove a node
>    2. Cleanly add a node
>
>
>
> 😉
>
>
>
> Kind regards,
>
> Ulrich Windl
>
>
>
> *From:* Users <users-bounces at clusterlabs.org> *On Behalf Of *Fabrizio
> Ermini
> *Sent:* Monday, May 12, 2025 12:56 PM
> *To:* Cluster Labs - All topics related to open-source clustering
> welcomed <users at clusterlabs.org>
> *Subject:* [EXT] Re: [ClusterLabs] Rebuild of failed node
>
>
>
> Hi Ulrich, and thanks for your reply. In this case, the missing node has
> been wiped and formatted. Reason wasn't an actual fault, but rather the
> need to change storage configuration (we had to comply with a security
> requirement that involved adding cryptography at rest of cluster volumes).
> I intended to use this activity also as a test bed to learn what was the
> correct procedure to follow in case of a node loss, so that I could be
> already sure on how to proceed in an emergency. These servers are not under
> backup: they are meant to work as edge servers, collecting data and
> shipping them up towards main production servers. We have a golden image
> that allows us to reinstall them quickly, but the installation procedure is
> based on the assumption that both nodes are installed together: at the
> moment I haven't a procedure to reinstall just one of the nodes, and that's
> what I'd like to create.
>
>
>
> I hope that this can clarify your doubts,  best regards
>
> Fabrizio
>
>
>
>
>
> Il giorno lun 12 mag 2025 alle ore 08:41 Windl, Ulrich <u.windl at ukr.de>
> ha scritto:
>
> Maybe explain what “failed node” and “rebuild” actually means:
>
> It was fenced, or was it reinstalled, or did you have a fatal disk failure?
>
> Usually a backup is your best friend.
>
>
>
> Kind regards,
>
> Ulrich Windl
>
>
>
> *From:* Users <users-bounces at clusterlabs.org> *On Behalf Of *Fabrizio
> Ermini
> *Sent:* Friday, May 9, 2025 4:26 PM
> *To:* users at clusterlabs.org
> *Subject:* [EXT] [ClusterLabs] Rebuild of failed node
>
>
>
> Hi all! Freshmen here, just joined.
>
>
>
> I'm currently in the need to rebuild a failed node on a
> pacemaker2.1/corosync3.1 2-node cluster with drbd storage.
>
> I've searched in Pacemaker docs and in the list archives, but I haven't
> found a clear guide on how to proceed in this task. So far, I've
> reinstalled a new server, configured the same IP and hostname of the failed
> one, and installed all the software. I've also fixed DRBD layer and started
> the resync of the volumes. But it's not clear to me how to proceed - I've
> found some hints online pointing to the need of manually copying corosync
> config, but they were quite old and probably obsolete. I'm using pcs as a
> shell and I haven't found a command designed to replace a node, only to add
> or remove them.
>
> It seems really strange to me that there isn't a guide, since this should
> be a very basic operation and it's quite important to know how to do it -
> HW breaks, as a matter of fact :D
>
> So I'll be very grateful if anyone can point me in the right direction.
>
> Thanks in advance, and best regards
>
>
>
> Fabrizio
>
>
>
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
>
> ClusterLabs home: https://www.clusterlabs.org/
>
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
>
> ClusterLabs home: https://www.clusterlabs.org/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20250606/5abc45f7/attachment.htm>