[ClusterLabs] Quorum when reducing cluster from 3 nodes to 2 nodes

[Hayden,Robert]

I had a SysAdmin reduce the number of nodes in a OL 7.9 cluster from three nodes to two nodes.

>From internal testing, I found the following commands would work and the 2Node attribute would be automatically added.  The other cluster parameters we use are WaitForAll and LastManStanding.


pcs resource disable res_app03

pcs resource delete res_app03

pcs cluster node remove app03

pcs stonith delete fence_app03

Unfortunately, real world didn't go as planned.   I am unsure if the commands were ran out of order or something else was going on (e.g. unexpected location constraints).   When I got involved, I noticed that pcs status had the app3 node in an OFFLINE state, but the pcs cluster node remove app03 command was successful.   I noticed some leftover location constraints from past "moves" of resources.  I manually removed those constraints and I ended up removing the app03 node from the corosync configuration with "crm_node --remove=app3 --force" command.    Now pcs status no longer shows any information for app3 and crm_node -l does not show app3.

My concern is with Quorum.   From the pcs quorum status output below, I still see Quorum set at 2 (expected to be 1) and the 2Node attribute was not added.   Am I stuck in this state until the next full cluster downtime?  Or is there a way to manipulate the expected quorum votes in the run time cluster?

#17:25:08 # pcs quorum status
Quorum information
------------------
Date:             Wed May 26 17:25:16 2021
Quorum provider:  corosync_votequorum
Nodes:            2
Node ID:          3
Ring ID:          1/85
Quorate:          Yes

Votequorum information
----------------------
Expected votes:   2
Highest expected: 2
Total votes:      2
Quorum:           2
Flags:            Quorate WaitForAll LastManStanding

Membership information
----------------------
    Nodeid      Votes    Qdevice Name
         1          1         NR app1
         3          1         NR app2 (local)


CONFIDENTIALITY NOTICE This message and any included attachments are from Cerner Corporation and are intended only for the addressee. The information contained in this message is confidential and may constitute inside or non-public information under international, federal, or state securities laws. Unauthorized forwarding, printing, copying, distribution, or use of such information is strictly prohibited and may be unlawful. If you are not the addressee, please promptly delete this message and notify the sender of the delivery error by e-mail or you may call Cerner's corporate offices in Kansas City, Missouri, U.S.A at (+1) (816)221-1024.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20210526/2de80950/attachment.htm>