<html>
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
</head>
<body style="word-wrap: break-word; -webkit-nbsp-mode: space; line-break: after-white-space;" class="">
<div class=""><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;"><span><span><span style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;"><span><span><span><span><span><span><span><span><span>
<div style="caret-color: rgb(0, 0, 0); color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px; text-decoration: none;" class="">
<br class="">
</div>
</span></span></span></span></span></span></span></span></span></span></span></span></span></div>
<div>
<blockquote type="cite" class="">
<div class="">On Jan 11, 2019, at 3:53 AM, Jan Pokorný <<a href="mailto:jpokorny@redhat.com" class="">jpokorny@redhat.com</a>> wrote:</div>
<br class="Apple-interchange-newline">
<div class="">
<div class="">On 11/01/19 00:16 +0000, Israel Brewster wrote:<br class="">
<blockquote type="cite" class="">On Jan 10, 2019, at 10:57 AM, Israel Brewster <<a href="mailto:ibrewster@flyravn.com" class="">ibrewster@flyravn.com</a><<a href="mailto:ibrewster@flyravn.com" class="">mailto:ibrewster@flyravn.com</a>>> wrote:<br class="">
<blockquote type="cite" class=""><br class="">
So in my ongoing work to upgrade my cluster to CentOS 7, I got one<br class="">
box up and running on CentOS 7, with the cluster fully configured<br class="">
and functional, and moved all my services over to it. Now I'm trying<br class="">
to add a second node, following the directions here:<br class="">
<br class="">
<a href="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/high_availability_add-on_reference/s1-clusternodemanage-haar#s2-nodeadd-HAAR" class="">https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/high_availability_add-on_reference/s1-clusternodemanage-haar#s2-nodeadd-HAAR</a><br class="">
<br class="">
However, it doesn't appear to be working. The existing node is named<br class="">
"follow3", and the new node I am trying to add is named "follow1":<br class="">
<br class="">
- The auth command run from follow3 returns "follow1: Authorized", so that looks good.<br class="">
- The "pcs cluster node add follow1" command, again run on follow3, gives the following output:<br class="">
<br class="">
Disabling SBD service...<br class="">
follow1: sbd disabled<br class="">
Sending remote node configuration files to 'follow1'<br class="">
follow1: successful distribution of the file 'pacemaker_remote authkey'<br class="">
follow3: Corosync updated<br class="">
Setting up corosync...<br class="">
follow1: Succeeded<br class="">
Synchronizing pcsd certificates on nodes follow1...<br class="">
follow1: Success<br class="">
Restarting pcsd on the nodes in order to reload the certificates...<br class="">
follow1: Success<br class="">
<br class="">
...So it would appear that that worked as well. I then issued the<br class="">
"pcs cluster start --all" command, which gave the following output:<br class="">
<br class="">
[root@follow3 ~]# pcs cluster start --all<br class="">
follow3: Starting Cluster (corosync)...<br class="">
follow1: Starting Cluster (corosync)...<br class="">
follow3: Starting Cluster (pacemaker)...<br class="">
follow1: Starting Cluster (pacemaker)...<br class="">
<br class="">
So again, everything looks good (to me). However, when I run "pcs<br class="">
status" on the existing node, I get the following:<br class="">
<br class="">
[root@follow3 ~]# pcs status<br class="">
Cluster name: follow<br class="">
Stack: corosync<br class="">
Current DC: follow3 (version 1.1.19-8.el7_6.2-c3c624ea3d) - partition with quorum<br class="">
Last updated: Thu Jan 10 10:47:33 2019<br class="">
Last change: Wed Jan 9 21:39:37 2019 by root via cibadmin on follow3<br class="">
<br class="">
1 node configured<br class="">
29 resources configured<br class="">
<br class="">
Online: [ follow3 ]<br class="">
<br class="">
Full list of resources:<br class="">
<br class="">
which would seem to indicate that it doesn't know about the node I<br class="">
just added (follow1). Meanwhile, follow1 "pcs status" shows this:<br class="">
<br class="">
[root@follow1 ~]# pcs status<br class="">
Cluster name: follow<br class="">
Stack: corosync<br class="">
Current DC: follow1 (version 1.1.19-8.el7_6.2-c3c624ea3d) - partition WITHOUT quorum<br class="">
Last updated: Thu Jan 10 10:54:25 2019<br class="">
Last change: Thu Jan 10 10:54:13 2019 by root via cibadmin on follow1<br class="">
<br class="">
2 nodes configured<br class="">
0 resources configured<br class="">
<br class="">
Online: [ follow1 ]<br class="">
OFFLINE: [ follow3 ]<br class="">
<br class="">
No resources<br class="">
<br class="">
<br class="">
Daemon Status:<br class="">
corosync: active/disabled<br class="">
pacemaker: active/disabled<br class="">
pcsd: active/enabled<br class="">
<br class="">
So it got at least *some* of the config, but apparently not the full<br class="">
thing (no resources), and it shows follow3 as offline, even though<br class="">
it is online and reachable. Oddly "pcs cluster status" shows both<br class="">
follow1 and follow3 pcsd status as online. What am I missing here?<br class="">
</blockquote>
<br class="">
As a follow-up to the above, restarting corosync on the functioning<br class="">
node (follow3) at least allows the second node (follow1) to show up<br class="">
when I do a pcs status, however the second node still shows as<br class="">
OFFLINE (and follow3 shows as offline on follow1), and follow1 is<br class="">
still missing pretty much all of the config. If I try to remove and<br class="">
re-add follow1, the removal works as expected (node count on follow3<br class="">
drops to 1), but the add behaves exactly the same as before, with<br class="">
pcs status not acknowledging the added node.<br class="">
</blockquote>
<br class="">
What do the logs on follow1 have to say about this?<br class="">
E.g. journalctl -b --no-hostname -u corosync -u pacemaker, focusing<br class="">
on the respective suspect time.<br class="">
<br class="">
If there's nothing sufficiently explaining what actually happened,<br class="">
you can still review the underlying pcs communication itself if you<br class="">
pass --debug to it.<br class="">
<br class="">
I suspect that simply one corosync instance doesn't see the other<br class="">
for whatever reason (firewall, bad addresses or not on the same<br class="">
network at all, addresses out of sync between particular nodes,<br class="">
in corosync.conf, or possibly even in /etc/hosts or DNS source,<br class="">
...).<br class="">
<br class="">
</div>
</div>
</blockquote>
<div><br class="">
</div>
<div>So apparently this was something messed up on Follow3, although I don't know what. I ended up doing the following, which worked:</div>
<div><br class="">
</div>
<div>1) Set up a new VM ('follow4')</div>
<div>2) cluster it with follow1</div>
<div>3) Dump JUST the resources and constraints from follow3</div>
<div>4) load the above .xml files to the new cluster (follow1 and follow4)</div>
<div><br class="">
</div>
<div>Once I did the above, I was able to add an additional node (follow2) to the new follow1/follow4 cluster with no problems. So while I don't know what was going on with follow3, at least I now have a properly functioning cluster again!</div>
<br class="">
<blockquote type="cite" class="">
<div class="">
<div class="">-- <br class="">
Nazdar,<br class="">
Jan (Poki)<br class="">
_______________________________________________<br class="">
Users mailing list: <a href="mailto:Users@clusterlabs.org" class="">Users@clusterlabs.org</a><br class="">
<a href="https://lists.clusterlabs.org/mailman/listinfo/users" class="">https://lists.clusterlabs.org/mailman/listinfo/users</a><br class="">
<br class="">
Project Home: http://www.clusterlabs.org<br class="">
Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf<br class="">
Bugs: http://bugs.clusterlabs.org<br class="">
</div>
</div>
</blockquote>
</div>
<br class="">
</body>
</html>