<html>
<head>
<meta http-equiv="content-type" content="text/html; charset=UTF-8">
</head>
<body>
<p>Hi,</p>
<p>I used to have a working cluster with 3 nodes (and stonith
disabled). After an unexpected restart of one node, the cluster
split. The node #2 started to see the others as unclean. Nodes 1
and 2 were cooperating with each other, showing #2 as offline.
There were no network connection problems.<br>
</p>
<p>I removed #2 (operating from #1) with<br>
<font face="monospace">pcs cluster node remove n2</font></p>
<p>I verified that it had removed all configuration from #2, both
for corosync and for pacemaker. The cluster looks like working
correctly with two nodes (and no traces of #2).<br>
</p>
<p>Now I am trying to add the third node back.<br>
<font face="monospace">pcs cluster node add n2<br>
Disabling SBD service...<br>
n2: sbd disabled<br>
Sending 'corosync authkey', 'pacemaker authkey' to 'n2'<br>
n2: successful distribution of the file 'corosync authkey'<br>
n2: successful distribution of the file 'pacemaker authkey'<br>
Sending updated corosync.conf to nodes...<br>
n3: Succeeded<br>
n2: Succeeded<br>
n1: Succeeded<br>
n3: Corosync configuration reloaded</font><br>
</p>
<p>I am able to start #2 operating from #1</p>
<p><font face="monospace">pcs cluster pcsd-status<br>
n2: Online<br>
n3: Online<br>
n1: Online</font></p>
<font face="monospace"> </font>
<p><font face="monospace">pcs cluster enable n2<br>
pcs cluster start n2<br>
</font> </p>
<font face="monospace"> </font>
<p>I can see that corosync's configuration has been updated, but
pacemaker's not.</p>
<p><u>Checking from #1:</u><br>
</p>
<p><font face="monospace">pcs config<br>
Cluster Name: n<br>
Corosync Nodes:<br>
n1 n3 n2<br>
Pacemaker Nodes:<br>
n1 n3<br>
[...]</font></p>
<p><font face="monospace">pcs status<br>
* 2 nodes configured<br>
Node List:<br>
* Online: [ n1 n3 ]<br>
[...]</font></p>
<p><font face="monospace">pcs cluster cib scope=nodes<br>
<nodes><br>
<node id="1" uname="n1"/><br>
<node id="3" uname="n3"/><br>
</nodes></font><br>
</p>
<p><u>#2 is seeing the state differently:</u></p>
<p><font face="monospace">pcs config<br>
Cluster Name: n<br>
Corosync Nodes:<br>
n1 n3 n2<br>
Pacemaker Nodes:<br>
n1 n2 n3</font></p>
<p><font face="monospace">pcs status<br>
* 3 nodes configured<br>
Node List:<br>
* Online: [ n2 ]<br>
* OFFLINE: [ n1 n3 ]<br>
Full List of Resources:<br>
* No resources<br>
[...]</font><br>
(there are resources configured on #1 and #3)</p>
<p><font face="monospace">pcs cluster cib scope=nodes<br>
<nodes><br>
<node id="1" uname="n1"/><br>
<node id="3" uname="n3"/><br>
<node id="2" uname="n2"/><br>
</nodes></font></p>
<p>Help me diagnose it please. Where should I look for the problem?
(I have already tried a few things more - I see nothing helpful in
log files, pcs --debug shows nothing suspicious, tried even
editing the CIB manually)<br>
</p>
<p>Best regards,</p>
<p>Piotr Szafarczyk<br>
</p>
</body>
</html>