<p class="MsoNormal" style="border-collapse:collapse;color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px"><span lang="EN-US">H<span style="color:rgb(31,73,125)">i</span>,<u></u><u></u></span></p>
<p class="MsoNormal" style="border-collapse:collapse;color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px"><span lang="EN-US"><u></u> <u></u></span></p>
<p class="MsoNormal" style="border-collapse:collapse;color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px"><span lang="EN-US">I have a cluster 4 nodes (CentOS 5.2) using pacemaker<span style="color:rgb(31,73,125)">-</span>1.0.11<span style="color:rgb(31,73,125)">,</span> with heartbeat<span style="color:rgb(31,73,125)">-</span>3.0.3.<u></u><u></u></span></p>
<p class="MsoNormal" style="border-collapse:collapse;color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px"><span lang="EN-US">The configuration is:</span></p>
<p class="MsoNormal" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px"><span lang="EN-US" style="border-collapse:collapse;font-size:14px"><font class="Apple-style-span" color="#222222" face="arial, sans-serif"></font></span></p>
<p class="MsoNormal" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px"><font class="Apple-style-span" color="#222222" face="arial, sans-serif">[root@node-0 ~]# crm configure show</font></p><p class="MsoNormal" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px">
<font class="Apple-style-span" color="#222222" face="arial, sans-serif">node $id="25b34bc9-06d0-491c-b019-76b7acdfe30f" node-1</font></p><p class="MsoNormal" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px">
<font class="Apple-style-span" color="#222222" face="arial, sans-serif">node $id="578988ce-5e15-4931-a659-e174fc015785" node-0</font></p><p class="MsoNormal" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px">
<font class="Apple-style-span" color="#222222" face="arial, sans-serif">node $id="8a5a9f5c-43d1-4752-921f-4f2eebf16b64" node-3</font></p><p class="MsoNormal" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px">
<font class="Apple-style-span" color="#222222" face="arial, sans-serif">node $id="fd2256ce-027f-4545-b28a-6b73a077e1d2" node-2</font></p><p class="MsoNormal" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px">
<font class="Apple-style-span" color="#222222" face="arial, sans-serif">primitive failover-ip ocf:heartbeat:IPaddr2 \</font></p><p class="MsoNormal" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px">
<font class="Apple-style-span" color="#222222" face="arial, sans-serif"><span class="Apple-tab-span" style="white-space:pre">     </span>params ip="10.10.5.192" \</font></p><p class="MsoNormal" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px">
<font class="Apple-style-span" color="#222222" face="arial, sans-serif"><span class="Apple-tab-span" style="white-space:pre">     </span>op monitor interval="10s"</font></p><p class="MsoNormal" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px">
<font class="Apple-style-span" color="#222222" face="arial, sans-serif">primitive master-app-rsc lsb:cluster-master \</font></p><p class="MsoNormal" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px">
<font class="Apple-style-span" color="#222222" face="arial, sans-serif"><span class="Apple-tab-span" style="white-space:pre">     </span>op monitor interval="10s"</font></p><p class="MsoNormal" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px">
<font class="Apple-style-span" color="#222222" face="arial, sans-serif">primitive node-app-rsc lsb:cluster-node \</font></p><p class="MsoNormal" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px"><font class="Apple-style-span" color="#222222" face="arial, sans-serif"><span class="Apple-tab-span" style="white-space:pre">        </span>op monitor interval="10s"</font></p>
<p class="MsoNormal" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px"><font class="Apple-style-span" color="#222222" face="arial, sans-serif">group group-dc failover-ip master-app-rsc</font></p><p class="MsoNormal" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px">
<font class="Apple-style-span" color="#222222" face="arial, sans-serif">clone clone-node-app-rsc node-app-rsc</font></p><p class="MsoNormal" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px"><font class="Apple-style-span" color="#222222" face="arial, sans-serif">location rule-group-dc group-dc \</font></p>
<p class="MsoNormal" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px"><font class="Apple-style-span" color="#222222" face="arial, sans-serif"><span class="Apple-tab-span" style="white-space:pre">    </span>rule $id="rule-group-dc-rule" -inf: #is_dc eq false</font></p>
<p class="MsoNormal" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px"><font class="Apple-style-span" color="#222222" face="arial, sans-serif">property $id="cib-bootstrap-options" \</font></p>
<p class="MsoNormal" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px"><font class="Apple-style-span" color="#222222" face="arial, sans-serif"><span class="Apple-tab-span" style="white-space:pre">    </span>start-failure-is-fatal="false" \</font></p>
<p class="MsoNormal" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px"><font class="Apple-style-span" color="#222222" face="arial, sans-serif"><span class="Apple-tab-span" style="white-space:pre">    </span>no-quorum-policy="ignore" \</font></p>
<p class="MsoNormal" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px"><font class="Apple-style-span" color="#222222" face="arial, sans-serif"><span class="Apple-tab-span" style="white-space:pre">    </span>symmetric-cluster="true" \</font></p>
<p class="MsoNormal" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px"><font class="Apple-style-span" color="#222222" face="arial, sans-serif"><span class="Apple-tab-span" style="white-space:pre">    </span>stonith-enabled="false" \</font></p>
<p class="MsoNormal" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px"><font class="Apple-style-span" color="#222222" face="arial, sans-serif"><span class="Apple-tab-span" style="white-space:pre">    </span>dc-version="1.0.11-1554a83db0d3c3e546cfd3aaff6af1184f79ee87" \</font></p>
<p class="MsoNormal" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px"><font class="Apple-style-span" color="#222222" face="arial, sans-serif"><span class="Apple-tab-span" style="white-space:pre">    </span>cluster-infrastructure="Heartbeat"</font></p>
<p></p><p class="MsoNormal" style="border-collapse:collapse;color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px"><br></p><p class="MsoNormal" style="border-collapse:collapse;color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px">
<br></p><p class="MsoNormal" style="border-collapse:collapse;color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px">The problem:</p><p class="MsoNormal" style="border-collapse:collapse;color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px">
The "node-2" and "node-3" are shutdown, and their status change to offline.</p><p class="MsoNormal" style="border-collapse:collapse;color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px">
At first, I tried to delete them one by one. </p><p class="MsoNormal" style="border-collapse:collapse;color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px">
When running "/usr/share/heartbeat/hb_delnode node-3" on node-0, the /var/log/messages print:</p><p class="MsoNormal" style="border-collapse:collapse;color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px">
<br></p><p class="MsoNormal" style="border-collapse:collapse;color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px;margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px">--------------------------------------------</p>
<p class="MsoNormal" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px"><font class="Apple-style-span" color="#222222" face="arial, sans-serif"><span class="Apple-style-span" style="border-collapse:collapse;font-size:14px">139558 Mar 31 23:43:59 node-0 heartbeat: [13142]: ERROR: HBDoMsg_T_DELNODE: deletion failed. We don't have all required nodes alive (node-2 is dead)</span></font></p>
<p class="MsoNormal" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px"><font class="Apple-style-span" color="#222222" face="arial, sans-serif"><span class="Apple-style-span" style="border-collapse:collapse;font-size:14px">-------------------------------------------</span></font></p>
<p class="MsoNormal" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px"><font class="Apple-style-span" color="#222222" face="arial, sans-serif"><span class="Apple-style-span" style="border-collapse:collapse;font-size:14px"><br>
</span></font></p><p class="MsoNormal" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px"><font class="Apple-style-span" color="#222222" face="arial, sans-serif"><span class="Apple-style-span" style="border-collapse:collapse;font-size:14px">So I think them should be deleted together.</span></font></p>
<p class="MsoNormal" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px"><font class="Apple-style-span" color="#222222" face="arial, sans-serif"><span class="Apple-style-span" style="border-collapse:collapse;font-size:14px">Then running "</span></font><span class="Apple-style-span" style="border-collapse:collapse;color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px">/usr/share/heartbeat/hb_delnode node-2 node-3</span><span class="Apple-style-span" style="border-collapse:collapse;color:rgb(34,34,34);font-family:arial,sans-serif;font-size:14px">" on node-0, but the /var/log/messages print:</span></p>
<p class="MsoNormal" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px"><font class="Apple-style-span" color="#222222" face="arial, sans-serif"><span class="Apple-style-span" style="border-collapse:collapse;font-size:14px"><br>
</span></font></p><p class="MsoNormal" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px"><font class="Apple-style-span" color="#222222" face="arial, sans-serif"><span class="Apple-style-span" style="border-collapse:collapse;font-size:14px">--------------------------------------</span></font></p>
<p class="MsoNormal" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px"><font class="Apple-style-span" color="#222222" face="arial, sans-serif"><span class="Apple-style-span" style="border-collapse:collapse;font-size:14px"></span></font></p>
<p class="MsoNormal" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px"><font class="Apple-style-span" color="#222222" face="arial, sans-serif">Apr  1 00:00:21 node-0 ccm: [13194]: ERROR: ccm_control_process: Node count from node node-0 does not agree: local count=2, count in message=3</font></p>
<p class="MsoNormal" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px"><font class="Apple-style-span" color="#222222" face="arial, sans-serif">Apr  1 00:00:21 node-0 ccm: [13194]: ERROR: Please make sure <a href="http://ha.cf">ha.cf</a> files on all nodes have same nodes list or add "autojoin any" to <a href="http://ha.cf">ha.cf</a></font></p>
<p class="MsoNormal" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px"><font class="Apple-style-span" color="#222222" face="arial, sans-serif">Apr  1 00:00:21 node-0 ccm: [13194]: info: If this problem persists, check the heartbeat 'hostcache' files in the cluster to look for problems.</font></p>
<p class="MsoNormal" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px"><font class="Apple-style-span" color="#222222" face="arial, sans-serif">--------------------------------------</font></p><p class="MsoNormal" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px">
<font class="Apple-style-span" color="#222222" face="arial, sans-serif"><br></font></p><p class="MsoNormal" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px"><font class="Apple-style-span" color="#222222" face="arial, sans-serif">These two ways are both failed. :(</font></p>
<p class="MsoNormal" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px"><font class="Apple-style-span" color="#222222" face="arial, sans-serif">How could I do, please?</font></p><p class="MsoNormal" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px">
<font class="Apple-style-span" color="#222222" face="arial, sans-serif">Any help is appreciated. </font></p><p class="MsoNormal" style="margin-top:0px;margin-right:0px;margin-bottom:0px;margin-left:0px"><font class="Apple-style-span" color="#222222" face="arial, sans-serif"><br>
</font></p><p></p>