<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;"><br><div><div><div>On Nov 7, 2013, at 8:34 PM, Andrew Beekhof <<a href="mailto:andrew@beekhof.net">andrew@beekhof.net</a>> wrote:</div><br class="Apple-interchange-newline"><blockquote type="cite"><div style="font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><br>On 8 Nov 2013, at 4:45 am, Sean Lutner <<a href="mailto:sean@rentul.net">sean@rentul.net</a>> wrote:<br><br><blockquote type="cite">I have a confusing situation that I'm hoping to get help with. Last night after configuring STONITH on my two node cluster, I suddenly have a "ghost" node in my cluster. I'm looking to understand the best way to remove this node from the config.<br><br>I'm using the fence_ec2 device for for STONITH. I dropped the script on each node, registered the device with stonith_admin -R -a fence_ec2 and confirmed the registration with both<br><br># stonith_admin -I<br># pcs stonith list<br><br>I then configured STONITH per the Clusters from Scratch doc<br><br><a href="http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html/Clusters_from_Scratch/_example.html">http://clusterlabs.org/doc/en-US/Pacemaker/1.1-pcs/html/Clusters_from_Scratch/_example.html</a><br><br>Here are my commands:<br># pcs cluster cib stonith_cfg<br># pcs -f stonith_cfg stonith create ec2-fencing fence_ec2 ec2-home="/opt/ec2-api-tools" pcmk_host_check="static-list" pcmk_host_list="ip-10-50-3-122 ip-10-50-3-251" op monitor interval="300s" timeout="150s" op start start-delay="30s" interval="0"<br># pcs -f stonith_cfg stonith<br># pcs -f stonith_cfg property set stonith-enabled=true<br># pcs -f stonith_cfg property<br># pcs cluster push cib stonith_cfg<br><br>After that I saw that STONITH appears to be functioning but a new node listed in pcs status output:<br></blockquote><br>Do the EC2 instances have fixed IPs?<br>I didn't have much luck with EC2 because every time they came back up it was with a new name/address which confused corosync and created situations like this.<br></div></blockquote><div><br></div><div>The IPs persist across reboots as far as I can tell. I thought the problem was due to stonith being enabled but not working so I removed the stonith_id and disabled stonith. After that I restarted pacemaker and cman on both nodes and things started as expected but the ghost node it still there. <div><br></div><div>Someone else working on the cluster exported the CIB, removed the node and then imported the CIB. They used this process <a href="http://clusterlabs.org/doc/en-US/Pacemaker/1.0/html/Pacemaker_Explained/s-config-updates.html">http://clusterlabs.org/doc/en-US/Pacemaker/1.0/html/Pacemaker_Explained/s-config-updates.html</a></div><div><br></div><div>Even after that, the ghost node is still there? Would pcs cluster cib > /tmp/cib-temp.xml and then pcs cluster push cib /tmp/cib-temp.xml after editing the node out of the config?</div><div><br></div><div>I may have to go back to the drawing board on a fencing device for the nodes. Are there any other recommendations for a cluster on EC2 nodes?</div><div><br></div><div>Thanks very much</div></div><br><blockquote type="cite"><div style="font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><br><blockquote type="cite"><br># pcs status<br>Last updated: Thu Nov 7 17:41:21 2013<br>Last change: Thu Nov 7 04:29:06 2013 via cibadmin on ip-10-50-3-122<br>Stack: cman<br>Current DC: ip-10-50-3-122 - partition with quorum<br>Version: 1.1.8-7.el6-394e906<br>3 Nodes configured, unknown expected votes<br>11 Resources configured.<br><br><br>Node ip-10-50-3-1251: UNCLEAN (offline)<br>Online: [ ip-10-50-3-122 ip-10-50-3-251 ]<br><br>Full list of resources:<br><br>ClusterEIP_54.215.143.166 (ocf::pacemaker:EIP): Started ip-10-50-3-122<br>Clone Set: EIP-AND-VARNISH-clone [EIP-AND-VARNISH]<br> Started: [ ip-10-50-3-122 ip-10-50-3-251 ]<br> Stopped: [ EIP-AND-VARNISH:2 ]<br>ec2-fencing (stonith:fence_ec2): Stopped<span class="Apple-converted-space"> </span><br><br>I have no idea where the node that is marked UNCLEAN came from, though it's a clear typo is a proper cluster node.<br><br>The only command I ran with the bad node ID was:<br><br># crm_resource --resource ClusterEIP_54.215.143.166 --cleanup --node ip-10-50-3-1251<br><br>Is there any possible way that could have caused the the node to be added?<br><br>I tried running pcs cluster node remove ip-10-50-3-1251 but since there is no node and thus no pcsd that failed. Is there a way I can safely remove this ghost node from the cluster? I can provide logs from pacemaker or corosync as needed.<br>_______________________________________________<br>Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br><a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br><br>Project Home: http://www.clusterlabs.org<br>Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf<br>Bugs: http://bugs.clusterlabs.org<br></blockquote><br><br>_______________________________________________<br>Pacemaker mailing list:<span class="Apple-converted-space"> </span><a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br><a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br><br>Project Home:<span class="Apple-converted-space"> </span><a href="http://www.clusterlabs.org/">http://www.clusterlabs.org</a><br>Getting started:<span class="Apple-converted-space"> </span><a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>Bugs:<span class="Apple-converted-space"> </span><a href="http://bugs.clusterlabs.org/">http://bugs.clusterlabs.org</a></div></blockquote></div><br></div></body></html>