<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">On Oct 5, 2016, at 9:38 AM, Ken Gaillot <<a href="mailto:kgaillot@redhat.com" class="">kgaillot@redhat.com</a>> wrote:<br class=""><div><blockquote type="cite" class=""><br class="Apple-interchange-newline"><div class=""><div class="">On 10/05/2016 11:56 AM, Israel Brewster wrote<blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class="">I never did any specific configuring of CMAN, Perhaps that's the<br class="">problem? I missed some configuration steps on setup? I just<br class="">followed the<br class="">directions<br class="">here:<br class=""><a href="http://jensd.be/156/linux/building-a-high-available-failover-cluster-with-pacemaker-corosync-pcs" class="">http://jensd.be/156/linux/building-a-high-available-failover-cluster-with-pacemaker-corosync-pcs</a>,<br class="">which disabled stonith in pacemaker via the<br class="">"pcs property set stonith-enabled=false" command. Is there<br class="">separate CMAN<br class="">configs I need to do to get everything copacetic? If so, can you<br class="">point<br class="">me to some sort of guide/tutorial for that?<br class=""></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote></blockquote><br class="">If you ran "pcs cluster setup", it configured CMAN for you. Normally you<br class="">don't need to modify those values, but you can see them in<br class="">/etc/cluster/cluster.conf.<br class=""></div></div></blockquote><div><br class=""></div><div>Good to know. So I'm probably OK on that front.</div><br class=""><blockquote type="cite" class=""><div class=""><div class=""><blockquote type="cite" class=""><br class="">So in any case, I guess the next step here is to figure out how to do<br class="">fencing properly, using controllable power strips or the like. Back to<br class="">the drawing board!<br class=""></blockquote><br class="">It sounds like you're on the right track for fencing, but it may not be<br class="">your best next step. Currently, your nodes are trying to fence each<br class="">other endlessly, so if you get fencing working, one of them will<br class="">succeed, and you just have a new problem. :-)<br class=""><br class="">Check the logs for the earliest occurrence (after starting the cluster)<br class="">of the "Requesting Pacemaker fence" message. Look back from that time in<br class="">/var/log/messages, /var/log/cluster/*, and /var/log/pacemaker.log (not<br class="">necessarily all will be present on your system) to try to figure out why<br class="">it wants to fence.<br class=""><br class="">One thing I noticed is that you're running CentOS 6.8, but your<br class="">pacemaker version is 1.1.11. CentOS 6.8 shipped with 1.1.14, so maybe<br class="">you partially upgraded your system from an earlier OS version? I'd try<br class="">applying all updates (especially cman, libqb, corosync, and pacemaker).<br class=""></div></div></blockquote><div><br class=""></div><div>I think what's you're seeing is pacemaker on my primary DB server, which is still at CentOS 6.7. The other servers I've managed to update, but I haven't figured out a *good* HA solution for my DB server (PostgreSQL 9.4 running streaming replication with named replication slots). That is, I can fail over *relatively* easily (touch a file on the secondary, move the IP, and hope all the persistent DB connections reconnect without issue), but getting the demoted primary back up and running is more of a chore (the pg_rewind feature of PostgreSQL 9.5 looks to help with this, but I'm not up to 9.5 yet). As such, I haven't updated the primary DB server as much as some of the others.</div><div><br class=""></div><div>Proper integration of the DB with pacemaker is something I need to look into again, but I took a stab at it when I was first setting up the application cluster, and didn't have much luck.</div><br class=""><blockquote type="cite" class=""><div class=""><div class=""><blockquote type="cite" class=""><blockquote type="cite" class=""><blockquote type="cite" class="">Now if there is a version of fencing that simply<br class="">e-mails/texts/whatever me and says "Ummm... something is wrong with<br class="">that machine over there, you need to do something about it, because I<br class="">can't guarantee operation otherwise", I could go for that. <br class=""></blockquote></blockquote></blockquote><br class="">As digimer mentioned elsewhere, one variation is to use "fabric"<br class="">fencing, i.e. cutting off all external access (disk and/or network) to<br class="">the node. That leaves it up but unable to cause any trouble, so you can<br class="">investigate.<br class=""><br class="">If the disk is all local, or accessed over the network, then asking an<br class="">intelligent switch to cut off network access is sufficient. If the disk<br class="">is shared (e.g. iSCSI), then you need to cut it off, too.<br class=""></div></div></blockquote><div><br class=""></div><div>All disks are local, which would simplify this option, especially considering that I don't have any remote power control options available at the moment. I mentioned getting switched PDU's to my boss, and he'll look into it, but thinks it might not fit into his budget. If I could simply down the proper ports on the Cisco switch(s) the machines are connected to, that could be a viable alternative without any additional hardware needed.</div><div><br class=""></div><div>Thanks!</div><div><br class=""></div><div><div style="text-align: -webkit-auto; font-variant-ligatures: normal; font-variant-position: normal; font-variant-numeric: normal; font-variant-alternates: normal; font-variant-east-asian: normal; line-height: normal; orphans: 2; widows: 2; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="font-family: Helvetica, sans-serif;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">-----------------------------------------------<o:p class=""></o:p></span></div></div><div style="font-family: Helvetica, sans-serif;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">Israel Brewster<o:p class=""></o:p></span></div></div><div style="font-family: Helvetica, sans-serif;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">Systems Analyst II<o:p class=""></o:p></span></div></div><div style="font-family: Helvetica, sans-serif;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">Ravn Alaska<o:p class=""></o:p></span></div></div><div style="font-family: Helvetica, sans-serif;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">5245 Airport Industrial Rd<o:p class=""></o:p></span></div></div><div style="font-family: Helvetica, sans-serif;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">Fairbanks, AK 99709<o:p class=""></o:p></span></div></div><div style="font-family: Helvetica, sans-serif;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">(907) 450-7293<o:p class=""></o:p></span></div></div><div style="font-family: Helvetica, sans-serif;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">-----------------------------------------------</span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""></span></div></div></div></div><br class=""><blockquote type="cite" class=""><div class=""><div class=""><br class=""><blockquote type="cite" class=""><blockquote type="cite" class="">No, that is not fencing.<br class=""><br class="">-- <br class="">Digimer<br class="">Papers and Projects: <a href="https://alteeve.ca/w/" class="">https://alteeve.ca/w/</a><br class="">What if the cure for cancer is trapped in the mind of a person without<br class="">access to education?<br class=""></blockquote></blockquote><br class="">_______________________________________________<br class="">Users mailing list: <a href="mailto:Users@clusterlabs.org" class="">Users@clusterlabs.org</a><br class=""><a href="http://clusterlabs.org/mailman/listinfo/users" class="">http://clusterlabs.org/mailman/listinfo/users</a><br class=""><br class="">Project Home: http://www.clusterlabs.org<br class="">Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf<br class="">Bugs: http://bugs.clusterlabs.org<br class=""></div></div></blockquote></div><br class=""></body></html>