<html><body><p>Poki, <br><br>Once again, I must apologize for presenting you and the users group with some mis-information. After triple checking my<br>note log ... it seems that I described the two actions to you backwards, as it was the kill, not the gentle shutdown that I had issues with,<br>and I had done them in the reverse order. <br><br>In reality, I had attempted the individual `pcs cluster kill` commands first, and it was the kills that were ineffective... and by "ineffective", <br>I mean that the resources were not stopping (I impatiently only waited about 4 minutes before making that determination). <br>I then ran the `pcs cluster stop --all` which seemed to work... and by "work", I mean subsequent attempts to issue <br>pcs status returned the message: "Error: cluster is not currently running on this node". <br><br>You are probably wondering why I would choose the last resort controversial "kill" method over the gentler stop method as my first <br>attempt to stop the cluster. I do not usually do this first, but... so many times I have tried using 'stop' when<br>I have resources in 'failed" state, it takes up to 20 minutes for the stop to quiesce / complete. So, in this case<br>I thought I'd expedite things and try the kill first and see what happens. I'm wondering now if having orphaned <br>virtual domains running is the expected behavior after the kill? <br><br>Full disclosure / with time stamps ... (and a bit of a digression)<br><br>What led up to shutting down the cluster in the first place was that I had numerous VirtualDomain resources and their domains<br>running on multiple hosts concurrently, a disastrous situation which resulted in corruption of many of my virtual images volumes. <br><br>For example: <br><br>zs95kjg109082_res (ocf::heartbeat:VirtualDomain): Started[ zs95KLpcs1 zs93kjpcs1 ]<br> zs95kjg109083_res (ocf::heartbeat:VirtualDomain): Started[ zs95KLpcs1 zs95kjpcs1 ]<br> zs95kjg109084_res (ocf::heartbeat:VirtualDomain): Started zs93KLpcs1<br> zs95kjg109085_res (ocf::heartbeat:VirtualDomain): Started[ zs95kjpcs1 zs93kjpcs1 ]<br> zs95kjg109086_res (ocf::heartbeat:VirtualDomain): Started zs93kjpcs1<br> zs95kjg109087_res (ocf::heartbeat:VirtualDomain): Started zs95KLpcs1<br> zs95kjg109088_res (ocf::heartbeat:VirtualDomain): Started zs93KLpcs1<br> zs95kjg109089_res (ocf::heartbeat:VirtualDomain): Started[ zs95KLpcs1 zs93kjpcs1 ]<br> zs95kjg109090_res (ocf::heartbeat:VirtualDomain): Started zs93kjpcs1<br> zs95kjg109091_res (ocf::heartbeat:VirtualDomain): Started[ zs95KLpcs1 zs95kjpcs1 ]<br> zs95kjg109092_res (ocf::heartbeat:VirtualDomain): Started zs93KLpcs1<br> zs95kjg109094_res (ocf::heartbeat:VirtualDomain): Started[ zs95kjpcs1 zs93kjpcs1 ]<br> zs95kjg109095_res (ocf::heartbeat:VirtualDomain): Started zs93kjpcs1<br> zs95kjg109096_res (ocf::heartbeat:VirtualDomain): Started[ zs95KLpcs1 zs95kjpcs1 ]<br> zs95kjg109097_res (ocf::heartbeat:VirtualDomain): Started zs93KLpcs1<br> zs95kjg109099_res (ocf::heartbeat:VirtualDomain): Started[ zs95kjpcs1 zs93kjpcs1 ]<br> zs95kjg109100_res (ocf::heartbeat:VirtualDomain): Started zs93kjpcs1<br> zs95kjg109101_res (ocf::heartbeat:VirtualDomain): Started zs95KLpcs1<br> zs95kjg109102_res (ocf::heartbeat:VirtualDomain): Started zs93KLpcs1<br> zs95kjg109104_res (ocf::heartbeat:VirtualDomain): Started zs95KLpcs1<br> zs95kjg109105_res (ocf::heartbeat:VirtualDomain): Started zs93kjpcs1<br> zs95kjg110061_res (ocf::heartbeat:VirtualDomain): Started[ zs95KLpcs1 zs95kjpcs1 ]<br><br><br>I also had numerous FAILED resources, which... I strongly suspect, were due to corruption of<br>the virtual system image volumes, which I later had to recover via fsck. <br><br> zs95kjg110099_res (ocf::heartbeat:VirtualDomain): FAILED zs93kjpcs1<br> zs95kjg110100_res (ocf::heartbeat:VirtualDomain): FAILED zs95kjpcs1<br> zs95kjg110101_res (ocf::heartbeat:VirtualDomain): FAILED zs93KLpcs1<br> zs95kjg110102_res (ocf::heartbeat:VirtualDomain): FAILED zs95KLpcs1<br> zs95kjg110098_res (ocf::heartbeat:VirtualDomain): FAILED zs93kjpcs1<br> WebSite (ocf::heartbeat:apache): FAILED zs95kjg110090<br> fence_S90HMC1 (stonith:fence_ibmz): Started zs95kjpcs1<br> zs95kjg110105_res (ocf::heartbeat:VirtualDomain): FAILED zs93KLpcs1<br> zs95kjg110106_res (ocf::heartbeat:VirtualDomain): FAILED zs95KLpcs1<br> zs95kjg110107_res (ocf::heartbeat:VirtualDomain): FAILED zs95kjpcs1<br> zs95kjg110108_res (ocf::heartbeat:VirtualDomain): FAILED[ zs95kjpcs1 zs93kjpcs1 ]<br><br><br>The pacemaker logs were jam packed with this message for many of my VirtualDomain resources: <br><br>Sep 07 15:10:50 [32366] zs93kl crm_resource: ( native.c:97 ) debug: native_add_running: zs95kjg110195_res is active on 2 nodes including zs93kjpcs1: attempting recovery<br><br>I actually reported an earlier occurrence of this in an earlier thread, subject: "[ClusterLabs] "VirtualDomain is active on 2 nodes" due to transient network failure". <br>This happens to be a more recent occurrence of that issue, which we suspect was caused by rebooting a cluster node without first stopping pacemaker on that host. <br>We typically put the node in 'cluster standby', wait for resources to move away from the node, and then issue 'reboot'. <br>The reboot action on our LPARs is configured to perform a halt (deactivate), and then activate. It is not a graceful system shutdown. (end of digression). <br><br>Anyway, in an attempt to stabilize and recover this mess... I did the cluster kills, followed by the cluster stop all as follows:<br><br>[root@<b>zs95kj </b>VD]# date;pcs cluster kill<br>Wed Sep 7 15:28:26 EDT 2016<br>[root@zs95kj VD]#<br><br>[root@<b>zs93kl </b>VD]# date;pcs cluster kill<br>Wed Sep 7 15:28:44 EDT 2016<br><br>[root@zs95kj VD]# date;pcs cluster kill<br>Wed Sep 7 15:29:06 EDT 2016<br>[root@zs95kj VD]#<br><br>[root@zs95KL VD]# date;pcs cluster kill<br>Wed Sep 7 15:29:30 EDT 2016<br>[root@<b>zs95KL </b>VD]#<br><br>[root@zs93kj ~]# date;pcs cluster kill<br>Wed Sep 7 15:30:06 EDT 2016<br>[root@zs93kj ~]#<br><br>[root@zs95kj VD]# pcs status |less<br>Cluster name: test_cluster_2<br>Last updated: Wed Sep 7 15:31:24 2016 Last change: Wed Sep 7 15:14:07 2016 by hacluster via crmd on zs93kjpcs1<br>Stack: corosync<br>Current DC: zs93KLpcs1 (version 1.1.13-10.el7_2.ibm.1-44eb2dd) - partition with quorum<br>106 nodes and 304 resources configured<br><br>Online: [ zs93KLpcs1 zs93kjpcs1 zs95KLpcs1 zs95kjpcs1 ]<br>OFFLINE: [ zs90kppcs1 ]<br><br><br>As I said, I waited about 4 minutes and it didn't look like the resources were stopping, and it also didn't look like the<br>nodes were going offline (note, zs90kppcs1 was shut down, so of course it's offline). So, impatiently, I then did the cluster stop, <br>which surprisingly completed very quickly ...<br><br>[root@zs95kj VD]# date;pcs cluster stop --all<br>Wed Sep 7 15:32:27 EDT 2016<br>zs90kppcs1: Unable to connect to zs90kppcs1 ([Errno 113] No route to host)<br>zs93kjpcs1: Stopping Cluster (pacemaker)...<br>zs95kjpcs1: Stopping Cluster (pacemaker)...<br>zs95KLpcs1: Stopping Cluster (pacemaker)...<br>zs93KLpcs1: Stopping Cluster (pacemaker)...<br>Error: unable to stop all nodes<br>zs90kppcs1: Unable to connect to zs90kppcs1 ([Errno 113] No route to host)<br>You have new mail in /var/spool/mail/root<br><br><br>This was when I discovered that the virtual domains themselves were still running on all the hosts, I ssh'ed this script <br>to "destroy" (forcible shutdown) them...<br><br>[root@zs95kj VD]# cat destroyall.sh<br>for guest in `virsh list |grep running |awk '{print $2}'`; do virsh destroy $guest; done<br><br>[root@zs95kj VD]# ./destroyall.sh<br>Domain zs95kjg110190 destroyed<br><br>Domain zs95kjg110211 destroyed<br><br>.<br>. (omitted dozens of Domain xxx destroyed messages) <br>.<br><br>and then checked them:<br><br>[root@zs95kj VD]# for host in zs93KLpcs1 zs95KLpcs1 zs95kjpcs1 zs93kjpcs1 ; do ssh $host virsh list; done<br> Id Name State<br>----------------------------------------------------<br><br> Id Name State<br>----------------------------------------------------<br><br> Id Name State<br>----------------------------------------------------<br><br> Id Name State<br>----------------------------------------------------<br><br><br>Next, I decided to run this quorum test...which resulted in the unexpected behavior (as originally reported in this thread): <br><br><b>TEST: With pacemaker initially down on all nodes, start cluster on one cluster node at a time, and see what happens when we reach quorum at 3. </b><br><br>[root@zs95kj VD]# date;for host in zs93KLpcs1 zs95KLpcs1 zs95kjpcs1 zs93kjpcs1 ; do ssh $host pcs status; done<br>Wed Sep 7 15:49:27 EDT 2016<br>Error: cluster is not currently running on this node<br>Error: cluster is not currently running on this node<br>Error: cluster is not currently running on this node<br>Error: cluster is not currently running on this node<br><br><br>[root@zs95kj VD]# date;pcs cluster start<br>Wed Sep 7 15:50:00 EDT 2016<br>Starting Cluster...<br>[root@zs95kj VD]#<br><br><br>[root@zs95kj VD]# while true; do date;./ckrm.sh; sleep 10; done<br>Wed Sep 7 15:50:09 EDT 2016<br><br> ### VirtualDomain Resource Statistics: ###<br><br>"_res" Virtual Domain resources:<br> Started on zs95kj: 0<br> Started on zs93kj: 0<br> Started on zs95KL: 0<br> Started on zs93KL: 0<br> Started on zs90KP: 0<br> Total Started: 0<br> Total NOT Started: 200<br><br><br><br>To my surprise, the resources are starting up on zs95kj. Apparently, I have quorum? <br><br><br>[root@zs95kj VD]# date;pcs status |less<br>Wed Sep 7 15:51:17 EDT 2016<br>Cluster name: test_cluster_2<br>Last updated: Wed Sep 7 15:51:18 2016 Last change: Wed Sep 7 15:30:12 2016 by hacluster via crmd on zs93kjpcs1<br>Stack: corosync<br>Current DC: zs95kjpcs1 (version 1.1.13-10.el7_2.ibm.1-44eb2dd) -<b> </b><b><font color="#FF0000">partition with quorum <<< WHY DO I HAVE QUORUM?</font></b><br>106 nodes and 304 resources configured<br><br><font color="#FF0000">Node zs93KLpcs1: pending</font><br><font color="#FF0000">Node zs93kjpcs1: pending</font><br><font color="#FF0000">Node zs95KLpcs1: pending</font><br><font color="#008000">Online: [ zs95kjpcs1 ]</font><br>OFFLINE: [ zs90kppcs1 ]<br><br>Full list of resources:<br><br> zs95kjg109062_res (ocf::heartbeat:VirtualDomain): Started zs95kjpcs1<br> zs95kjg109063_res (ocf::heartbeat:VirtualDomain): Started zs95kjpcs1<br> zs95kjg109064_res (ocf::heartbeat:VirtualDomain): Started zs95kjpcs1<br> zs95kjg109065_res (ocf::heartbeat:VirtualDomain): Started zs95kjpcs1<br> zs95kjg109066_res (ocf::heartbeat:VirtualDomain): Started zs95kjpcs1<br> zs95kjg109067_res (ocf::heartbeat:VirtualDomain): Started zs95kjpcs1<br> zs95kjg109068_res (ocf::heartbeat:VirtualDomain): Started zs95kjpcs1<br>.,<br>.<br>.<br>PCSD Status:<br><font color="#008000"> zs93kjpcs1: Online</font><br><font color="#008000"> zs95kjpcs1: Online</font><br><font color="#008000"> zs95KLpcs1: Online</font><br><font color="#FF0000"> zs90kppcs1: Offline</font><br><font color="#008000"> zs93KLpcs1: Online</font><br><br><br>Check resources again: <br><br>Wed Sep 7 16:09:52 EDT 2016<br><br> ### VirtualDomain Resource Statistics: ###<br><br>"_res" Virtual Domain resources:<br> Started on zs95kj: 199<br> Started on zs93kj: 0<br> Started on zs95KL: 0<br> Started on zs93KL: 0<br> Started on zs90KP: 0<br> Total Started: 199<br> Total NOT Started: 1<br><br><br>I have since isolated all the corrupted virtual domain images and disabled their VirtualDomain resources. <br>We already rebooted all five cluster nodes, after installing a new KVM driver on them. <br><br>Now, the quorum calculation and behavior seems to be working perfectly as expected. <br><br>I started pacemaker on the nodes, one at a time... and, after 3 of the 5 nodes had pacemaker "Online" ...<br>resources activated and were evenly distributed across them. <br><br>In summary, a lesson learned here is to check status of the pcs process to be certain pacemaker and corosync<br>are indeed "offline" and that all threads to that process have terminated. You had mentioned this command: <br><br><tt>pstree -p | grep -A5 $(pidof -x pcs)</tt><br><br>I'm not quite sure what the $(pidof -x pcs) represents?? <br><br>On an "Online" cluster node, I see:<br><br>[root@zs93kj ~]# ps -ef |grep pcs |grep -v grep<br>root 18876 1 0 Sep07 ? 00:00:00 /bin/sh /usr/lib/pcsd/pcsd start<br>root 18905 18876 0 Sep07 ? 00:00:00 /bin/bash -c ulimit -S -c 0 >/dev/null 2>&1 ; /usr/bin/ruby -I/usr/lib/pcsd /usr/lib/pcsd/ssl.rb<br>root 18906 18905 0 Sep07 ? 00:04:22 /usr/bin/ruby -I/usr/lib/pcsd /usr/lib/pcsd/ssl.rb<br>[root@zs93kj ~]#<br><br>If I use the 18876 PID on a healthy node, I get..<br><br>[root@zs93kj ~]# pstree -p |grep -A5 18876<br> |-pcsd(18876)---bash(18905)---ruby(18906)-+-{ruby}(19102)<br> | |-{ruby}(20212)<br> | `-{ruby}(224258)<br> |-pkcsslotd(18851)<br> |-polkitd(19091)-+-{polkitd}(19100)<br> | |-{polkitd}(19101)<br><br><br>Is this what you meant for me to do? If so, I'll be sure to do that next time I suspect processes are not exiting on cluster kill or stop. <br><br>Thanks <br><br><br>Scott Greenlese ... IBM z/BX Solutions Test, Poughkeepsie, N.Y.<br> INTERNET: swgreenl@us.ibm.com <br> PHONE: 8/293-7301 (845-433-7301) M/S: POK 42HA/P966<br><br><br><img width="16" height="16" src="cid:1__=8FBB0ABADFDD77518f9e8a93df938690918c8FB@" border="0" alt="Inactive hide details for Jan Pokorný ---09/08/2016 02:43:21 PM---On 08/09/16 10:20 -0400, Scott Greenlese wrote: > Correction."><font color="#424282">Jan Pokorný ---09/08/2016 02:43:21 PM---On 08/09/16 10:20 -0400, Scott Greenlese wrote: > Correction...</font><br><br><font size="2" color="#5F5F5F">From: </font><font size="2">Jan Pokorný <jpokorny@redhat.com></font><br><font size="2" color="#5F5F5F">To: </font><font size="2">Cluster Labs - All topics related to open-source clustering welcomed <users@clusterlabs.org></font><br><font size="2" color="#5F5F5F">Cc: </font><font size="2">Si Bo Niu <niusibo@cn.ibm.com>, Scott Loveland/Poughkeepsie/IBM@IBMUS, Michael Tebolt/Poughkeepsie/IBM@IBMUS</font><br><font size="2" color="#5F5F5F">Date: </font><font size="2">09/08/2016 02:43 PM</font><br><font size="2" color="#5F5F5F">Subject: </font><font size="2">Re: [ClusterLabs] Pacemaker quorum behavior</font><br><hr width="100%" size="2" align="left" noshade style="color:#8091A5; "><br><br><br><tt>On 08/09/16 10:20 -0400, Scott Greenlese wrote:<br>> Correction...<br>> <br>> When I stopped pacemaker/corosync on the four (powered on / active)<br>> cluster node hosts, I was having an issue with the gentle method of<br>> stopping the cluster (pcs cluster stop --all),<br><br>Can you elaborate on what went wrong with this gentle method, please?<br><br>If it seemed to have stuck, you can perhaps run some diagnostics like:<br><br> pstree -p | grep -A5 $(pidof -x pcs)<br><br>across the nodes to see if what process(es) pcs waits on, next time.<br><br>> so I ended up doing individual (pcs cluster kill <cluster_node>) on<br>> each of the four cluster nodes. I then had to stop the virtual<br>> domains manually via 'virsh destroy <guestname>' on each host.<br>> Perhaps there was some residual node status affecting my quorum?<br><br>Hardly if corosync processes were indeed dead.<br><br>-- <br>Jan (Poki)<br>[attachment "attyopgs.dat" deleted by Scott Greenlese/Poughkeepsie/IBM] _______________________________________________<br>Users mailing list: Users@clusterlabs.org<br></tt><tt><a href="http://clusterlabs.org/mailman/listinfo/users">http://clusterlabs.org/mailman/listinfo/users</a></tt><tt><br><br>Project Home: </tt><tt><a href="http://www.clusterlabs.org">http://www.clusterlabs.org</a></tt><tt><br>Getting started: </tt><tt><a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a></tt><tt><br>Bugs: </tt><tt><a href="http://bugs.clusterlabs.org">http://bugs.clusterlabs.org</a></tt><tt><br></tt><br><br><BR>
</body></html>