<p dir="ltr">Finally we added a third node and fencing works great this way when one of them fails. I had to use no-quorum-policy set to freeze for this configuration on SLES 11 SP3.</p>
<div class="gmail_quote">On Jun 10, 2015 9:21 AM, "Digimer" <<a href="mailto:lists@alteeve.ca">lists@alteeve.ca</a>> wrote:<br type="attribution"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">On 10/06/15 04:11 AM, Jonathan Vargas wrote:<br>
> Thanks Digimer,<br>
><br>
> I read an old post where you mention the configuration. However after<br>
> adding "start-delay=15" to my stonith resource, yet both nodes reboot at<br>
> the same time on network disconnect.<br>
<br>
Not 'start-delay', just 'delay'.<br>
<br>
> This is my current configuration after the "start-delay" change:<br>
><br>
> <a href="http://i.imgur.com/1o5bGvj.png" rel="noreferrer" target="_blank">http://i.imgur.com/1o5bGvj.png</a><br>
><br>
> And this is the status of the cluster:<br>
><br>
> <a href="http://i.imgur.com/TJNsHVD.png" rel="noreferrer" target="_blank">http://i.imgur.com/TJNsHVD.png</a><br>
><br>
> I don't have a hardware stonith device, so I think linux watchdog is<br>
> being used. Is ok that the stonith resource be placed on a single node?<br>
<br>
I've not used it.<br>
<br>
The test though is to see if the fencing workings when you crash each<br>
machine (echo c > /proc/sysrq-trigger) and when the machine is alive,<br>
but the network is failed.<br>
<br>
> Any idea about what should I fix?<br>
><br>
> Thanks in advance.<br>
><br>
><br>
><br>
> 2015-06-10 0:27 GMT-06:00 Digimer <<a href="mailto:lists@alteeve.ca">lists@alteeve.ca</a><br>
> <mailto:<a href="mailto:lists@alteeve.ca">lists@alteeve.ca</a>>>:<br>
><br>
> On 10/06/15 01:50 AM, Jonathan Vargas wrote:<br>
> ><br>
> > 2015-06-09 23:26 GMT-06:00 Digimer <<a href="mailto:lists@alteeve.ca">lists@alteeve.ca</a> <mailto:<a href="mailto:lists@alteeve.ca">lists@alteeve.ca</a>><br>
> > <mailto:<a href="mailto:lists@alteeve.ca">lists@alteeve.ca</a> <mailto:<a href="mailto:lists@alteeve.ca">lists@alteeve.ca</a>>>>:<br>
> ><br>
> > On 10/06/15 01:19 AM, Jonathan Vargas wrote:<br>
> > > Thanks Andrei, Digimer.<br>
> > ><br>
> > > I see. Since I need to address this discussion to a<br>
> definitive solution,<br>
> > > I am sharing you a diagram of how we are designing this HA<br>
> architecture,<br>
> > > to clarify the problem we are trying to solve:<br>
> > ><br>
> > > <a href="http://i.imgur.com/BFPcZSx.png" rel="noreferrer" target="_blank">http://i.imgur.com/BFPcZSx.png</a><br>
> ><br>
> > Last block is DRBD. If DRBD will be managed by the cluster, it<br>
> must have<br>
> > fencing.<br>
> ><br>
> > This is your definitive answer.<br>
> ><br>
> > Without it, you *will* get a split-brain. That leads to, at<br>
> best, data<br>
> > divergence or data loss.<br>
> ><br>
> > > The first layer, Load Balancer; and the third later,<br>
> Database, are both<br>
> > > already setup. The Load Balancer cluster uses only an VIP<br>
> resource,<br>
> > > while Database cluster uses DRBD+VIP resources. They are on<br>
> production<br>
> > > and work fine, test passed :-)<br>
> > ><br>
> > > Now we are handling the Web Server layer, which I am<br>
> discussing with<br>
> > > experts like you. These servers require to be all active and<br>
> see the<br>
> > > same data for read & write, as quickly as possible, mainly<br>
> reads.<br>
> > ><br>
> > > *So, If we stay with OCFS2: *Since we need to protect the<br>
> service<br>
> > > availability and keep most of nodes up, what choices do I<br>
> have to avoid<br>
> > > reboots on both Web nodes caused by a split-brain situation<br>
> when one of<br>
> > > them is disconnected from network?<br>
> ><br>
> > None of this matters relative to the importance of working, tested<br>
> > fencing for replicated storage.<br>
> ><br>
> > In any HA setup, the reboot of a node should matter not. If<br>
> you are<br>
> > afraid of rebooting a node, you need to reconsider your design.<br>
> ><br>
> ><br>
> ><br>
> > Well, the problem is caused by a pretty common scenario: A simple<br>
> > network disconnection on node 1 causes both nodes to reboot, even when<br>
> > the node 1 is still offline, it will keep rebooting the active node 2.<br>
> > There were no disk issues, but the service availability was lost.<br>
> > *That's the main complain now :-/*<br>
><br>
> This is a symptom of a configuration issue. It is a separate topic for<br>
> using/not using fencing.<br>
><br>
> First, don't start the cluster when the node boots.<br>
><br>
> A node will boot for one of two reasons only;<br>
><br>
> 1. Node was fenced; You don't want it back into the cluster until you<br>
> know it is safe to do so.<br>
><br>
> 2. Scheduled maintenance; A human is there, so rejoining it after the<br>
> maintenance is over is a non-issue.<br>
><br>
> This solves the fence-on-boot issue. Also, corosync's wait_for_all<br>
> should be used to further protect against this.<br>
><br>
> If the problem is that both fence before they die, then set a delay<br>
> against a node to give it a head-start in fencing the peer. I find<br>
> delay="15" to be a good value.<br>
><br>
><br>
><br>
> Okay. It will solve the problem about one node fencing the other one<br>
> after reboots. But it will require manual intervention to make the<br>
> service available again.<br>
><br>
> What if I disable fencing at all, and I keep syncing a local copy of the<br>
> data on each node's own disk.<br>
><br>
><br>
><br>
><br>
> > > Correct me if I'm wrong:<br>
> > ><br>
> > > *1. Redundant Channel:* This is pretty difficult, since we would<br>
> > have to<br>
> > > add two new physical netword cards to the virtual machine hosts, and<br>
> > > that changes network configuration a lot in the virtualization platform.<br>
> ><br>
> > High Availability must put priorities like hassle and cost second to<br>
> > what makes a system more resilient. If you choose not to spend the extra<br>
> > money or time, then you must accept the risks.<br>
> ><br>
> ><br>
> > > *2. Three Node Cluster:* This is possible, but it will consume more<br>
> > > resources. We can have it only for cluster communication though, not for<br>
> > > web processing, that will decrease load.<br>
> ><br>
> > Quorum is NOT a substitution for fencing. They solve different problems.<br>
> ><br>
> > Quorum is a tool for when all nodes are behaving properly. Fencing is a<br>
> > tool for when a node is not behaving properly.<br>
> ><br>
> ><br>
> ><br>
> > Yes, but by adding a 3rd node, it will help to determine which node<br>
> > could be failing and which are not, to fence the proper one. Right?<br>
><br>
> If you have a 3rd node and you fail the network on one, then in theory,<br>
> yes it will help. In practice, if you down the network on one node, it<br>
> won't be able to fence the other node anyway and will be the fence<br>
> victim.<br>
><br>
> > > *3. Disable Fencing:* You said this should not happen at all if we<br>
> > use a<br>
> > > shared disk like OCFS. So I am discarding it.<br>
> ><br>
> > Correct.<br>
> ><br>
> > > *4. Use NFS: *Yes, this will cause a SPoF, and to solve it we<br>
> > would have<br>
> > > to setup another cluster with DRBD as described here<br>
> > ><br>
> > <<a href="https://www.suse.com/documentation/sle_ha/singlehtml/book_sleha_techguides/book_sleha_techguides.html" rel="noreferrer" target="_blank">https://www.suse.com/documentation/sle_ha/singlehtml/book_sleha_techguides/book_sleha_techguides.html</a>>,<br>
> > > and add more infrastructure resources, or do we can setup NFS over OCFS2?<br>
> ><br>
> > ... Which would require fencing anyway, so you gain nothing but another<br>
> > layer of things to break. First rule of HA; Keep it simple.<br>
> ><br>
> > Complexity is the enemy of availability.<br>
> ><br>
> ><br>
> ><br>
> > Sure, fencing must be added to if this would be the case.<br>
><br>
> Fencing is always needed in HA clusters, full stop.<br>
><br>
><br>
> --<br>
> Digimer<br>
> Papers and Projects: <a href="https://alteeve.ca/w/" rel="noreferrer" target="_blank">https://alteeve.ca/w/</a><br>
> What if the cure for cancer is trapped in the mind of a person without<br>
> access to education?<br>
><br>
> _______________________________________________<br>
> Users mailing list: <a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a> <mailto:<a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a>><br>
> <a href="http://clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://clusterlabs.org/mailman/listinfo/users</a><br>
><br>
> Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
> Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
> Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
><br>
><br>
><br>
><br>
> _______________________________________________<br>
> Users mailing list: <a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a><br>
> <a href="http://clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://clusterlabs.org/mailman/listinfo/users</a><br>
><br>
> Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
> Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
> Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
><br>
<br>
<br>
--<br>
Digimer<br>
Papers and Projects: <a href="https://alteeve.ca/w/" rel="noreferrer" target="_blank">https://alteeve.ca/w/</a><br>
What if the cure for cancer is trapped in the mind of a person without<br>
access to education?<br>
<br>
_______________________________________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a><br>
<a href="http://clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://clusterlabs.org/mailman/listinfo/users</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
</blockquote></div>