<div dir="ltr">Hi <span style="font-family:arial,sans-serif;font-size:13px">Chrissie,</span><div><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><div><span style="font-family:arial,sans-serif;font-size:13px">But wait_for_all doesn't help when there is no connection between the nodes.</span></div>
<div><span style="font-family:arial,sans-serif;font-size:13px">Because in case I need to reboot the remaining working node I won't get working cluster after that - both nodes will be waiting connection between them.</span></div>
<div><font face="arial, sans-serif">That's why I am looking for the solution which could help me to get one node working in this situation (after reboot).</font></div><div><font face="arial, sans-serif">I've been thinking about some kind of marker which could help a node to determine a state of the other node.</font></div>
<div><font face="arial, sans-serif">Like external disk and SCSI reservation command. Maybe you could suggest another kind of marker?</font></div><div><font face="arial, sans-serif">I am not sure can we use a presents of a file on external SSD as the marker. Kind of: if there is a file - the other node is alive, if no - node is dead.</font></div>
<div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif">Digimer,<br></font></div><div><font face="arial, sans-serif"><br></font></div><div><font face="arial, sans-serif">Thanks for the links and information.</font></div>
<div><font face="arial, sans-serif">Anyway if I go this way, I will write my own daemon to determine a state of the other node.</font></div><div><font face="arial, sans-serif">Also the information about fence loop is new for me, thanks =)</font></div>
</div><div class="gmail_extra"><br clear="all"><div><div dir="ltr">Thank you,<div>Kostya</div></div></div>
<br><br><div class="gmail_quote">On Tue, Jun 24, 2014 at 10:55 AM, Christine Caulfield <span dir="ltr"><<a href="mailto:ccaulfie@redhat.com" target="_blank">ccaulfie@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="">On 23/06/14 15:49, Digimer wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Kostya,<br>
<br>
I'm having a little trouble understanding your question, sorry.<br>
<br>
On boot, the node will not start anything, so after booting it, you<br>
log in, check that it can talk to the peer node (a simple ping is<br>
generally enough), then start the cluster. It will join the peer's<br>
existing cluster (even if it's a cluster on just itself).<br>
<br>
If you booted both nodes, say after a power outage, you will check<br>
the connection (again, a simple ping is fine) and then start the cluster<br>
on both nodes at the same time.<br>
</blockquote>
<br>
<br></div>
wait_for_all helps with most of these situations. If a node goes down then it won't start services until it's seen the non-failed node because wait_for_all prevents a newly rebooted node from doing anything on its own. This also takes care of the case where both nodes are rebooted together of course, because that's the same as a new start.<br>
<br>
Chrissie<div class="HOEnZb"><div class="h5"><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
If one of the nodes needs to be shut down, say for repairs or<br>
upgrades, you migrate the services off of it and over to the peer node,<br>
then you stop the cluster (which tells the peer that the node is leaving<br>
the cluster). After that, the remaining node operates by itself. When<br>
you turn it back on, you rejoin the cluster and migrate the services back.<br>
<br>
I think, maybe, you are looking at things more complicated than you<br>
need to. Pacemaker and corosync will handle most of this for you, once<br>
setup properly. What operating system do you plan to use, and what<br>
cluster stack? I suspect it will be corosync + pacemaker, which should<br>
work fine.<br>
<br>
digimer<br>
<br>
On 23/06/14 10:36 AM, Kostiantyn Ponomarenko wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Digimer,<br>
<br>
Suppose I disabled to cluster on start up, but what about remaining<br>
node, if I need to reboot it?<br>
So, even in case of connection lost between these two nodes I need to<br>
have one node working and providing resources.<br>
How did you solve this situation?<br>
Should it be a separate daemon which checks somehow connection between<br>
the two nodes and decides to run corosync and pacemaker or to keep them<br>
down?<br>
<br>
Thank you,<br>
Kostya<br>
<br>
<br>
On Mon, Jun 23, 2014 at 4:34 PM, Digimer <<a href="mailto:lists@alteeve.ca" target="_blank">lists@alteeve.ca</a><br>
<mailto:<a href="mailto:lists@alteeve.ca" target="_blank">lists@alteeve.ca</a>>> wrote:<br>
<br>
On 23/06/14 09:11 AM, Kostiantyn Ponomarenko wrote:<br>
<br>
Hi guys,<br>
I want to gather all possible configuration variants for 2-node<br>
cluster,<br>
because it has a lot of pitfalls and there are not a lot of<br>
information<br>
across the internet about it. And also I have some questions<br>
about<br>
configurations and their specific problems.<br>
VARIANT 1:<br>
-----------------<br>
We can use "two_node" and "wait_for_all" option from Corosync's<br>
votequorum, and set up fencing agents with delay on one of them.<br>
Here is a workflow(diagram) of this configuration:<br>
1. Node start.<br>
2. Cluster start (Corosync and Pacemaker) at the boot time.<br>
3. Wait for all nodes. All nodes joined?<br>
No. Go to step 3.<br>
Yes. Go to step 4.<br>
4. Start resources.<br>
5. Split brain situation (something with connection between<br>
nodes).<br>
6. Fencing agent on the one of the nodes reboots the other node<br>
(there<br>
is a configured delay on one of the Fencing agents).<br>
7. Rebooted node go to step 1.<br>
There are two (or more?) important things in this configuration:<br>
1. Rebooted node remains waiting for all nodes to be visible<br>
(connection<br>
should be restored).<br>
2. Suppose connection problem still exists and the node which<br>
rebooted<br>
the other guy has to be rebooted also (for some reasons). After<br>
reboot<br>
he is also stuck on step 3 because of connection problem.<br>
QUESTION:<br>
-----------------<br>
Is it possible somehow to assign to the guy who won the reboot<br>
race<br>
(rebooted other guy) a status like a "primary" and allow him not<br>
to wait<br>
for all nodes after reboot. And neglect this status after<br>
other node<br>
joined this one.<br>
So is it possible?<br>
Right now that's the only configuration I know for 2 node<br>
cluster.<br>
Other variants are very appreciated =)<br>
VARIANT 2 (not implemented, just a suggestion):<br>
-----------------<br>
I've been thinking about using external SSD drive (or other<br>
external<br>
drive). So for example fencing agent can reserve SSD using SCSI<br>
command<br>
and after that reboot the other node.<br>
The main idea of this is the first node, as soon as a cluster<br>
starts on<br>
it, reserves SSD till the other node joins the cluster, after<br>
that SCSI<br>
reservation is removed.<br>
1. Node start<br>
2. Cluster start (Corosync and Pacemaker) at the boot time.<br>
3. Reserve SSD. Did it manage to reserve?<br>
No. Don't start resources (Wait for all).<br>
Yes. Go to step 4.<br>
4. Start resources.<br>
5. Remove SCSI reservation when the other node has joined.<br>
5. Split brain situation (something with connection between<br>
nodes).<br>
6. Fencing agent tries to reserve SSD. Did it manage to reserve?<br>
No. Maybe puts node in standby mode ...<br>
Yes. Reboot the other node.<br>
7. Optional: a single node can keep SSD reservation till he is<br>
alone in<br>
the cluster or till his shut-down.<br>
I am really looking forward to find the best solution (or a<br>
couple of<br>
them =)).<br>
Hope I am not the only person ho is interested in this topic.<br>
<br>
<br>
Thank you,<br>
Kostya<br>
<br>
<br>
Hi Kostya,<br>
<br>
I only build 2-node clusters, and I've not had problems with this<br>
going back to 2009 over dozens of clusters. The tricks I found are:<br>
<br>
* Disable quorum (of course)<br>
* Setup good fencing, and add a delay to the node you you prefer (or<br>
pick one at random, if equal value) to avoid dual-fences<br>
* Disable to cluster on start up, to prevent fence loops.<br>
<br>
That's it. With this, your 2-node cluster will be just fine.<br>
<br>
As for your question; Once a node is fenced successfully, the<br>
resource manager (pacemaker) will take over any services lost on the<br>
fenced node, if that is how you configured it. A node the either<br>
gracefully leaves or dies/fenced should not interfere with the<br>
remaining node.<br>
<br>
The problem is when a node vanishes and fencing fails. Then, not<br>
knowing what the other node might be doing, the only safe option is<br>
to block, otherwise you risk a split-brain. This is why fencing is<br>
so important.<br>
<br>
Cheers<br>
<br>
--<br>
Digimer<br>
Papers and Projects: <a href="https://alteeve.ca/w/" target="_blank">https://alteeve.ca/w/</a><br>
What if the cure for cancer is trapped in the mind of a person<br>
without access to education?<br>
<br>
______________________________<u></u>___________________<br>
Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org" target="_blank">Pacemaker@oss.clusterlabs.org</a><br>
<mailto:<a href="mailto:Pacemaker@oss.clusterlabs.org" target="_blank">Pacemaker@oss.<u></u>clusterlabs.org</a>><br>
<a href="http://oss.clusterlabs.org/__mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/__<u></u>mailman/listinfo/pacemaker</a><br>
<<a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/<u></u>mailman/listinfo/pacemaker</a>><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
Getting started:<br>
<a href="http://www.clusterlabs.org/__doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/__<u></u>doc/Cluster_from_Scratch.pdf</a><br>
<<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/<u></u>doc/Cluster_from_Scratch.pdf</a>><br>
Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a><br>
<br>
<br>
<br>
<br>
______________________________<u></u>_________________<br>
Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org" target="_blank">Pacemaker@oss.clusterlabs.org</a><br>
<a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/<u></u>mailman/listinfo/pacemaker</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/<u></u>doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a><br>
<br>
</blockquote>
<br>
<br>
</blockquote>
<br>
<br>
______________________________<u></u>_________________<br>
Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org" target="_blank">Pacemaker@oss.clusterlabs.org</a><br>
<a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/<u></u>mailman/listinfo/pacemaker</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/<u></u>doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a><br>
</div></div></blockquote></div><br></div>