<div dir="ltr"><span style="font-family:arial,sans-serif;font-size:13px">Digimer,</span><div><span style="font-family:arial,sans-serif;font-size:13px"><br></span></div><div><div><font face="arial, sans-serif">I am using Debian as OS and Corosync + Pacemaker as cluster stack.</font></div>
<div><font face="arial, sans-serif">I understand your suggestion.</font></div><div><font face="arial, sans-serif">I don't have any questions about it.</font></div><div><font face="arial, sans-serif"> </font></div><div>
<font face="arial, sans-serif">My main question is how to do it automatically?</font></div><div><font face="arial, sans-serif">So that it could work without human interruption for a while (nodes could be rebooted, but not repaired).</font></div>
<div><font face="arial, sans-serif">That is my question.</font></div><div><font face="arial, sans-serif"> </font></div><div><font face="arial, sans-serif">The only way for me to do it is:</font></div><div><font face="arial, sans-serif">Write a daemon which will run Corosync and Pacemaker.</font></div>
<div><font face="arial, sans-serif">But before that this daemon will check the connection between the two nodes.</font></div><div><font face="arial, sans-serif">Then make a decision whether to start cluster or not, based on that check.</font></div>
<div><font face="arial, sans-serif"> </font></div><div><font face="arial, sans-serif">Maybe you have some thoughts how can I do it in other way?</font></div><div><font face="arial, sans-serif">Instead of doing ping the daemon could run Corosync, get the number of nodes in cluster, and based on that it decides whether to run Pacemaker or not.</font></div>
</div><div><font face="arial, sans-serif"><br></font></div></div><div class="gmail_extra"><br clear="all"><div><div dir="ltr">Thank you,<div>Kostya</div></div></div>
<br><br><div class="gmail_quote">On Mon, Jun 23, 2014 at 5:49 PM, Digimer <span dir="ltr"><<a href="mailto:lists@alteeve.ca" target="_blank">lists@alteeve.ca</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Kostya,<br>
<br>
I'm having a little trouble understanding your question, sorry.<br>
<br>
On boot, the node will not start anything, so after booting it, you log in, check that it can talk to the peer node (a simple ping is generally enough), then start the cluster. It will join the peer's existing cluster (even if it's a cluster on just itself).<br>
<br>
If you booted both nodes, say after a power outage, you will check the connection (again, a simple ping is fine) and then start the cluster on both nodes at the same time.<br>
<br>
If one of the nodes needs to be shut down, say for repairs or upgrades, you migrate the services off of it and over to the peer node, then you stop the cluster (which tells the peer that the node is leaving the cluster). After that, the remaining node operates by itself. When you turn it back on, you rejoin the cluster and migrate the services back.<br>
<br>
I think, maybe, you are looking at things more complicated than you need to. Pacemaker and corosync will handle most of this for you, once setup properly. What operating system do you plan to use, and what cluster stack? I suspect it will be corosync + pacemaker, which should work fine.<br>
<br>
digimer<br>
<br>
On 23/06/14 10:36 AM, Kostiantyn Ponomarenko wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hi Digimer,<div class=""><br>
<br>
Suppose I disabled to cluster on start up, but what about remaining<br>
node, if I need to reboot it?<br>
So, even in case of connection lost between these two nodes I need to<br>
have one node working and providing resources.<br>
How did you solve this situation?<br>
Should it be a separate daemon which checks somehow connection between<br>
the two nodes and decides to run corosync and pacemaker or to keep them<br>
down?<br>
<br>
Thank you,<br>
Kostya<br>
<br>
<br>
On Mon, Jun 23, 2014 at 4:34 PM, Digimer <<a href="mailto:lists@alteeve.ca" target="_blank">lists@alteeve.ca</a><br></div><div><div class="h5">
<mailto:<a href="mailto:lists@alteeve.ca" target="_blank">lists@alteeve.ca</a>>> wrote:<br>
<br>
On 23/06/14 09:11 AM, Kostiantyn Ponomarenko wrote:<br>
<br>
Hi guys,<br>
I want to gather all possible configuration variants for 2-node<br>
cluster,<br>
because it has a lot of pitfalls and there are not a lot of<br>
information<br>
across the internet about it. And also I have some questions about<br>
configurations and their specific problems.<br>
VARIANT 1:<br>
-----------------<br>
We can use "two_node" and "wait_for_all" option from Corosync's<br>
votequorum, and set up fencing agents with delay on one of them.<br>
Here is a workflow(diagram) of this configuration:<br>
1. Node start.<br>
2. Cluster start (Corosync and Pacemaker) at the boot time.<br>
3. Wait for all nodes. All nodes joined?<br>
No. Go to step 3.<br>
Yes. Go to step 4.<br>
4. Start resources.<br>
5. Split brain situation (something with connection between nodes).<br>
6. Fencing agent on the one of the nodes reboots the other node<br>
(there<br>
is a configured delay on one of the Fencing agents).<br>
7. Rebooted node go to step 1.<br>
There are two (or more?) important things in this configuration:<br>
1. Rebooted node remains waiting for all nodes to be visible<br>
(connection<br>
should be restored).<br>
2. Suppose connection problem still exists and the node which<br>
rebooted<br>
the other guy has to be rebooted also (for some reasons). After<br>
reboot<br>
he is also stuck on step 3 because of connection problem.<br>
QUESTION:<br>
-----------------<br>
Is it possible somehow to assign to the guy who won the reboot race<br>
(rebooted other guy) a status like a "primary" and allow him not<br>
to wait<br>
for all nodes after reboot. And neglect this status after other node<br>
joined this one.<br>
So is it possible?<br>
Right now that's the only configuration I know for 2 node cluster.<br>
Other variants are very appreciated =)<br>
VARIANT 2 (not implemented, just a suggestion):<br>
-----------------<br>
I've been thinking about using external SSD drive (or other external<br>
drive). So for example fencing agent can reserve SSD using SCSI<br>
command<br>
and after that reboot the other node.<br>
The main idea of this is the first node, as soon as a cluster<br>
starts on<br>
it, reserves SSD till the other node joins the cluster, after<br>
that SCSI<br>
reservation is removed.<br>
1. Node start<br>
2. Cluster start (Corosync and Pacemaker) at the boot time.<br>
3. Reserve SSD. Did it manage to reserve?<br>
No. Don't start resources (Wait for all).<br>
Yes. Go to step 4.<br>
4. Start resources.<br>
5. Remove SCSI reservation when the other node has joined.<br>
5. Split brain situation (something with connection between nodes).<br>
6. Fencing agent tries to reserve SSD. Did it manage to reserve?<br>
No. Maybe puts node in standby mode ...<br>
Yes. Reboot the other node.<br>
7. Optional: a single node can keep SSD reservation till he is<br>
alone in<br>
the cluster or till his shut-down.<br>
I am really looking forward to find the best solution (or a<br>
couple of<br>
them =)).<br>
Hope I am not the only person ho is interested in this topic.<br>
<br>
<br>
Thank you,<br>
Kostya<br>
<br>
<br>
Hi Kostya,<br>
<br>
I only build 2-node clusters, and I've not had problems with this<br>
going back to 2009 over dozens of clusters. The tricks I found are:<br>
<br>
* Disable quorum (of course)<br>
* Setup good fencing, and add a delay to the node you you prefer (or<br>
pick one at random, if equal value) to avoid dual-fences<br>
* Disable to cluster on start up, to prevent fence loops.<br>
<br>
That's it. With this, your 2-node cluster will be just fine.<br>
<br>
As for your question; Once a node is fenced successfully, the<br>
resource manager (pacemaker) will take over any services lost on the<br>
fenced node, if that is how you configured it. A node the either<br>
gracefully leaves or dies/fenced should not interfere with the<br>
remaining node.<br>
<br>
The problem is when a node vanishes and fencing fails. Then, not<br>
knowing what the other node might be doing, the only safe option is<br>
to block, otherwise you risk a split-brain. This is why fencing is<br>
so important.<br>
<br>
Cheers<br>
<br>
--<br>
Digimer<br>
Papers and Projects: <a href="https://alteeve.ca/w/" target="_blank">https://alteeve.ca/w/</a><br>
What if the cure for cancer is trapped in the mind of a person<br>
without access to education?<br>
<br></div></div>
______________________________<u></u>___________________<br>
Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org" target="_blank">Pacemaker@oss.clusterlabs.org</a><br>
<mailto:<a href="mailto:Pacemaker@oss.clusterlabs.org" target="_blank">Pacemaker@oss.<u></u>clusterlabs.org</a>><br>
<a href="http://oss.clusterlabs.org/__mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/__<u></u>mailman/listinfo/pacemaker</a><div class=""><br>
<<a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/<u></u>mailman/listinfo/pacemaker</a>><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
Getting started:<br></div>
<a href="http://www.clusterlabs.org/__doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/__<u></u>doc/Cluster_from_Scratch.pdf</a><div class=""><br>
<<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/<u></u>doc/Cluster_from_Scratch.pdf</a>><br>
Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a><br>
<br>
<br>
<br>
<br>
______________________________<u></u>_________________<br>
Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org" target="_blank">Pacemaker@oss.clusterlabs.org</a><br>
<a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/<u></u>mailman/listinfo/pacemaker</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/<u></u>doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a><br>
<br>
</div></blockquote><div class="HOEnZb"><div class="h5">
<br>
<br>
-- <br>
Digimer<br>
Papers and Projects: <a href="https://alteeve.ca/w/" target="_blank">https://alteeve.ca/w/</a><br>
What if the cure for cancer is trapped in the mind of a person without access to education?<br>
<br>
______________________________<u></u>_________________<br>
Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org" target="_blank">Pacemaker@oss.clusterlabs.org</a><br>
<a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/<u></u>mailman/listinfo/pacemaker</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/<u></u>doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a><br>
</div></div></blockquote></div><br></div>