<div dir="ltr"><div><div>I'm back with confirmation that DLM is the one that triggers Mount resources to stop when the stopped/suspended node rejoins <br><br>When the DLM resource is started, on the rejoining node, it tries to get a free journal but always gets one that is already occupied by another node and triggers a domino effect, one node jumps to the next node occupied dlm journal, stopping and restarting the mount resources.<br><br></div><div>I have plenty of DLM journals (allocated 10 for 3 node configuration) so for sure there are unused ones available.<br></div><div><br></div>There is a way to make DLM keep its journal when the node is stopped and use it when it starts ? or a way to make it keep it's active allocation forcing the rejoining node to look for another unoccupied journal without pushing away the nodes that occupy the journal it allocates for the node that rejoins?<br><br></div>Best regards<br></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Aug 28, 2017 at 6:04 PM, Octavian Ciobanu <span dir="ltr"><<a href="mailto:coctavian1979@gmail.com" target="_blank">coctavian1979@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr"><div><div><div><div><div>Thank you for info.<br></div>Looking over the output of the crm_simulate I've noticed the "notice" messages and with the help of the debug mode I've found this sequence in log<br><br>Aug 28 16:23:19 [13802] node03 crm_simulate: debug: native_assign_node: Assigning node01 to DLM:2<br>Aug 28 16:23:19 [13802] node03 crm_simulate: notice: color_instance: Pre-allocation failed: got node01 instead of node02<br>Aug 28 16:23:19 [13802] node03 crm_simulate: info: native_deallocate: Deallocating DLM:2 from core01<br>Aug 28 16:23:19 [13802] node03 crm_simulate: debug: native_assign_node: Assigning core01 to DLM:3<br>Aug 28 16:23:19 [13802] node03 crm_simulate: notice: color_instance: Pre-allocation failed: got node01 instead of node03<br>Aug 28 16:23:19 [13802] node03 crm_simulate: info: native_deallocate: Deallocating DLM:3 from core01<br>Aug 28 16:23:19 [13802] node03 crm_simulate: debug: native_assign_node: Assigning core01 to DLM:0<br>Aug 28 16:23:19 [13802] node03 crm_simulate: info: rsc_merge_weights: DLM:2: Rolling back scores from iSCSI2-clone<br>Aug 28 16:23:19 [13802] node03 crm_simulate: info: rsc_merge_weights: DLM:2: Rolling back scores from iSCSI2-clone<br>Aug 28 16:23:19 [13802] node03 crm_simulate: debug: native_assign_node: Assigning node03 to DLM:2<br>Aug 28 16:23:19 [13802] node03 crm_simulate: info: rsc_merge_weights: DLM:3: Rolling back scores from iSCSI2-clone<br>Aug 28 16:23:19 [13802] node03 crm_simulate: info: rsc_merge_weights: DLM:3: Rolling back scores from iSCSI2-clone<br>Aug 28 16:23:19 [13802] node03 crm_simulate: debug: native_assign_node: Assigning node02 to DLM:3<br> <br></div>This suggests that the restarted node attempts to occupy a dlm journal that is allocated to another note and by doing so triggers a chain reaction leading to all resources being restarted on all nodes.<br><br></div>I will try a different approach (based on your suggestions) on starting DLM, iSCSI and Mount resources and see if this changes anything. <br><br></div>If based on the log you have any suggestions they are welcome.<br><br></div>Thank you again for the help.<br></div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Aug 28, 2017 at 3:53 PM, Vladislav Bogdanov <span dir="ltr"><<a href="mailto:bubble@hoster-ok.com" target="_blank">bubble@hoster-ok.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>28.08.2017 14:03, Octavian Ciobanu wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Hey Vladislav,<br>
<br>
Thank you for the info. I've tried you suggestions but the behavior is<br>
still the same. When an offline/standby node rejoins the cluster all the<br>
resources are first stopped and then started. I've added the changes<br>
I've made, see below in reply message, next to your suggestions.<br>
</blockquote>
<br></span>
Logs on DC (node where you see logs from the pengine process) should contain references to pe-input-XX.bz2 files. Something like "notice: Calculated transition XXXX, saving inputs in /var/lib/pacemaker/pengine/pe-<wbr>input-XX.bz2"<br>
Locate one for which Stop actions occur.<br>
You can replay them with 'crm_simulate -S -x /var/lib/pacemaker/pengine/pe-<wbr>input-XX.bz2' to see if that is the correct one (look in the middle of output).<br>
<br>
After that you may add some debugging:<br>
PCMK_debug=yes PCMK_logfile=./pcmk.log crm_simulate -S -x /var/lib/pacemaker/pengine/pe-<wbr>input-XX.bz2<br>
<br>
That will produce a big file with all debugging messages enabled.<br>
<br>
Try to locate a reason for restarts there.<br>
<br>
Best,<br>
Vladislav<br>
<br>
Also please look inline (may be info there will be enough so you won't need to debug).<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>
<br>
Once again thank you for info.<br>
<br>
Best regards.<br>
Octavian Ciobanu<br>
<br>
On Sat, Aug 26, 2017 at 8:17 PM, Vladislav Bogdanov<br></span>
<<a href="mailto:bubble@hoster-ok.com" target="_blank">bubble@hoster-ok.com</a> <mailto:<a href="mailto:bubble@hoster-ok.com" target="_blank">bubble@hoster-ok.com</a>>> wrote:<span><br>
<br>
<a href="tel:26.08.2017%2019" value="+12608201719" target="_blank">26.08.2017 19</a> <tel:26.08.2017%2019>:36, Octavian Ciobanu wrote:<br>
<br>
Thank you for your reply.<br>
<br>
There is no reason to set location for the resources, I think,<br>
because<br>
all the resources are set with clone options so they are started<br>
on all<br>
nodes at the same time.<br>
<br>
<br>
You still need to colocate "upper" resources with their<br>
dependencies. Otherwise pacemaker will try to start them even if<br>
their dependencies fail. Order without colocation has very limited<br>
use (usually when resources may run on different nodes). For clones<br>
that is even more exotic.<br>
<br>
<br>
I've added collocation<br>
<br>
pcs constraint colocation add iSCSI1-clone with DLM-clone<br>
pcs constraint colocation add iSCSI2-clone with DLM-clone<br>
pcs constraint colocation add iSCSI3-clone with DLM-clone<br>
pcs constraint colocation add Mount1-clone with iSCSI1-clone<br>
pcs constraint colocation add Mount2-clone with iSCSI2-clone<br>
pcs constraint colocation add Mount4-clone with iSCSI3-clone<br>
<br>
The result is the same ... all clones are first stopped and then started<br>
beginning with DLM resource and ending with the Mount ones.<br>
</span></blockquote>
<br>
Yep, that was not meant to fix your problem. Just to prevent future issues.<span><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<br>
For you original question: ensure you have interleave=true set for<br>
all your clones. You seem to miss it for iSCSI ones.<br>
interleave=false (default) is for different uses (when upper<br>
resources require all clone instances to be up).<br>
<br>
<br>
Modified iSCSI resources and added interleave="true" and still no change<br>
in behavior.<br>
</blockquote>
<br></span>
Weird... Probably you also do not need 'ordered="true"' for your DLM clone? Knowing what is DLM, it does not need ordering, its instances may be safely started in the parallel.<span><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<br>
Also, just a minor note, iSCSI resources do not actually depend on<br>
dlm, mounts should depend on it.<br>
<br>
<br>
I know but the mount resource must know when the iSCSI resource to whom<br>
is connected is started so the only solution I've seen was to place DLM<br>
before iSCSI and then Mount. If there is another solution, a proper way<br>
to do it, please can you give a reference or a place from where to read<br>
on how to do it ?<br>
</blockquote>
<br></span>
You would want to colocate (and order) mount with both DLM and iSCSI. Multiple colocations/orders for the same resource are allowed.<br>
For mount you need DLM running and iSCSI disk connected. But you actually do not need DLM to connect iSCSI disk (so DLM and iSCSI resources may start in the parallel).<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span>
<br>
<br>
And when it comes to stickiness I forgot to<br>
mention that but it set to 200. and also I have stonith<br>
configured to<br>
use vmware esxi.<br>
<br>
Best regards<br>
Octavian Ciobanu<br>
<br>
On Sat, Aug 26, 2017 at 6:16 PM, John Keates <<a href="mailto:john@keates.nl" target="_blank">john@keates.nl</a><br>
<mailto:<a href="mailto:john@keates.nl" target="_blank">john@keates.nl</a>><br></span><span>
<mailto:<a href="mailto:john@keates.nl" target="_blank">john@keates.nl</a> <mailto:<a href="mailto:john@keates.nl" target="_blank">john@keates.nl</a>>>> wrote:<br>
<br>
While I am by no means a CRM/Pacemaker expert, I only see the<br>
resource primitives and the order constraints. Wouldn’t you need<br>
location and/or colocation as well as stickiness settings to<br>
prevent<br>
this from happening? What I think it might be doing is<br>
seeing the<br>
new node, then trying to move the resources (but not finding<br>
it a<br>
suitable target) and then moving them back where they came<br>
from, but<br>
fast enough for you to only see it as a restart.<br>
<br>
If you crm_resource -P, it should also restart all<br>
resources, but<br>
put them in the preferred spot. If they end up in the same<br>
place,<br>
you probably didn’t put and weighing in the config or have<br>
stickiness set to INF.<br>
<br>
Kind regards,<br>
<br>
John Keates<br>
<br>
On 26 Aug 2017, at 14:23, Octavian Ciobanu<br>
<<a href="mailto:coctavian1979@gmail.com" target="_blank">coctavian1979@gmail.com</a><br>
<mailto:<a href="mailto:coctavian1979@gmail.com" target="_blank">coctavian1979@gmail.co<wbr>m</a>><br></span>
<mailto:<a href="mailto:coctavian1979@gmail.com" target="_blank">coctavian1979@gmail.co<wbr>m</a><span><br>
<mailto:<a href="mailto:coctavian1979@gmail.com" target="_blank">coctavian1979@gmail.co<wbr>m</a>>>> wrote:<br>
<br>
Hello all,<br>
<br>
While playing with cluster configuration I noticed a strange<br>
behavior. If I stop/standby cluster services on one node and<br>
reboot it, when it joins the cluster all the resources<br>
that were<br>
started and working on active nodes get stopped and<br>
restarted.<br>
<br>
My testing configuration is based on 4 nodes. One node is a<br>
storage node that makes 3 iSCSI targets available for<br>
the other<br>
nodes to use,it is not configured to join cluster, and<br>
three nodes<br>
that are configured in a cluster using the following<br>
commands.<br>
<br>
pcs resource create DLM ocf:pacemaker:controld op monitor<br>
interval="60" on-fail="fence" clone meta clone-max="3"<br>
clone-node-max="1" interleave="true" ordered="true"<br>
pcs resource create iSCSI1 ocf:heartbeat:iscsi<br></span>
portal="<a href="http://10.0.0.1:3260" rel="noreferrer" target="_blank">10.0.0.1:3260</a> <<a href="http://10.0.0.1:3260" rel="noreferrer" target="_blank">http://10.0.0.1:3260</a>><span><br>
<<a href="http://10.0.0.1:3260/" rel="noreferrer" target="_blank">http://10.0.0.1:3260/</a>>"<br>
target="<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">iqn.2017-08.example.co<wbr>m</a><br>
<<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a>><br></span>
<<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a><span><br>
<<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a>>>:tgt1" op start interval="0"<br>
timeout="20" op stop interval="0" timeout="20" op monitor<br>
interval="120" timeout="30" clone meta clone-max="3"<br>
clone-node-max="1"<br>
pcs resource create iSCSI2 ocf:heartbeat:iscsi<br></span>
portal="<a href="http://10.0.0.1:3260" rel="noreferrer" target="_blank">10.0.0.1:3260</a> <<a href="http://10.0.0.1:3260" rel="noreferrer" target="_blank">http://10.0.0.1:3260</a>><span><br>
<<a href="http://10.0.0.1:3260/" rel="noreferrer" target="_blank">http://10.0.0.1:3260/</a>>"<br>
target="<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">iqn.2017-08.example.co<wbr>m</a><br>
<<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a>><br></span>
<<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a><span><br>
<<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a>>>:tgt2" op start interval="0"<br>
timeout="20" op stop interval="0" timeout="20" op monitor<br>
interval="120" timeout="30" clone meta clone-max="3"<br>
clone-node-max="1"<br>
pcs resource create iSCSI3 ocf:heartbeat:iscsi<br></span>
portal="<a href="http://10.0.0.1:3260" rel="noreferrer" target="_blank">10.0.0.1:3260</a> <<a href="http://10.0.0.1:3260" rel="noreferrer" target="_blank">http://10.0.0.1:3260</a>><span><br>
<<a href="http://10.0.0.1:3260/" rel="noreferrer" target="_blank">http://10.0.0.1:3260/</a>>"<br>
target="<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">iqn.2017-08.example.co<wbr>m</a><br>
<<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a>><br></span>
<<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a><div><div class="m_3293396309552553063h5"><br>
<<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a>>>:tgt3" op start interval="0"<br>
<br>
timeout="20" op stop interval="0" timeout="20" op monitor<br>
interval="120" timeout="30" clone meta clone-max="3"<br>
clone-node-max="1"<br>
pcs resource create Mount1 ocf:heartbeat:Filesystem<br>
device="/dev/disk/by-label/MyC<wbr>luster:Data1"<br>
directory="/mnt/data1"<br>
fstype="gfs2" options="noatime,nodiratime,rw<wbr>" op monitor<br>
interval="90" on-fail="fence" clone meta clone-max="3"<br>
clone-node-max="1" interleave="true"<br>
pcs resource create Mount2 ocf:heartbeat:Filesystem<br>
device="/dev/disk/by-label/MyC<wbr>luster:Data2"<br>
directory="/mnt/data2"<br>
fstype="gfs2" options="noatime,nodiratime,rw<wbr>" op monitor<br>
interval="90" on-fail="fence" clone meta clone-max="3"<br>
clone-node-max="1" interleave="true"<br>
pcs resource create Mount3 ocf:heartbeat:Filesystem<br>
device="/dev/disk/by-label/MyC<wbr>luster:Data3"<br>
directory="/mnt/data3"<br>
fstype="gfs2" options="noatime,nodiratime,rw<wbr>" op monitor<br>
interval="90" on-fail="fence" clone meta clone-max="3"<br>
clone-node-max="1" interleave="true"<br>
pcs constraint order DLM-clone then iSCSI1-clone<br>
pcs constraint order DLM-clone then iSCSI2-clone<br>
pcs constraint order DLM-clone then iSCSI3-clone<br>
pcs constraint order iSCSI1-clone then Mount1-clone<br>
pcs constraint order iSCSI2-clone then Mount2-clone<br>
pcs constraint order iSCSI3-clone then Mount3-clone<br>
<br>
If I issue the command "pcs cluster standby node1" or<br>
"pcs cluster<br>
stop" on node 1 and after that I reboot the node. When<br>
the node<br>
gets back online (unstandby if it was put in standby<br>
mode) all the<br>
"MountX" resources get stopped on node 3 and 4 and<br>
started again.<br>
<br>
Can anyone help me figure out where and what is the<br>
mistake in my<br>
configuration as I would like to keep the started<br>
resources on<br>
active nodes (avoid stop and start of resources)?<br>
<br>
Thank you in advance<br>
Octavian Ciobanu<br>
______________________________<wbr>_________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
<mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a>><br></div></div>
<mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><span><br>
<mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a>><wbr>><br>
<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/m<wbr>ailman/listinfo/users</a><br>
<<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a>><br>
<<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
<<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a>>><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started:<br>
<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc<wbr>/Cluster_from_Scratch.pdf</a><br>
<<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a>><br>
<<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a><br>
<<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a>>><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
<br>
<br>
<br>
______________________________<wbr>_________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br></span>
<mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a>> <mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><div><div class="m_3293396309552553063h5"><br>
<mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a>><wbr>><br>
<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/m<wbr>ailman/listinfo/users</a><br>
<<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a>><br>
<<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
<<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a>>><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started:<br>
<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc<wbr>/Cluster_from_Scratch.pdf</a><br>
<<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a>><br>
<<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a><br>
<<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a>>><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
<br>
<br>
<br>
<br>
______________________________<wbr>_________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
<mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a>><br>
<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/m<wbr>ailman/listinfo/users</a><br>
<<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a>><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started:<br>
<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc<wbr>/Cluster_from_Scratch.pdf</a><br>
<<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a>><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
<br>
<br>
<br>
______________________________<wbr>_________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a> <mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a>><br>
<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/m<wbr>ailman/listinfo/users</a><br>
<<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a>><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started:<br>
<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc<wbr>/Cluster_from_Scratch.pdf</a><br>
<<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a>><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
<br>
<br>
<br>
<br>
______________________________<wbr>_________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/m<wbr>ailman/listinfo/users</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc<wbr>/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
<br>
</div></div></blockquote><div class="m_3293396309552553063HOEnZb"><div class="m_3293396309552553063h5">
<br>
<br>
______________________________<wbr>_________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/m<wbr>ailman/listinfo/users</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc<wbr>/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>