<div dir="ltr"><div>Yes.<br><br></div>I've removed ordered="true" from DLM resource and also added collocation constraints for Mount clones with both DLM and iSCSI.<br><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Aug 31, 2017 at 3:34 PM, Vladislav Bogdanov <span dir="ltr"><<a href="mailto:bubble@hoster-ok.com" target="_blank">bubble@hoster-ok.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class=""><a href="tel:31.08.2017%2014" value="+13108201714" target="_blank">31.08.2017 14</a>:53, Octavian Ciobanu wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I'm back with confirmation that DLM is the one that triggers Mount<br>
resources to stop when the stopped/suspended node rejoins<br>
<br>
When the DLM resource is started, on the rejoining node, it tries to get<br>
a free journal but always gets one that is already occupied by another<br>
node and triggers a domino effect, one node jumps to the next node<br>
occupied dlm journal, stopping and restarting the mount resources.<br>
<br>
I have plenty of DLM journals (allocated 10 for 3 node configuration) so<br>
for sure there are unused ones available.<br>
<br>
There is a way to make DLM keep its journal when the node is stopped and<br>
use it when it starts ? or a way to make it keep it's active allocation<br>
forcing the rejoining node to look for another unoccupied journal<br>
without pushing away the nodes that occupy the journal it allocates for<br>
the node that rejoins?<br>
</blockquote>
<br></span>
Did you try my suggestion from the previous e-mail?<br>
<br>
'Probably you also do not need 'ordered="true"' for your DLM clone? Knowing what is DLM, it does not need ordering, its instances may be safely started in the parallel.'<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
<br>
Best regards<br>
<br>
On Mon, Aug 28, 2017 at 6:04 PM, Octavian Ciobanu<br></span><div><div class="h5">
<<a href="mailto:coctavian1979@gmail.com" target="_blank">coctavian1979@gmail.com</a> <mailto:<a href="mailto:coctavian1979@gmail.com" target="_blank">coctavian1979@gmail.co<wbr>m</a>>> wrote:<br>
<br>
Thank you for info.<br>
Looking over the output of the crm_simulate I've noticed the<br>
"notice" messages and with the help of the debug mode I've found<br>
this sequence in log<br>
<br>
Aug 28 16:23:19 [13802] node03 crm_simulate: debug:<br>
native_assign_node: Assigning node01 to DLM:2<br>
Aug 28 16:23:19 [13802] node03 crm_simulate: notice:<br>
color_instance: Pre-allocation failed: got node01 instead of node02<br>
Aug 28 16:23:19 [13802] node03 crm_simulate: info:<br>
native_deallocate: Deallocating DLM:2 from core01<br>
Aug 28 16:23:19 [13802] node03 crm_simulate: debug:<br>
native_assign_node: Assigning core01 to DLM:3<br>
Aug 28 16:23:19 [13802] node03 crm_simulate: notice:<br>
color_instance: Pre-allocation failed: got node01 instead of node03<br>
Aug 28 16:23:19 [13802] node03 crm_simulate: info:<br>
native_deallocate: Deallocating DLM:3 from core01<br>
Aug 28 16:23:19 [13802] node03 crm_simulate: debug:<br>
native_assign_node: Assigning core01 to DLM:0<br>
Aug 28 16:23:19 [13802] node03 crm_simulate: info:<br>
rsc_merge_weights: DLM:2: Rolling back scores from iSCSI2-clone<br>
Aug 28 16:23:19 [13802] node03 crm_simulate: info:<br>
rsc_merge_weights: DLM:2: Rolling back scores from iSCSI2-clone<br>
Aug 28 16:23:19 [13802] node03 crm_simulate: debug:<br>
native_assign_node: Assigning node03 to DLM:2<br>
Aug 28 16:23:19 [13802] node03 crm_simulate: info:<br>
rsc_merge_weights: DLM:3: Rolling back scores from iSCSI2-clone<br>
Aug 28 16:23:19 [13802] node03 crm_simulate: info:<br>
rsc_merge_weights: DLM:3: Rolling back scores from iSCSI2-clone<br>
Aug 28 16:23:19 [13802] node03 crm_simulate: debug:<br>
native_assign_node: Assigning node02 to DLM:3<br>
<br>
This suggests that the restarted node attempts to occupy a dlm<br>
journal that is allocated to another note and by doing so triggers a<br>
chain reaction leading to all resources being restarted on all nodes.<br>
<br>
I will try a different approach (based on your suggestions) on<br>
starting DLM, iSCSI and Mount resources and see if this changes<br>
anything.<br>
<br>
If based on the log you have any suggestions they are welcome.<br>
<br>
Thank you again for the help.<br>
<br>
On Mon, Aug 28, 2017 at 3:53 PM, Vladislav Bogdanov<br></div></div><div><div class="h5">
<<a href="mailto:bubble@hoster-ok.com" target="_blank">bubble@hoster-ok.com</a> <mailto:<a href="mailto:bubble@hoster-ok.com" target="_blank">bubble@hoster-ok.com</a>>> wrote:<br>
<br>
28.08.2017 14:03, Octavian Ciobanu wrote:<br>
<br>
Hey Vladislav,<br>
<br>
Thank you for the info. I've tried you suggestions but the<br>
behavior is<br>
still the same. When an offline/standby node rejoins the<br>
cluster all the<br>
resources are first stopped and then started. I've added the<br>
changes<br>
I've made, see below in reply message, next to your suggestions.<br>
<br>
<br>
Logs on DC (node where you see logs from the pengine process)<br>
should contain references to pe-input-XX.bz2 files. Something<br>
like "notice: Calculated transition XXXX, saving inputs in<br>
/var/lib/pacemaker/pengine/pe-<wbr>input-XX.bz2"<br>
Locate one for which Stop actions occur.<br>
You can replay them with 'crm_simulate -S -x<br>
/var/lib/pacemaker/pengine/pe-<wbr>input-XX.bz2' to see if that is<br>
the correct one (look in the middle of output).<br>
<br>
After that you may add some debugging:<br>
PCMK_debug=yes PCMK_logfile=./pcmk.log crm_simulate -S -x<br>
/var/lib/pacemaker/pengine/pe-<wbr>input-XX.bz2<br>
<br>
That will produce a big file with all debugging messages enabled.<br>
<br>
Try to locate a reason for restarts there.<br>
<br>
Best,<br>
Vladislav<br>
<br>
Also please look inline (may be info there will be enough so you<br>
won't need to debug).<br>
<br>
<br>
Once again thank you for info.<br>
<br>
Best regards.<br>
Octavian Ciobanu<br>
<br>
On Sat, Aug 26, 2017 at 8:17 PM, Vladislav Bogdanov<br>
<<a href="mailto:bubble@hoster-ok.com" target="_blank">bubble@hoster-ok.com</a> <mailto:<a href="mailto:bubble@hoster-ok.com" target="_blank">bubble@hoster-ok.com</a>><br></div></div>
<mailto:<a href="mailto:bubble@hoster-ok.com" target="_blank">bubble@hoster-ok.com</a> <mailto:<a href="mailto:bubble@hoster-ok.com" target="_blank">bubble@hoster-ok.com</a>>><wbr>><div><div class="h5"><br>
wrote:<br>
<br>
<a href="tel:26.08.2017%2019" value="+12608201719" target="_blank">26.08.2017 19</a> <tel:26.08.2017%2019><br>
<tel:26.08.2017%2019>:36, Octavian Ciobanu wrote:<br>
<br>
Thank you for your reply.<br>
<br>
There is no reason to set location for the<br>
resources, I think,<br>
because<br>
all the resources are set with clone options so they<br>
are started<br>
on all<br>
nodes at the same time.<br>
<br>
<br>
You still need to colocate "upper" resources with their<br>
dependencies. Otherwise pacemaker will try to start them<br>
even if<br>
their dependencies fail. Order without colocation has<br>
very limited<br>
use (usually when resources may run on different nodes).<br>
For clones<br>
that is even more exotic.<br>
<br>
<br>
I've added collocation<br>
<br>
pcs constraint colocation add iSCSI1-clone with DLM-clone<br>
pcs constraint colocation add iSCSI2-clone with DLM-clone<br>
pcs constraint colocation add iSCSI3-clone with DLM-clone<br>
pcs constraint colocation add Mount1-clone with iSCSI1-clone<br>
pcs constraint colocation add Mount2-clone with iSCSI2-clone<br>
pcs constraint colocation add Mount4-clone with iSCSI3-clone<br>
<br>
The result is the same ... all clones are first stopped and<br>
then started<br>
beginning with DLM resource and ending with the Mount ones.<br>
<br>
<br>
Yep, that was not meant to fix your problem. Just to prevent<br>
future issues.<br>
<br>
<br>
<br>
For you original question: ensure you have<br>
interleave=true set for<br>
all your clones. You seem to miss it for iSCSI ones.<br>
interleave=false (default) is for different uses (when upper<br>
resources require all clone instances to be up).<br>
<br>
<br>
Modified iSCSI resources and added interleave="true" and<br>
still no change<br>
in behavior.<br>
<br>
<br>
Weird... Probably you also do not need 'ordered="true"' for your<br>
DLM clone? Knowing what is DLM, it does not need ordering, its<br>
instances may be safely started in the parallel.<br>
<br>
<br>
<br>
Also, just a minor note, iSCSI resources do not actually<br>
depend on<br>
dlm, mounts should depend on it.<br>
<br>
<br>
I know but the mount resource must know when the iSCSI<br>
resource to whom<br>
is connected is started so the only solution I've seen was<br>
to place DLM<br>
before iSCSI and then Mount. If there is another solution, a<br>
proper way<br>
to do it, please can you give a reference or a place from<br>
where to read<br>
on how to do it ?<br>
<br>
<br>
You would want to colocate (and order) mount with both DLM and<br>
iSCSI. Multiple colocations/orders for the same resource are<br>
allowed.<br>
For mount you need DLM running and iSCSI disk connected. But you<br>
actually do not need DLM to connect iSCSI disk (so DLM and iSCSI<br>
resources may start in the parallel).<br>
<br>
<br>
<br>
And when it comes to stickiness I forgot to<br>
mention that but it set to 200. and also I have stonith<br>
configured to<br>
use vmware esxi.<br>
<br>
Best regards<br>
Octavian Ciobanu<br>
<br>
On Sat, Aug 26, 2017 at 6:16 PM, John Keates<br>
<<a href="mailto:john@keates.nl" target="_blank">john@keates.nl</a> <mailto:<a href="mailto:john@keates.nl" target="_blank">john@keates.nl</a>><br>
<mailto:<a href="mailto:john@keates.nl" target="_blank">john@keates.nl</a> <mailto:<a href="mailto:john@keates.nl" target="_blank">john@keates.nl</a>>><br>
<mailto:<a href="mailto:john@keates.nl" target="_blank">john@keates.nl</a> <mailto:<a href="mailto:john@keates.nl" target="_blank">john@keates.nl</a>><br>
<mailto:<a href="mailto:john@keates.nl" target="_blank">john@keates.nl</a> <mailto:<a href="mailto:john@keates.nl" target="_blank">john@keates.nl</a>>>>> wrote:<br>
<br>
While I am by no means a CRM/Pacemaker expert, I<br>
only see the<br>
resource primitives and the order constraints.<br>
Wouldn’t you need<br>
location and/or colocation as well as stickiness<br>
settings to<br>
prevent<br>
this from happening? What I think it might be<br>
doing is<br>
seeing the<br>
new node, then trying to move the resources (but<br>
not finding<br>
it a<br>
suitable target) and then moving them back where<br>
they came<br>
from, but<br>
fast enough for you to only see it as a restart.<br>
<br>
If you crm_resource -P, it should also restart all<br>
resources, but<br>
put them in the preferred spot. If they end up<br>
in the same<br>
place,<br>
you probably didn’t put and weighing in the<br>
config or have<br>
stickiness set to INF.<br>
<br>
Kind regards,<br>
<br>
John Keates<br>
<br>
On 26 Aug 2017, at 14:23, Octavian Ciobanu<br>
<<a href="mailto:coctavian1979@gmail.com" target="_blank">coctavian1979@gmail.com</a><br>
<mailto:<a href="mailto:coctavian1979@gmail.com" target="_blank">coctavian1979@gmail.co<wbr>m</a>><br>
<mailto:<a href="mailto:coctavian1979@gmail.com" target="_blank">coctavian1979@gmail.co<wbr>m</a><br>
<mailto:<a href="mailto:coctavian1979@gmail.com" target="_blank">coctavian1979@gmail.co<wbr>m</a>>><br>
<mailto:<a href="mailto:coctavian1979@gmail.com" target="_blank">coctavian1979@gmail.co<wbr>m</a><br>
<mailto:<a href="mailto:coctavian1979@gmail.com" target="_blank">coctavian1979@gmail.co<wbr>m</a>><br>
<mailto:<a href="mailto:coctavian1979@gmail.com" target="_blank">coctavian1979@gmail.co<wbr>m</a><br>
<mailto:<a href="mailto:coctavian1979@gmail.com" target="_blank">coctavian1979@gmail.co<wbr>m</a>>>>> wrote:<br>
<br>
Hello all,<br>
<br>
While playing with cluster configuration I<br>
noticed a strange<br>
behavior. If I stop/standby cluster services<br>
on one node and<br>
reboot it, when it joins the cluster all the<br>
resources<br>
that were<br>
started and working on active nodes get<br>
stopped and<br>
restarted.<br>
<br>
My testing configuration is based on 4<br>
nodes. One node is a<br>
storage node that makes 3 iSCSI targets<br>
available for<br>
the other<br>
nodes to use,it is not configured to join<br>
cluster, and<br>
three nodes<br>
that are configured in a cluster using the<br>
following<br>
commands.<br>
<br>
pcs resource create DLM<br>
ocf:pacemaker:controld op monitor<br>
interval="60" on-fail="fence" clone meta<br>
clone-max="3"<br>
clone-node-max="1" interleave="true"<br>
ordered="true"<br>
pcs resource create iSCSI1 ocf:heartbeat:iscsi<br>
portal="<a href="http://10.0.0.1:3260" rel="noreferrer" target="_blank">10.0.0.1:3260</a> <<a href="http://10.0.0.1:3260" rel="noreferrer" target="_blank">http://10.0.0.1:3260</a>><br>
<<a href="http://10.0.0.1:3260" rel="noreferrer" target="_blank">http://10.0.0.1:3260</a>><br>
<<a href="http://10.0.0.1:3260/" rel="noreferrer" target="_blank">http://10.0.0.1:3260/</a>>"<br>
target="<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">iqn.2017-08.example.co<wbr>m</a><br>
<<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a>><br>
<<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a><br>
<<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a>>><br>
<<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a><br>
<<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a>><br>
<<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a><br>
<<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a>>>>:tgt1" op start interval="0"<br>
timeout="20" op stop interval="0"<br>
timeout="20" op monitor<br>
interval="120" timeout="30" clone meta<br>
clone-max="3"<br>
clone-node-max="1"<br>
pcs resource create iSCSI2 ocf:heartbeat:iscsi<br>
portal="<a href="http://10.0.0.1:3260" rel="noreferrer" target="_blank">10.0.0.1:3260</a> <<a href="http://10.0.0.1:3260" rel="noreferrer" target="_blank">http://10.0.0.1:3260</a>><br>
<<a href="http://10.0.0.1:3260" rel="noreferrer" target="_blank">http://10.0.0.1:3260</a>><br>
<<a href="http://10.0.0.1:3260/" rel="noreferrer" target="_blank">http://10.0.0.1:3260/</a>>"<br>
target="<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">iqn.2017-08.example.co<wbr>m</a><br>
<<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a>><br>
<<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a><br>
<<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a>>><br>
<<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a><br>
<<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a>><br>
<<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a><br></div></div><span class="">
<<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a>>>>:tgt2" op start interval="0"<br>
timeout="20" op stop interval="0"<br>
timeout="20" op monitor<br>
interval="120" timeout="30" clone meta<br>
clone-max="3"<br>
clone-node-max="1"<br>
pcs resource create iSCSI3 ocf:heartbeat:iscsi<br>
portal="<a href="http://10.0.0.1:3260" rel="noreferrer" target="_blank">10.0.0.1:3260</a> <<a href="http://10.0.0.1:3260" rel="noreferrer" target="_blank">http://10.0.0.1:3260</a>><br></span><span class="">
<<a href="http://10.0.0.1:3260" rel="noreferrer" target="_blank">http://10.0.0.1:3260</a>><br>
<<a href="http://10.0.0.1:3260/" rel="noreferrer" target="_blank">http://10.0.0.1:3260/</a>>"<br>
target="<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">iqn.2017-08.example.co<wbr>m</a><br>
<<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a>><br>
<<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a><br>
<<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a>>><br>
<<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a><br>
<<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a>><br>
<br>
<<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a><br></span><div><div class="h5">
<<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a>>>>:tgt3" op start interval="0"<br>
<br>
timeout="20" op stop interval="0"<br>
timeout="20" op monitor<br>
interval="120" timeout="30" clone meta<br>
clone-max="3"<br>
clone-node-max="1"<br>
pcs resource create Mount1<br>
ocf:heartbeat:Filesystem<br>
device="/dev/disk/by-label/MyC<wbr>luster:Data1"<br>
directory="/mnt/data1"<br>
fstype="gfs2"<br>
options="noatime,nodiratime,rw<wbr>" op monitor<br>
interval="90" on-fail="fence" clone meta<br>
clone-max="3"<br>
clone-node-max="1" interleave="true"<br>
pcs resource create Mount2<br>
ocf:heartbeat:Filesystem<br>
device="/dev/disk/by-label/MyC<wbr>luster:Data2"<br>
directory="/mnt/data2"<br>
fstype="gfs2"<br>
options="noatime,nodiratime,rw<wbr>" op monitor<br>
interval="90" on-fail="fence" clone meta<br>
clone-max="3"<br>
clone-node-max="1" interleave="true"<br>
pcs resource create Mount3<br>
ocf:heartbeat:Filesystem<br>
device="/dev/disk/by-label/MyC<wbr>luster:Data3"<br>
directory="/mnt/data3"<br>
fstype="gfs2"<br>
options="noatime,nodiratime,rw<wbr>" op monitor<br>
interval="90" on-fail="fence" clone meta<br>
clone-max="3"<br>
clone-node-max="1" interleave="true"<br>
pcs constraint order DLM-clone then iSCSI1-clone<br>
pcs constraint order DLM-clone then iSCSI2-clone<br>
pcs constraint order DLM-clone then iSCSI3-clone<br>
pcs constraint order iSCSI1-clone then<br>
Mount1-clone<br>
pcs constraint order iSCSI2-clone then<br>
Mount2-clone<br>
pcs constraint order iSCSI3-clone then<br>
Mount3-clone<br>
<br>
If I issue the command "pcs cluster standby<br>
node1" or<br>
"pcs cluster<br>
stop" on node 1 and after that I reboot the<br>
node. When<br>
the node<br>
gets back online (unstandby if it was put in<br>
standby<br>
mode) all the<br>
"MountX" resources get stopped on node 3 and<br>
4 and<br>
started again.<br>
<br>
Can anyone help me figure out where and what<br>
is the<br>
mistake in my<br>
configuration as I would like to keep the<br>
started<br>
resources on<br>
active nodes (avoid stop and start of<br>
resources)?<br>
<br>
Thank you in advance<br>
Octavian Ciobanu<br>
______________________________<wbr>_________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
<mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a>><br>
<mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
<mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a>><wbr>><br>
<mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
<mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a>><br>
<mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
<mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a>><wbr>>><br>
<br>
<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/m<wbr>ailman/listinfo/users</a><br>
<<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a>><br>
<br>
<<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
<<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a>>><br>
<br>
<<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
<<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a>><br>
<br>
<<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
<<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a>>>><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started:<br>
<br>
<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc<wbr>/Cluster_from_Scratch.pdf</a><br>
<<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a>><br>
<br>
<<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a><br>
<<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a>>><br>
<br>
<<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a><br>
<<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a>><br>
<br>
<<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a><br>
<<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a>>>><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
<br>
<br>
<br>
______________________________<wbr>_________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
<mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a>><br>
<mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
<mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a>><wbr>><br>
<mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a> <mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a>><br>
<br>
<mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
<mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a>><wbr>>><br>
<br>
<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/m<wbr>ailman/listinfo/users</a><br>
<<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a>><br>
<<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
<<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a>>><br>
<br>
<<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
<<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a>><br>
<<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
<<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a>>>><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started:<br>
<br>
<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc<wbr>/Cluster_from_Scratch.pdf</a><br>
<<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a>><br>
<br>
<<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a><br>
<<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a>>><br>
<br>
<<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a><br>
<<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a>><br>
<br>
<<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a><br>
<<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a>>>><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
<br>
<br>
<br>
<br>
______________________________<wbr>_________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
<mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a>><br>
<mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
<mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a>><wbr>><br>
<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/m<wbr>ailman/listinfo/users</a><br>
<<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a>><br>
<<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
<<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a>>><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started:<br>
<br>
<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc<wbr>/Cluster_from_Scratch.pdf</a><br>
<<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a>><br>
<br>
<<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a><br>
<<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a>>><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
<br>
<br>
<br>
______________________________<wbr>_________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
<mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a>> <mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
<mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a>><wbr>><br>
<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/m<wbr>ailman/listinfo/users</a><br>
<<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a>><br>
<<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
<<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a>>><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started:<br>
<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc<wbr>/Cluster_from_Scratch.pdf</a><br>
<<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a>><br>
<<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a><br>
<<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a>>><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
<br>
<br>
<br>
<br>
______________________________<wbr>_________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
<mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a>><br>
<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/m<wbr>ailman/listinfo/users</a><br>
<<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a>><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started:<br>
<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc<wbr>/Cluster_from_Scratch.pdf</a><br>
<<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a>><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
<br>
<br>
<br>
______________________________<wbr>_________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
<mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a>><br>
<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/m<wbr>ailman/listinfo/users</a><br>
<<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a>><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started:<br>
<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc<wbr>/Cluster_from_Scratch.pdf</a><br>
<<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a>><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
<br>
<br>
<br>
<br>
<br>
______________________________<wbr>_________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/m<wbr>ailman/listinfo/users</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc<wbr>/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
<br>
</div></div></blockquote><div class="HOEnZb"><div class="h5">
<br>
<br>
______________________________<wbr>_________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/m<wbr>ailman/listinfo/users</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc<wbr>/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
</div></div></blockquote></div><br></div>