<div dir="ltr"><div>Yes.<br><br></div>I've removed ordered="true" from DLM resource and also added collocation constraints for Mount clones with both DLM and iSCSI.<br><br></div><div class="gmail_extra"><br><div class="gmail_quote">On Thu, Aug 31, 2017 at 3:34 PM, Vladislav Bogdanov <span dir="ltr"><<a href="mailto:bubble@hoster-ok.com" target="_blank">bubble@hoster-ok.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class=""><a href="tel:31.08.2017%2014" value="+13108201714" target="_blank">31.08.2017 14</a>:53, Octavian Ciobanu wrote:<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
I'm back with confirmation that DLM is the one that triggers Mount<br>
resources to stop when the stopped/suspended node rejoins<br>
<br>
When the DLM resource is started, on the rejoining node, it tries to get<br>
a free journal but always gets one that is already occupied by another<br>
node and triggers a domino effect, one node jumps to the next node<br>
occupied dlm journal, stopping and restarting the mount resources.<br>
<br>
I have plenty of DLM journals (allocated 10 for 3 node configuration) so<br>
for sure there are unused ones available.<br>
<br>
There is a way to make DLM keep its journal when the node is stopped and<br>
use it when it starts ? or a way to make it keep it's active allocation<br>
forcing the rejoining node to look for another unoccupied journal<br>
without pushing away the nodes that occupy the journal it allocates for<br>
the node that rejoins?<br>
</blockquote>
<br></span>
Did you try my suggestion from the previous e-mail?<br>
<br>
'Probably you also do not need 'ordered="true"' for your DLM clone? Knowing what is DLM, it does not need ordering, its instances may be safely started in the parallel.'<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">
<br>
Best regards<br>
<br>
On Mon, Aug 28, 2017 at 6:04 PM, Octavian Ciobanu<br></span><div><div class="h5">
<<a href="mailto:coctavian1979@gmail.com" target="_blank">coctavian1979@gmail.com</a> <mailto:<a href="mailto:coctavian1979@gmail.com" target="_blank">coctavian1979@gmail.co<wbr>m</a>>> wrote:<br>
<br>
    Thank you for info.<br>
    Looking over the output of the crm_simulate I've noticed the<br>
    "notice" messages and with the help of the debug mode I've found<br>
    this sequence in log<br>
<br>
    Aug 28 16:23:19 [13802] node03 crm_simulate:    debug:<br>
    native_assign_node:      Assigning node01 to DLM:2<br>
    Aug 28 16:23:19 [13802] node03 crm_simulate:   notice:<br>
    color_instance:  Pre-allocation failed: got node01 instead of node02<br>
    Aug 28 16:23:19 [13802] node03 crm_simulate:     info:<br>
    native_deallocate:       Deallocating DLM:2 from core01<br>
    Aug 28 16:23:19 [13802] node03 crm_simulate:    debug:<br>
    native_assign_node:      Assigning core01 to DLM:3<br>
    Aug 28 16:23:19 [13802] node03 crm_simulate:   notice:<br>
    color_instance:  Pre-allocation failed: got node01 instead of node03<br>
    Aug 28 16:23:19 [13802] node03 crm_simulate:     info:<br>
    native_deallocate:       Deallocating DLM:3 from core01<br>
    Aug 28 16:23:19 [13802] node03 crm_simulate:    debug:<br>
    native_assign_node:      Assigning core01 to DLM:0<br>
    Aug 28 16:23:19 [13802] node03 crm_simulate:     info:<br>
    rsc_merge_weights:       DLM:2: Rolling back scores from iSCSI2-clone<br>
    Aug 28 16:23:19 [13802] node03 crm_simulate:     info:<br>
    rsc_merge_weights:       DLM:2: Rolling back scores from iSCSI2-clone<br>
    Aug 28 16:23:19 [13802] node03 crm_simulate:    debug:<br>
    native_assign_node:      Assigning node03 to DLM:2<br>
    Aug 28 16:23:19 [13802] node03 crm_simulate:     info:<br>
    rsc_merge_weights:       DLM:3: Rolling back scores from iSCSI2-clone<br>
    Aug 28 16:23:19 [13802] node03 crm_simulate:     info:<br>
    rsc_merge_weights:       DLM:3: Rolling back scores from iSCSI2-clone<br>
    Aug 28 16:23:19 [13802] node03 crm_simulate:    debug:<br>
    native_assign_node:      Assigning node02 to DLM:3<br>
<br>
    This suggests that the restarted node attempts to occupy a dlm<br>
    journal that is allocated to another note and by doing so triggers a<br>
    chain reaction leading to all resources being restarted on all nodes.<br>
<br>
    I will try a different approach (based on your suggestions) on<br>
    starting DLM, iSCSI and Mount resources and see if this changes<br>
    anything.<br>
<br>
    If based on the log you have any suggestions they are welcome.<br>
<br>
    Thank you again for the help.<br>
<br>
    On Mon, Aug 28, 2017 at 3:53 PM, Vladislav Bogdanov<br></div></div><div><div class="h5">
    <<a href="mailto:bubble@hoster-ok.com" target="_blank">bubble@hoster-ok.com</a> <mailto:<a href="mailto:bubble@hoster-ok.com" target="_blank">bubble@hoster-ok.com</a>>> wrote:<br>
<br>
        28.08.2017 14:03, Octavian Ciobanu wrote:<br>
<br>
            Hey Vladislav,<br>
<br>
            Thank you for the info. I've tried you suggestions but the<br>
            behavior is<br>
            still the same. When an offline/standby node rejoins the<br>
            cluster all the<br>
            resources are first stopped and then started. I've added the<br>
            changes<br>
            I've made, see below in reply message, next to your suggestions.<br>
<br>
<br>
        Logs on DC (node where you see logs from the pengine process)<br>
        should contain references to pe-input-XX.bz2 files. Something<br>
        like "notice: Calculated transition XXXX, saving inputs in<br>
        /var/lib/pacemaker/pengine/pe-<wbr>input-XX.bz2"<br>
        Locate one for which Stop actions occur.<br>
        You can replay them with 'crm_simulate -S -x<br>
        /var/lib/pacemaker/pengine/pe-<wbr>input-XX.bz2' to see if that is<br>
        the correct one (look in the middle of output).<br>
<br>
        After that you may add some debugging:<br>
        PCMK_debug=yes PCMK_logfile=./pcmk.log crm_simulate -S -x<br>
        /var/lib/pacemaker/pengine/pe-<wbr>input-XX.bz2<br>
<br>
        That will produce a big file with all debugging messages enabled.<br>
<br>
        Try to locate a reason for restarts there.<br>
<br>
        Best,<br>
        Vladislav<br>
<br>
        Also please look inline (may be info there will be enough so you<br>
        won't need to debug).<br>
<br>
<br>
            Once again thank you for info.<br>
<br>
            Best regards.<br>
            Octavian Ciobanu<br>
<br>
            On Sat, Aug 26, 2017 at 8:17 PM, Vladislav Bogdanov<br>
            <<a href="mailto:bubble@hoster-ok.com" target="_blank">bubble@hoster-ok.com</a> <mailto:<a href="mailto:bubble@hoster-ok.com" target="_blank">bubble@hoster-ok.com</a>><br></div></div>
            <mailto:<a href="mailto:bubble@hoster-ok.com" target="_blank">bubble@hoster-ok.com</a> <mailto:<a href="mailto:bubble@hoster-ok.com" target="_blank">bubble@hoster-ok.com</a>>><wbr>><div><div class="h5"><br>
            wrote:<br>
<br>
                <a href="tel:26.08.2017%2019" value="+12608201719" target="_blank">26.08.2017 19</a> <tel:26.08.2017%2019><br>
            <tel:26.08.2017%2019>:36, Octavian Ciobanu wrote:<br>
<br>
                    Thank you for your reply.<br>
<br>
                    There is no reason to set location for the<br>
            resources, I think,<br>
                    because<br>
                    all the resources are set with clone options so they<br>
            are started<br>
                    on all<br>
                    nodes at the same time.<br>
<br>
<br>
                You still need to colocate "upper" resources with their<br>
                dependencies. Otherwise pacemaker will try to start them<br>
            even if<br>
                their dependencies fail. Order without colocation has<br>
            very limited<br>
                use (usually when resources may run on different nodes).<br>
            For clones<br>
                that is even more exotic.<br>
<br>
<br>
            I've added collocation<br>
<br>
            pcs constraint colocation add iSCSI1-clone with DLM-clone<br>
            pcs constraint colocation add iSCSI2-clone with DLM-clone<br>
            pcs constraint colocation add iSCSI3-clone with DLM-clone<br>
            pcs constraint colocation add Mount1-clone with iSCSI1-clone<br>
            pcs constraint colocation add Mount2-clone with iSCSI2-clone<br>
            pcs constraint colocation add Mount4-clone with iSCSI3-clone<br>
<br>
            The result is the same ... all clones are first stopped and<br>
            then started<br>
            beginning with DLM resource and ending with the Mount ones.<br>
<br>
<br>
        Yep, that was not meant to fix your problem. Just to prevent<br>
        future issues.<br>
<br>
<br>
<br>
                For you original question: ensure you have<br>
            interleave=true set for<br>
                all your clones. You seem to miss it for iSCSI ones.<br>
                interleave=false (default) is for different uses (when upper<br>
                resources require all clone instances to be up).<br>
<br>
<br>
            Modified iSCSI resources and added interleave="true" and<br>
            still no change<br>
            in behavior.<br>
<br>
<br>
        Weird... Probably you also do not need 'ordered="true"' for your<br>
        DLM clone? Knowing what is DLM, it does not need ordering, its<br>
        instances may be safely started in the parallel.<br>
<br>
<br>
<br>
                Also, just a minor note, iSCSI resources do not actually<br>
            depend on<br>
                dlm, mounts should depend on it.<br>
<br>
<br>
            I know but the mount resource must know when the iSCSI<br>
            resource to whom<br>
            is connected is started so the only solution I've seen was<br>
            to place DLM<br>
            before iSCSI and then Mount. If there is another solution, a<br>
            proper way<br>
            to do it, please can you give a reference or a place from<br>
            where to read<br>
            on how to do it ?<br>
<br>
<br>
        You would want to colocate (and order) mount with both DLM and<br>
        iSCSI. Multiple colocations/orders for the same resource are<br>
        allowed.<br>
        For mount you need DLM running and iSCSI disk connected. But you<br>
        actually do not need DLM to connect iSCSI disk (so DLM and iSCSI<br>
        resources may start in the parallel).<br>
<br>
<br>
<br>
                    And when it comes to stickiness I forgot to<br>
                    mention that but it set to 200. and also I have stonith<br>
                    configured  to<br>
                    use vmware esxi.<br>
<br>
                    Best regards<br>
                    Octavian Ciobanu<br>
<br>
                    On Sat, Aug 26, 2017 at 6:16 PM, John Keates<br>
            <<a href="mailto:john@keates.nl" target="_blank">john@keates.nl</a> <mailto:<a href="mailto:john@keates.nl" target="_blank">john@keates.nl</a>><br>
                    <mailto:<a href="mailto:john@keates.nl" target="_blank">john@keates.nl</a> <mailto:<a href="mailto:john@keates.nl" target="_blank">john@keates.nl</a>>><br>
                    <mailto:<a href="mailto:john@keates.nl" target="_blank">john@keates.nl</a> <mailto:<a href="mailto:john@keates.nl" target="_blank">john@keates.nl</a>><br>
            <mailto:<a href="mailto:john@keates.nl" target="_blank">john@keates.nl</a> <mailto:<a href="mailto:john@keates.nl" target="_blank">john@keates.nl</a>>>>> wrote:<br>
<br>
                        While I am by no means a CRM/Pacemaker expert, I<br>
            only see the<br>
                        resource primitives and the order constraints.<br>
            Wouldn’t you need<br>
                        location and/or colocation as well as stickiness<br>
            settings to<br>
                    prevent<br>
                        this from happening? What I think it might be<br>
            doing is<br>
                    seeing the<br>
                        new node, then trying to move the resources (but<br>
            not finding<br>
                    it a<br>
                        suitable target) and then moving them back where<br>
            they came<br>
                    from, but<br>
                        fast enough for you to only see it as a restart.<br>
<br>
                        If you crm_resource -P, it should also restart all<br>
                    resources, but<br>
                        put them in the preferred spot. If they end up<br>
            in the same<br>
                    place,<br>
                        you probably didn’t put and weighing in the<br>
            config or have<br>
                        stickiness set to INF.<br>
<br>
                        Kind regards,<br>
<br>
                        John Keates<br>
<br>
                            On 26 Aug 2017, at 14:23, Octavian Ciobanu<br>
                            <<a href="mailto:coctavian1979@gmail.com" target="_blank">coctavian1979@gmail.com</a><br>
            <mailto:<a href="mailto:coctavian1979@gmail.com" target="_blank">coctavian1979@gmail.co<wbr>m</a>><br>
                        <mailto:<a href="mailto:coctavian1979@gmail.com" target="_blank">coctavian1979@gmail.co<wbr>m</a><br>
            <mailto:<a href="mailto:coctavian1979@gmail.com" target="_blank">coctavian1979@gmail.co<wbr>m</a>>><br>
                        <mailto:<a href="mailto:coctavian1979@gmail.com" target="_blank">coctavian1979@gmail.co<wbr>m</a><br>
            <mailto:<a href="mailto:coctavian1979@gmail.com" target="_blank">coctavian1979@gmail.co<wbr>m</a>><br>
                        <mailto:<a href="mailto:coctavian1979@gmail.com" target="_blank">coctavian1979@gmail.co<wbr>m</a><br>
            <mailto:<a href="mailto:coctavian1979@gmail.com" target="_blank">coctavian1979@gmail.co<wbr>m</a>>>>> wrote:<br>
<br>
                            Hello all,<br>
<br>
                            While playing with cluster configuration I<br>
            noticed a strange<br>
                            behavior. If I stop/standby cluster services<br>
            on one node and<br>
                            reboot it, when it joins the cluster all the<br>
            resources<br>
                        that were<br>
                            started and working on active nodes get<br>
            stopped and<br>
                        restarted.<br>
<br>
                            My testing configuration is based on 4<br>
            nodes. One node is a<br>
                            storage node that makes 3 iSCSI targets<br>
            available for<br>
                        the other<br>
                            nodes to use,it is not configured to join<br>
            cluster, and<br>
                        three nodes<br>
                            that are configured in a cluster using the<br>
            following<br>
                        commands.<br>
<br>
                            pcs resource create DLM<br>
            ocf:pacemaker:controld op monitor<br>
                            interval="60" on-fail="fence" clone meta<br>
            clone-max="3"<br>
                            clone-node-max="1" interleave="true"<br>
            ordered="true"<br>
                            pcs resource create iSCSI1 ocf:heartbeat:iscsi<br>
                            portal="<a href="http://10.0.0.1:3260" rel="noreferrer" target="_blank">10.0.0.1:3260</a> <<a href="http://10.0.0.1:3260" rel="noreferrer" target="_blank">http://10.0.0.1:3260</a>><br>
            <<a href="http://10.0.0.1:3260" rel="noreferrer" target="_blank">http://10.0.0.1:3260</a>><br>
                        <<a href="http://10.0.0.1:3260/" rel="noreferrer" target="_blank">http://10.0.0.1:3260/</a>>"<br>
                            target="<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">iqn.2017-08.example.co<wbr>m</a><br>
            <<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a>><br>
                        <<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a><br>
            <<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a>>><br>
                            <<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a><br>
            <<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a>><br>
                        <<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a><br>
            <<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a>>>>:tgt1" op start interval="0"<br>
                            timeout="20" op stop interval="0"<br>
            timeout="20" op monitor<br>
                            interval="120" timeout="30" clone meta<br>
            clone-max="3"<br>
                            clone-node-max="1"<br>
                            pcs resource create iSCSI2 ocf:heartbeat:iscsi<br>
                            portal="<a href="http://10.0.0.1:3260" rel="noreferrer" target="_blank">10.0.0.1:3260</a> <<a href="http://10.0.0.1:3260" rel="noreferrer" target="_blank">http://10.0.0.1:3260</a>><br>
            <<a href="http://10.0.0.1:3260" rel="noreferrer" target="_blank">http://10.0.0.1:3260</a>><br>
                        <<a href="http://10.0.0.1:3260/" rel="noreferrer" target="_blank">http://10.0.0.1:3260/</a>>"<br>
                            target="<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">iqn.2017-08.example.co<wbr>m</a><br>
            <<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a>><br>
                        <<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a><br>
            <<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a>>><br>
                            <<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a><br>
            <<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a>><br>
                        <<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a><br></div></div><span class="">
            <<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a>>>>:tgt2" op start interval="0"<br>
                            timeout="20" op stop interval="0"<br>
            timeout="20" op monitor<br>
                            interval="120" timeout="30" clone meta<br>
            clone-max="3"<br>
                            clone-node-max="1"<br>
                            pcs resource create iSCSI3 ocf:heartbeat:iscsi<br>
                            portal="<a href="http://10.0.0.1:3260" rel="noreferrer" target="_blank">10.0.0.1:3260</a> <<a href="http://10.0.0.1:3260" rel="noreferrer" target="_blank">http://10.0.0.1:3260</a>><br></span><span class="">
            <<a href="http://10.0.0.1:3260" rel="noreferrer" target="_blank">http://10.0.0.1:3260</a>><br>
                        <<a href="http://10.0.0.1:3260/" rel="noreferrer" target="_blank">http://10.0.0.1:3260/</a>>"<br>
                            target="<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">iqn.2017-08.example.co<wbr>m</a><br>
            <<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a>><br>
                        <<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a><br>
            <<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a>>><br>
                            <<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a><br>
            <<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a>><br>
<br>
                        <<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a><br></span><div><div class="h5">
            <<a href="http://iqn.2017-08.example.com" rel="noreferrer" target="_blank">http://iqn.2017-08.example.co<wbr>m</a>>>>:tgt3" op start interval="0"<br>
<br>
                            timeout="20" op stop interval="0"<br>
            timeout="20" op monitor<br>
                            interval="120" timeout="30" clone meta<br>
            clone-max="3"<br>
                            clone-node-max="1"<br>
                            pcs resource create Mount1<br>
            ocf:heartbeat:Filesystem<br>
                            device="/dev/disk/by-label/MyC<wbr>luster:Data1"<br>
                        directory="/mnt/data1"<br>
                            fstype="gfs2"<br>
            options="noatime,nodiratime,rw<wbr>" op monitor<br>
                            interval="90" on-fail="fence" clone meta<br>
            clone-max="3"<br>
                            clone-node-max="1" interleave="true"<br>
                            pcs resource create Mount2<br>
            ocf:heartbeat:Filesystem<br>
                            device="/dev/disk/by-label/MyC<wbr>luster:Data2"<br>
                        directory="/mnt/data2"<br>
                            fstype="gfs2"<br>
            options="noatime,nodiratime,rw<wbr>" op monitor<br>
                            interval="90" on-fail="fence" clone meta<br>
            clone-max="3"<br>
                            clone-node-max="1" interleave="true"<br>
                            pcs resource create Mount3<br>
            ocf:heartbeat:Filesystem<br>
                            device="/dev/disk/by-label/MyC<wbr>luster:Data3"<br>
                        directory="/mnt/data3"<br>
                            fstype="gfs2"<br>
            options="noatime,nodiratime,rw<wbr>" op monitor<br>
                            interval="90" on-fail="fence" clone meta<br>
            clone-max="3"<br>
                            clone-node-max="1" interleave="true"<br>
                            pcs constraint order DLM-clone then iSCSI1-clone<br>
                            pcs constraint order DLM-clone then iSCSI2-clone<br>
                            pcs constraint order DLM-clone then iSCSI3-clone<br>
                            pcs constraint order iSCSI1-clone then<br>
            Mount1-clone<br>
                            pcs constraint order iSCSI2-clone then<br>
            Mount2-clone<br>
                            pcs constraint order iSCSI3-clone then<br>
            Mount3-clone<br>
<br>
                            If I issue the command "pcs cluster standby<br>
            node1" or<br>
                        "pcs cluster<br>
                            stop" on node 1 and after that I reboot the<br>
            node. When<br>
                        the node<br>
                            gets back online (unstandby if it was put in<br>
            standby<br>
                        mode) all the<br>
                            "MountX" resources get stopped on node 3 and<br>
            4 and<br>
                        started again.<br>
<br>
                            Can anyone help me figure out where and what<br>
            is the<br>
                        mistake in my<br>
                            configuration as I would like to keep the<br>
            started<br>
                        resources on<br>
                            active nodes (avoid stop and start of<br>
            resources)?<br>
<br>
                            Thank you in advance<br>
                            Octavian Ciobanu<br>
                            ______________________________<wbr>_________________<br>
                            Users mailing list: <a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
            <mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a>><br>
                        <mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
            <mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a>><wbr>><br>
                            <mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
            <mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a>><br>
                        <mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
            <mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a>><wbr>>><br>
<br>
            <a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/m<wbr>ailman/listinfo/users</a><br>
            <<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a>><br>
<br>
            <<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
            <<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a>>><br>
<br>
            <<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
            <<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a>><br>
<br>
            <<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
            <<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a>>>><br>
<br>
                            Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
                            Getting started:<br>
<br>
            <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc<wbr>/Cluster_from_Scratch.pdf</a><br>
            <<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a>><br>
<br>
            <<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a><br>
            <<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a>>><br>
<br>
            <<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a><br>
            <<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a>><br>
<br>
            <<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a><br>
            <<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a>>>><br>
                            Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
<br>
<br>
<br>
                        ______________________________<wbr>_________________<br>
                        Users mailing list: <a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
            <mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a>><br>
                    <mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
            <mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a>><wbr>><br>
            <mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a> <mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a>><br>
<br>
                    <mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
            <mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a>><wbr>>><br>
<br>
            <a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/m<wbr>ailman/listinfo/users</a><br>
            <<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a>><br>
                    <<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
            <<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a>>><br>
<br>
            <<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
            <<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a>><br>
                    <<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
            <<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a>>>><br>
<br>
                        Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
                        Getting started:<br>
<br>
            <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc<wbr>/Cluster_from_Scratch.pdf</a><br>
            <<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a>><br>
<br>
            <<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a><br>
            <<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a>>><br>
<br>
            <<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a><br>
            <<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a>><br>
<br>
            <<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a><br>
            <<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a>>>><br>
                        Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
<br>
<br>
<br>
<br>
                    ______________________________<wbr>_________________<br>
                    Users mailing list: <a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
            <mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a>><br>
                    <mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
            <mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a>><wbr>><br>
                    <a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/m<wbr>ailman/listinfo/users</a><br>
            <<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a>><br>
                    <<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
            <<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a>>><br>
<br>
                    Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
                    Getting started:<br>
<br>
            <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc<wbr>/Cluster_from_Scratch.pdf</a><br>
            <<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a>><br>
<br>
            <<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a><br>
            <<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a>>><br>
                    Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
<br>
<br>
<br>
                ______________________________<wbr>_________________<br>
                Users mailing list: <a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
            <mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a>> <mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
            <mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a>><wbr>><br>
                <a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/m<wbr>ailman/listinfo/users</a><br>
            <<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a>><br>
                <<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a><br>
            <<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a>>><br>
<br>
                Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
                Getting started:<br>
                <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc<wbr>/Cluster_from_Scratch.pdf</a><br>
            <<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a>><br>
                <<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a><br>
            <<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a>>><br>
                Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
<br>
<br>
<br>
<br>
            ______________________________<wbr>_________________<br>
            Users mailing list: <a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
            <mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a>><br>
            <a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/m<wbr>ailman/listinfo/users</a><br>
            <<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a>><br>
<br>
            Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
            Getting started:<br>
            <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc<wbr>/Cluster_from_Scratch.pdf</a><br>
            <<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a>><br>
            Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
<br>
<br>
<br>
        ______________________________<wbr>_________________<br>
        Users mailing list: <a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
        <mailto:<a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a>><br>
        <a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/m<wbr>ailman/listinfo/users</a><br>
        <<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/<wbr>mailman/listinfo/users</a>><br>
<br>
        Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
        Getting started:<br>
        <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc<wbr>/Cluster_from_Scratch.pdf</a><br>
        <<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/do<wbr>c/Cluster_from_Scratch.pdf</a>><br>
        Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
<br>
<br>
<br>
<br>
<br>
______________________________<wbr>_________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/m<wbr>ailman/listinfo/users</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc<wbr>/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
<br>
</div></div></blockquote><div class="HOEnZb"><div class="h5">
<br>
<br>
______________________________<wbr>_________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
<a href="http://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://lists.clusterlabs.org/m<wbr>ailman/listinfo/users</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc<wbr>/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
</div></div></blockquote></div><br></div>