[ClusterLabs] Resources are stopped and started when one node rejoins

Octavian Ciobanu coctavian1979 at gmail.com
Thu Aug 31 14:03:53 EDT 2017


Yes.

I've removed ordered="true" from DLM resource and also added collocation
constraints for Mount clones with both DLM and iSCSI.


On Thu, Aug 31, 2017 at 3:34 PM, Vladislav Bogdanov <bubble at hoster-ok.com>
wrote:

> 31.08.2017 14:53, Octavian Ciobanu wrote:
>
>> I'm back with confirmation that DLM is the one that triggers Mount
>> resources to stop when the stopped/suspended node rejoins
>>
>> When the DLM resource is started, on the rejoining node, it tries to get
>> a free journal but always gets one that is already occupied by another
>> node and triggers a domino effect, one node jumps to the next node
>> occupied dlm journal, stopping and restarting the mount resources.
>>
>> I have plenty of DLM journals (allocated 10 for 3 node configuration) so
>> for sure there are unused ones available.
>>
>> There is a way to make DLM keep its journal when the node is stopped and
>> use it when it starts ? or a way to make it keep it's active allocation
>> forcing the rejoining node to look for another unoccupied journal
>> without pushing away the nodes that occupy the journal it allocates for
>> the node that rejoins?
>>
>
> Did you try my suggestion from the previous e-mail?
>
> 'Probably you also do not need 'ordered="true"' for your DLM clone?
> Knowing what is DLM, it does not need ordering, its instances may be safely
> started in the parallel.'
>
>
>
>> Best regards
>>
>> On Mon, Aug 28, 2017 at 6:04 PM, Octavian Ciobanu
>> <coctavian1979 at gmail.com <mailto:coctavian1979 at gmail.com>> wrote:
>>
>>     Thank you for info.
>>     Looking over the output of the crm_simulate I've noticed the
>>     "notice" messages and with the help of the debug mode I've found
>>     this sequence in log
>>
>>     Aug 28 16:23:19 [13802] node03 crm_simulate:    debug:
>>     native_assign_node:      Assigning node01 to DLM:2
>>     Aug 28 16:23:19 [13802] node03 crm_simulate:   notice:
>>     color_instance:  Pre-allocation failed: got node01 instead of node02
>>     Aug 28 16:23:19 [13802] node03 crm_simulate:     info:
>>     native_deallocate:       Deallocating DLM:2 from core01
>>     Aug 28 16:23:19 [13802] node03 crm_simulate:    debug:
>>     native_assign_node:      Assigning core01 to DLM:3
>>     Aug 28 16:23:19 [13802] node03 crm_simulate:   notice:
>>     color_instance:  Pre-allocation failed: got node01 instead of node03
>>     Aug 28 16:23:19 [13802] node03 crm_simulate:     info:
>>     native_deallocate:       Deallocating DLM:3 from core01
>>     Aug 28 16:23:19 [13802] node03 crm_simulate:    debug:
>>     native_assign_node:      Assigning core01 to DLM:0
>>     Aug 28 16:23:19 [13802] node03 crm_simulate:     info:
>>     rsc_merge_weights:       DLM:2: Rolling back scores from iSCSI2-clone
>>     Aug 28 16:23:19 [13802] node03 crm_simulate:     info:
>>     rsc_merge_weights:       DLM:2: Rolling back scores from iSCSI2-clone
>>     Aug 28 16:23:19 [13802] node03 crm_simulate:    debug:
>>     native_assign_node:      Assigning node03 to DLM:2
>>     Aug 28 16:23:19 [13802] node03 crm_simulate:     info:
>>     rsc_merge_weights:       DLM:3: Rolling back scores from iSCSI2-clone
>>     Aug 28 16:23:19 [13802] node03 crm_simulate:     info:
>>     rsc_merge_weights:       DLM:3: Rolling back scores from iSCSI2-clone
>>     Aug 28 16:23:19 [13802] node03 crm_simulate:    debug:
>>     native_assign_node:      Assigning node02 to DLM:3
>>
>>     This suggests that the restarted node attempts to occupy a dlm
>>     journal that is allocated to another note and by doing so triggers a
>>     chain reaction leading to all resources being restarted on all nodes.
>>
>>     I will try a different approach (based on your suggestions) on
>>     starting DLM, iSCSI and Mount resources and see if this changes
>>     anything.
>>
>>     If based on the log you have any suggestions they are welcome.
>>
>>     Thank you again for the help.
>>
>>     On Mon, Aug 28, 2017 at 3:53 PM, Vladislav Bogdanov
>>     <bubble at hoster-ok.com <mailto:bubble at hoster-ok.com>> wrote:
>>
>>         28.08.2017 14:03, Octavian Ciobanu wrote:
>>
>>             Hey Vladislav,
>>
>>             Thank you for the info. I've tried you suggestions but the
>>             behavior is
>>             still the same. When an offline/standby node rejoins the
>>             cluster all the
>>             resources are first stopped and then started. I've added the
>>             changes
>>             I've made, see below in reply message, next to your
>> suggestions.
>>
>>
>>         Logs on DC (node where you see logs from the pengine process)
>>         should contain references to pe-input-XX.bz2 files. Something
>>         like "notice: Calculated transition XXXX, saving inputs in
>>         /var/lib/pacemaker/pengine/pe-input-XX.bz2"
>>         Locate one for which Stop actions occur.
>>         You can replay them with 'crm_simulate -S -x
>>         /var/lib/pacemaker/pengine/pe-input-XX.bz2' to see if that is
>>         the correct one (look in the middle of output).
>>
>>         After that you may add some debugging:
>>         PCMK_debug=yes PCMK_logfile=./pcmk.log crm_simulate -S -x
>>         /var/lib/pacemaker/pengine/pe-input-XX.bz2
>>
>>         That will produce a big file with all debugging messages enabled.
>>
>>         Try to locate a reason for restarts there.
>>
>>         Best,
>>         Vladislav
>>
>>         Also please look inline (may be info there will be enough so you
>>         won't need to debug).
>>
>>
>>             Once again thank you for info.
>>
>>             Best regards.
>>             Octavian Ciobanu
>>
>>             On Sat, Aug 26, 2017 at 8:17 PM, Vladislav Bogdanov
>>             <bubble at hoster-ok.com <mailto:bubble at hoster-ok.com>
>>             <mailto:bubble at hoster-ok.com <mailto:bubble at hoster-ok.com>>>
>>
>>             wrote:
>>
>>                 26.08.2017 19 <tel:26.08.2017%2019>
>>             <tel:26.08.2017%2019>:36, Octavian Ciobanu wrote:
>>
>>                     Thank you for your reply.
>>
>>                     There is no reason to set location for the
>>             resources, I think,
>>                     because
>>                     all the resources are set with clone options so they
>>             are started
>>                     on all
>>                     nodes at the same time.
>>
>>
>>                 You still need to colocate "upper" resources with their
>>                 dependencies. Otherwise pacemaker will try to start them
>>             even if
>>                 their dependencies fail. Order without colocation has
>>             very limited
>>                 use (usually when resources may run on different nodes).
>>             For clones
>>                 that is even more exotic.
>>
>>
>>             I've added collocation
>>
>>             pcs constraint colocation add iSCSI1-clone with DLM-clone
>>             pcs constraint colocation add iSCSI2-clone with DLM-clone
>>             pcs constraint colocation add iSCSI3-clone with DLM-clone
>>             pcs constraint colocation add Mount1-clone with iSCSI1-clone
>>             pcs constraint colocation add Mount2-clone with iSCSI2-clone
>>             pcs constraint colocation add Mount4-clone with iSCSI3-clone
>>
>>             The result is the same ... all clones are first stopped and
>>             then started
>>             beginning with DLM resource and ending with the Mount ones.
>>
>>
>>         Yep, that was not meant to fix your problem. Just to prevent
>>         future issues.
>>
>>
>>
>>                 For you original question: ensure you have
>>             interleave=true set for
>>                 all your clones. You seem to miss it for iSCSI ones.
>>                 interleave=false (default) is for different uses (when
>> upper
>>                 resources require all clone instances to be up).
>>
>>
>>             Modified iSCSI resources and added interleave="true" and
>>             still no change
>>             in behavior.
>>
>>
>>         Weird... Probably you also do not need 'ordered="true"' for your
>>         DLM clone? Knowing what is DLM, it does not need ordering, its
>>         instances may be safely started in the parallel.
>>
>>
>>
>>                 Also, just a minor note, iSCSI resources do not actually
>>             depend on
>>                 dlm, mounts should depend on it.
>>
>>
>>             I know but the mount resource must know when the iSCSI
>>             resource to whom
>>             is connected is started so the only solution I've seen was
>>             to place DLM
>>             before iSCSI and then Mount. If there is another solution, a
>>             proper way
>>             to do it, please can you give a reference or a place from
>>             where to read
>>             on how to do it ?
>>
>>
>>         You would want to colocate (and order) mount with both DLM and
>>         iSCSI. Multiple colocations/orders for the same resource are
>>         allowed.
>>         For mount you need DLM running and iSCSI disk connected. But you
>>         actually do not need DLM to connect iSCSI disk (so DLM and iSCSI
>>         resources may start in the parallel).
>>
>>
>>
>>                     And when it comes to stickiness I forgot to
>>                     mention that but it set to 200. and also I have
>> stonith
>>                     configured  to
>>                     use vmware esxi.
>>
>>                     Best regards
>>                     Octavian Ciobanu
>>
>>                     On Sat, Aug 26, 2017 at 6:16 PM, John Keates
>>             <john at keates.nl <mailto:john at keates.nl>
>>                     <mailto:john at keates.nl <mailto:john at keates.nl>>
>>                     <mailto:john at keates.nl <mailto:john at keates.nl>
>>             <mailto:john at keates.nl <mailto:john at keates.nl>>>> wrote:
>>
>>                         While I am by no means a CRM/Pacemaker expert, I
>>             only see the
>>                         resource primitives and the order constraints.
>>             Wouldn’t you need
>>                         location and/or colocation as well as stickiness
>>             settings to
>>                     prevent
>>                         this from happening? What I think it might be
>>             doing is
>>                     seeing the
>>                         new node, then trying to move the resources (but
>>             not finding
>>                     it a
>>                         suitable target) and then moving them back where
>>             they came
>>                     from, but
>>                         fast enough for you to only see it as a restart.
>>
>>                         If you crm_resource -P, it should also restart all
>>                     resources, but
>>                         put them in the preferred spot. If they end up
>>             in the same
>>                     place,
>>                         you probably didn’t put and weighing in the
>>             config or have
>>                         stickiness set to INF.
>>
>>                         Kind regards,
>>
>>                         John Keates
>>
>>                             On 26 Aug 2017, at 14:23, Octavian Ciobanu
>>                             <coctavian1979 at gmail.com
>>             <mailto:coctavian1979 at gmail.com>
>>                         <mailto:coctavian1979 at gmail.com
>>             <mailto:coctavian1979 at gmail.com>>
>>                         <mailto:coctavian1979 at gmail.com
>>             <mailto:coctavian1979 at gmail.com>
>>                         <mailto:coctavian1979 at gmail.com
>>             <mailto:coctavian1979 at gmail.com>>>> wrote:
>>
>>                             Hello all,
>>
>>                             While playing with cluster configuration I
>>             noticed a strange
>>                             behavior. If I stop/standby cluster services
>>             on one node and
>>                             reboot it, when it joins the cluster all the
>>             resources
>>                         that were
>>                             started and working on active nodes get
>>             stopped and
>>                         restarted.
>>
>>                             My testing configuration is based on 4
>>             nodes. One node is a
>>                             storage node that makes 3 iSCSI targets
>>             available for
>>                         the other
>>                             nodes to use,it is not configured to join
>>             cluster, and
>>                         three nodes
>>                             that are configured in a cluster using the
>>             following
>>                         commands.
>>
>>                             pcs resource create DLM
>>             ocf:pacemaker:controld op monitor
>>                             interval="60" on-fail="fence" clone meta
>>             clone-max="3"
>>                             clone-node-max="1" interleave="true"
>>             ordered="true"
>>                             pcs resource create iSCSI1 ocf:heartbeat:iscsi
>>                             portal="10.0.0.1:3260 <http://10.0.0.1:3260>
>>             <http://10.0.0.1:3260>
>>                         <http://10.0.0.1:3260/>"
>>                             target="iqn.2017-08.example.com
>>             <http://iqn.2017-08.example.com>
>>                         <http://iqn.2017-08.example.com
>>             <http://iqn.2017-08.example.com>>
>>                             <http://iqn.2017-08.example.com
>>             <http://iqn.2017-08.example.com>
>>                         <http://iqn.2017-08.example.com
>>             <http://iqn.2017-08.example.com>>>:tgt1" op start
>> interval="0"
>>                             timeout="20" op stop interval="0"
>>             timeout="20" op monitor
>>                             interval="120" timeout="30" clone meta
>>             clone-max="3"
>>                             clone-node-max="1"
>>                             pcs resource create iSCSI2 ocf:heartbeat:iscsi
>>                             portal="10.0.0.1:3260 <http://10.0.0.1:3260>
>>             <http://10.0.0.1:3260>
>>                         <http://10.0.0.1:3260/>"
>>                             target="iqn.2017-08.example.com
>>             <http://iqn.2017-08.example.com>
>>                         <http://iqn.2017-08.example.com
>>             <http://iqn.2017-08.example.com>>
>>                             <http://iqn.2017-08.example.com
>>             <http://iqn.2017-08.example.com>
>>                         <http://iqn.2017-08.example.com
>>             <http://iqn.2017-08.example.com>>>:tgt2" op start
>> interval="0"
>>                             timeout="20" op stop interval="0"
>>             timeout="20" op monitor
>>                             interval="120" timeout="30" clone meta
>>             clone-max="3"
>>                             clone-node-max="1"
>>                             pcs resource create iSCSI3 ocf:heartbeat:iscsi
>>                             portal="10.0.0.1:3260 <http://10.0.0.1:3260>
>>             <http://10.0.0.1:3260>
>>                         <http://10.0.0.1:3260/>"
>>                             target="iqn.2017-08.example.com
>>             <http://iqn.2017-08.example.com>
>>                         <http://iqn.2017-08.example.com
>>             <http://iqn.2017-08.example.com>>
>>                             <http://iqn.2017-08.example.com
>>             <http://iqn.2017-08.example.com>
>>
>>                         <http://iqn.2017-08.example.com
>>             <http://iqn.2017-08.example.com>>>:tgt3" op start
>> interval="0"
>>
>>                             timeout="20" op stop interval="0"
>>             timeout="20" op monitor
>>                             interval="120" timeout="30" clone meta
>>             clone-max="3"
>>                             clone-node-max="1"
>>                             pcs resource create Mount1
>>             ocf:heartbeat:Filesystem
>>                             device="/dev/disk/by-label/MyCluster:Data1"
>>                         directory="/mnt/data1"
>>                             fstype="gfs2"
>>             options="noatime,nodiratime,rw" op monitor
>>                             interval="90" on-fail="fence" clone meta
>>             clone-max="3"
>>                             clone-node-max="1" interleave="true"
>>                             pcs resource create Mount2
>>             ocf:heartbeat:Filesystem
>>                             device="/dev/disk/by-label/MyCluster:Data2"
>>                         directory="/mnt/data2"
>>                             fstype="gfs2"
>>             options="noatime,nodiratime,rw" op monitor
>>                             interval="90" on-fail="fence" clone meta
>>             clone-max="3"
>>                             clone-node-max="1" interleave="true"
>>                             pcs resource create Mount3
>>             ocf:heartbeat:Filesystem
>>                             device="/dev/disk/by-label/MyCluster:Data3"
>>                         directory="/mnt/data3"
>>                             fstype="gfs2"
>>             options="noatime,nodiratime,rw" op monitor
>>                             interval="90" on-fail="fence" clone meta
>>             clone-max="3"
>>                             clone-node-max="1" interleave="true"
>>                             pcs constraint order DLM-clone then
>> iSCSI1-clone
>>                             pcs constraint order DLM-clone then
>> iSCSI2-clone
>>                             pcs constraint order DLM-clone then
>> iSCSI3-clone
>>                             pcs constraint order iSCSI1-clone then
>>             Mount1-clone
>>                             pcs constraint order iSCSI2-clone then
>>             Mount2-clone
>>                             pcs constraint order iSCSI3-clone then
>>             Mount3-clone
>>
>>                             If I issue the command "pcs cluster standby
>>             node1" or
>>                         "pcs cluster
>>                             stop" on node 1 and after that I reboot the
>>             node. When
>>                         the node
>>                             gets back online (unstandby if it was put in
>>             standby
>>                         mode) all the
>>                             "MountX" resources get stopped on node 3 and
>>             4 and
>>                         started again.
>>
>>                             Can anyone help me figure out where and what
>>             is the
>>                         mistake in my
>>                             configuration as I would like to keep the
>>             started
>>                         resources on
>>                             active nodes (avoid stop and start of
>>             resources)?
>>
>>                             Thank you in advance
>>                             Octavian Ciobanu
>>                             ______________________________
>> _________________
>>                             Users mailing list: Users at clusterlabs.org
>>             <mailto:Users at clusterlabs.org>
>>                         <mailto:Users at clusterlabs.org
>>             <mailto:Users at clusterlabs.org>>
>>                             <mailto:Users at clusterlabs.org
>>             <mailto:Users at clusterlabs.org>
>>                         <mailto:Users at clusterlabs.org
>>             <mailto:Users at clusterlabs.org>>>
>>
>>             http://lists.clusterlabs.org/mailman/listinfo/users
>>             <http://lists.clusterlabs.org/mailman/listinfo/users>
>>
>>             <http://lists.clusterlabs.org/mailman/listinfo/users
>>             <http://lists.clusterlabs.org/mailman/listinfo/users>>
>>
>>             <http://lists.clusterlabs.org/mailman/listinfo/users
>>             <http://lists.clusterlabs.org/mailman/listinfo/users>
>>
>>             <http://lists.clusterlabs.org/mailman/listinfo/users
>>             <http://lists.clusterlabs.org/mailman/listinfo/users>>>
>>
>>                             Project Home: http://www.clusterlabs.org
>>                             Getting started:
>>
>>             http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>             <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>
>>
>>             <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>             <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>>
>>
>>             <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>             <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>
>>
>>             <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>             <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>>>
>>                             Bugs: http://bugs.clusterlabs.org
>>
>>
>>
>>                         _______________________________________________
>>                         Users mailing list: Users at clusterlabs.org
>>             <mailto:Users at clusterlabs.org>
>>                     <mailto:Users at clusterlabs.org
>>             <mailto:Users at clusterlabs.org>>
>>             <mailto:Users at clusterlabs.org <mailto:Users at clusterlabs.org>
>>
>>                     <mailto:Users at clusterlabs.org
>>             <mailto:Users at clusterlabs.org>>>
>>
>>             http://lists.clusterlabs.org/mailman/listinfo/users
>>             <http://lists.clusterlabs.org/mailman/listinfo/users>
>>                     <http://lists.clusterlabs.org/mailman/listinfo/users
>>             <http://lists.clusterlabs.org/mailman/listinfo/users>>
>>
>>             <http://lists.clusterlabs.org/mailman/listinfo/users
>>             <http://lists.clusterlabs.org/mailman/listinfo/users>
>>                     <http://lists.clusterlabs.org/mailman/listinfo/users
>>             <http://lists.clusterlabs.org/mailman/listinfo/users>>>
>>
>>                         Project Home: http://www.clusterlabs.org
>>                         Getting started:
>>
>>             http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>             <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>
>>
>>             <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>             <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>>
>>
>>             <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>             <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>
>>
>>             <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>             <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>>>
>>                         Bugs: http://bugs.clusterlabs.org
>>
>>
>>
>>
>>                     _______________________________________________
>>                     Users mailing list: Users at clusterlabs.org
>>             <mailto:Users at clusterlabs.org>
>>                     <mailto:Users at clusterlabs.org
>>             <mailto:Users at clusterlabs.org>>
>>                     http://lists.clusterlabs.org/mailman/listinfo/users
>>             <http://lists.clusterlabs.org/mailman/listinfo/users>
>>                     <http://lists.clusterlabs.org/mailman/listinfo/users
>>             <http://lists.clusterlabs.org/mailman/listinfo/users>>
>>
>>                     Project Home: http://www.clusterlabs.org
>>                     Getting started:
>>
>>             http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>             <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>
>>
>>             <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>             <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>>
>>                     Bugs: http://bugs.clusterlabs.org
>>
>>
>>
>>                 _______________________________________________
>>                 Users mailing list: Users at clusterlabs.org
>>             <mailto:Users at clusterlabs.org> <mailto:Users at clusterlabs.org
>>             <mailto:Users at clusterlabs.org>>
>>                 http://lists.clusterlabs.org/mailman/listinfo/users
>>             <http://lists.clusterlabs.org/mailman/listinfo/users>
>>                 <http://lists.clusterlabs.org/mailman/listinfo/users
>>             <http://lists.clusterlabs.org/mailman/listinfo/users>>
>>
>>                 Project Home: http://www.clusterlabs.org
>>                 Getting started:
>>                 http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>             <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>
>>                 <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>             <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>>
>>                 Bugs: http://bugs.clusterlabs.org
>>
>>
>>
>>
>>             _______________________________________________
>>             Users mailing list: Users at clusterlabs.org
>>             <mailto:Users at clusterlabs.org>
>>             http://lists.clusterlabs.org/mailman/listinfo/users
>>             <http://lists.clusterlabs.org/mailman/listinfo/users>
>>
>>             Project Home: http://www.clusterlabs.org
>>             Getting started:
>>             http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>             <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>
>>             Bugs: http://bugs.clusterlabs.org
>>
>>
>>
>>         _______________________________________________
>>         Users mailing list: Users at clusterlabs.org
>>         <mailto:Users at clusterlabs.org>
>>         http://lists.clusterlabs.org/mailman/listinfo/users
>>         <http://lists.clusterlabs.org/mailman/listinfo/users>
>>
>>         Project Home: http://www.clusterlabs.org
>>         Getting started:
>>         http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>         <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>
>>         Bugs: http://bugs.clusterlabs.org
>>
>>
>>
>>
>>
>> _______________________________________________
>> Users mailing list: Users at clusterlabs.org
>> http://lists.clusterlabs.org/mailman/listinfo/users
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>>
>>
>
> _______________________________________________
> Users mailing list: Users at clusterlabs.org
> http://lists.clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20170831/ee8b5526/attachment-0003.html>


More information about the Users mailing list