[ClusterLabs] Resources are stopped and started when one node rejoins
Octavian Ciobanu
coctavian1979 at gmail.com
Thu Aug 31 18:03:53 UTC 2017
Yes.
I've removed ordered="true" from DLM resource and also added collocation
constraints for Mount clones with both DLM and iSCSI.
On Thu, Aug 31, 2017 at 3:34 PM, Vladislav Bogdanov <bubble at hoster-ok.com>
wrote:
> 31.08.2017 14:53, Octavian Ciobanu wrote:
>
>> I'm back with confirmation that DLM is the one that triggers Mount
>> resources to stop when the stopped/suspended node rejoins
>>
>> When the DLM resource is started, on the rejoining node, it tries to get
>> a free journal but always gets one that is already occupied by another
>> node and triggers a domino effect, one node jumps to the next node
>> occupied dlm journal, stopping and restarting the mount resources.
>>
>> I have plenty of DLM journals (allocated 10 for 3 node configuration) so
>> for sure there are unused ones available.
>>
>> There is a way to make DLM keep its journal when the node is stopped and
>> use it when it starts ? or a way to make it keep it's active allocation
>> forcing the rejoining node to look for another unoccupied journal
>> without pushing away the nodes that occupy the journal it allocates for
>> the node that rejoins?
>>
>
> Did you try my suggestion from the previous e-mail?
>
> 'Probably you also do not need 'ordered="true"' for your DLM clone?
> Knowing what is DLM, it does not need ordering, its instances may be safely
> started in the parallel.'
>
>
>
>> Best regards
>>
>> On Mon, Aug 28, 2017 at 6:04 PM, Octavian Ciobanu
>> <coctavian1979 at gmail.com <mailto:coctavian1979 at gmail.com>> wrote:
>>
>> Thank you for info.
>> Looking over the output of the crm_simulate I've noticed the
>> "notice" messages and with the help of the debug mode I've found
>> this sequence in log
>>
>> Aug 28 16:23:19 [13802] node03 crm_simulate: debug:
>> native_assign_node: Assigning node01 to DLM:2
>> Aug 28 16:23:19 [13802] node03 crm_simulate: notice:
>> color_instance: Pre-allocation failed: got node01 instead of node02
>> Aug 28 16:23:19 [13802] node03 crm_simulate: info:
>> native_deallocate: Deallocating DLM:2 from core01
>> Aug 28 16:23:19 [13802] node03 crm_simulate: debug:
>> native_assign_node: Assigning core01 to DLM:3
>> Aug 28 16:23:19 [13802] node03 crm_simulate: notice:
>> color_instance: Pre-allocation failed: got node01 instead of node03
>> Aug 28 16:23:19 [13802] node03 crm_simulate: info:
>> native_deallocate: Deallocating DLM:3 from core01
>> Aug 28 16:23:19 [13802] node03 crm_simulate: debug:
>> native_assign_node: Assigning core01 to DLM:0
>> Aug 28 16:23:19 [13802] node03 crm_simulate: info:
>> rsc_merge_weights: DLM:2: Rolling back scores from iSCSI2-clone
>> Aug 28 16:23:19 [13802] node03 crm_simulate: info:
>> rsc_merge_weights: DLM:2: Rolling back scores from iSCSI2-clone
>> Aug 28 16:23:19 [13802] node03 crm_simulate: debug:
>> native_assign_node: Assigning node03 to DLM:2
>> Aug 28 16:23:19 [13802] node03 crm_simulate: info:
>> rsc_merge_weights: DLM:3: Rolling back scores from iSCSI2-clone
>> Aug 28 16:23:19 [13802] node03 crm_simulate: info:
>> rsc_merge_weights: DLM:3: Rolling back scores from iSCSI2-clone
>> Aug 28 16:23:19 [13802] node03 crm_simulate: debug:
>> native_assign_node: Assigning node02 to DLM:3
>>
>> This suggests that the restarted node attempts to occupy a dlm
>> journal that is allocated to another note and by doing so triggers a
>> chain reaction leading to all resources being restarted on all nodes.
>>
>> I will try a different approach (based on your suggestions) on
>> starting DLM, iSCSI and Mount resources and see if this changes
>> anything.
>>
>> If based on the log you have any suggestions they are welcome.
>>
>> Thank you again for the help.
>>
>> On Mon, Aug 28, 2017 at 3:53 PM, Vladislav Bogdanov
>> <bubble at hoster-ok.com <mailto:bubble at hoster-ok.com>> wrote:
>>
>> 28.08.2017 14:03, Octavian Ciobanu wrote:
>>
>> Hey Vladislav,
>>
>> Thank you for the info. I've tried you suggestions but the
>> behavior is
>> still the same. When an offline/standby node rejoins the
>> cluster all the
>> resources are first stopped and then started. I've added the
>> changes
>> I've made, see below in reply message, next to your
>> suggestions.
>>
>>
>> Logs on DC (node where you see logs from the pengine process)
>> should contain references to pe-input-XX.bz2 files. Something
>> like "notice: Calculated transition XXXX, saving inputs in
>> /var/lib/pacemaker/pengine/pe-input-XX.bz2"
>> Locate one for which Stop actions occur.
>> You can replay them with 'crm_simulate -S -x
>> /var/lib/pacemaker/pengine/pe-input-XX.bz2' to see if that is
>> the correct one (look in the middle of output).
>>
>> After that you may add some debugging:
>> PCMK_debug=yes PCMK_logfile=./pcmk.log crm_simulate -S -x
>> /var/lib/pacemaker/pengine/pe-input-XX.bz2
>>
>> That will produce a big file with all debugging messages enabled.
>>
>> Try to locate a reason for restarts there.
>>
>> Best,
>> Vladislav
>>
>> Also please look inline (may be info there will be enough so you
>> won't need to debug).
>>
>>
>> Once again thank you for info.
>>
>> Best regards.
>> Octavian Ciobanu
>>
>> On Sat, Aug 26, 2017 at 8:17 PM, Vladislav Bogdanov
>> <bubble at hoster-ok.com <mailto:bubble at hoster-ok.com>
>> <mailto:bubble at hoster-ok.com <mailto:bubble at hoster-ok.com>>>
>>
>> wrote:
>>
>> 26.08.2017 19 <tel:26.08.2017%2019>
>> <tel:26.08.2017%2019>:36, Octavian Ciobanu wrote:
>>
>> Thank you for your reply.
>>
>> There is no reason to set location for the
>> resources, I think,
>> because
>> all the resources are set with clone options so they
>> are started
>> on all
>> nodes at the same time.
>>
>>
>> You still need to colocate "upper" resources with their
>> dependencies. Otherwise pacemaker will try to start them
>> even if
>> their dependencies fail. Order without colocation has
>> very limited
>> use (usually when resources may run on different nodes).
>> For clones
>> that is even more exotic.
>>
>>
>> I've added collocation
>>
>> pcs constraint colocation add iSCSI1-clone with DLM-clone
>> pcs constraint colocation add iSCSI2-clone with DLM-clone
>> pcs constraint colocation add iSCSI3-clone with DLM-clone
>> pcs constraint colocation add Mount1-clone with iSCSI1-clone
>> pcs constraint colocation add Mount2-clone with iSCSI2-clone
>> pcs constraint colocation add Mount4-clone with iSCSI3-clone
>>
>> The result is the same ... all clones are first stopped and
>> then started
>> beginning with DLM resource and ending with the Mount ones.
>>
>>
>> Yep, that was not meant to fix your problem. Just to prevent
>> future issues.
>>
>>
>>
>> For you original question: ensure you have
>> interleave=true set for
>> all your clones. You seem to miss it for iSCSI ones.
>> interleave=false (default) is for different uses (when
>> upper
>> resources require all clone instances to be up).
>>
>>
>> Modified iSCSI resources and added interleave="true" and
>> still no change
>> in behavior.
>>
>>
>> Weird... Probably you also do not need 'ordered="true"' for your
>> DLM clone? Knowing what is DLM, it does not need ordering, its
>> instances may be safely started in the parallel.
>>
>>
>>
>> Also, just a minor note, iSCSI resources do not actually
>> depend on
>> dlm, mounts should depend on it.
>>
>>
>> I know but the mount resource must know when the iSCSI
>> resource to whom
>> is connected is started so the only solution I've seen was
>> to place DLM
>> before iSCSI and then Mount. If there is another solution, a
>> proper way
>> to do it, please can you give a reference or a place from
>> where to read
>> on how to do it ?
>>
>>
>> You would want to colocate (and order) mount with both DLM and
>> iSCSI. Multiple colocations/orders for the same resource are
>> allowed.
>> For mount you need DLM running and iSCSI disk connected. But you
>> actually do not need DLM to connect iSCSI disk (so DLM and iSCSI
>> resources may start in the parallel).
>>
>>
>>
>> And when it comes to stickiness I forgot to
>> mention that but it set to 200. and also I have
>> stonith
>> configured to
>> use vmware esxi.
>>
>> Best regards
>> Octavian Ciobanu
>>
>> On Sat, Aug 26, 2017 at 6:16 PM, John Keates
>> <john at keates.nl <mailto:john at keates.nl>
>> <mailto:john at keates.nl <mailto:john at keates.nl>>
>> <mailto:john at keates.nl <mailto:john at keates.nl>
>> <mailto:john at keates.nl <mailto:john at keates.nl>>>> wrote:
>>
>> While I am by no means a CRM/Pacemaker expert, I
>> only see the
>> resource primitives and the order constraints.
>> Wouldn’t you need
>> location and/or colocation as well as stickiness
>> settings to
>> prevent
>> this from happening? What I think it might be
>> doing is
>> seeing the
>> new node, then trying to move the resources (but
>> not finding
>> it a
>> suitable target) and then moving them back where
>> they came
>> from, but
>> fast enough for you to only see it as a restart.
>>
>> If you crm_resource -P, it should also restart all
>> resources, but
>> put them in the preferred spot. If they end up
>> in the same
>> place,
>> you probably didn’t put and weighing in the
>> config or have
>> stickiness set to INF.
>>
>> Kind regards,
>>
>> John Keates
>>
>> On 26 Aug 2017, at 14:23, Octavian Ciobanu
>> <coctavian1979 at gmail.com
>> <mailto:coctavian1979 at gmail.com>
>> <mailto:coctavian1979 at gmail.com
>> <mailto:coctavian1979 at gmail.com>>
>> <mailto:coctavian1979 at gmail.com
>> <mailto:coctavian1979 at gmail.com>
>> <mailto:coctavian1979 at gmail.com
>> <mailto:coctavian1979 at gmail.com>>>> wrote:
>>
>> Hello all,
>>
>> While playing with cluster configuration I
>> noticed a strange
>> behavior. If I stop/standby cluster services
>> on one node and
>> reboot it, when it joins the cluster all the
>> resources
>> that were
>> started and working on active nodes get
>> stopped and
>> restarted.
>>
>> My testing configuration is based on 4
>> nodes. One node is a
>> storage node that makes 3 iSCSI targets
>> available for
>> the other
>> nodes to use,it is not configured to join
>> cluster, and
>> three nodes
>> that are configured in a cluster using the
>> following
>> commands.
>>
>> pcs resource create DLM
>> ocf:pacemaker:controld op monitor
>> interval="60" on-fail="fence" clone meta
>> clone-max="3"
>> clone-node-max="1" interleave="true"
>> ordered="true"
>> pcs resource create iSCSI1 ocf:heartbeat:iscsi
>> portal="10.0.0.1:3260 <http://10.0.0.1:3260>
>> <http://10.0.0.1:3260>
>> <http://10.0.0.1:3260/>"
>> target="iqn.2017-08.example.com
>> <http://iqn.2017-08.example.com>
>> <http://iqn.2017-08.example.com
>> <http://iqn.2017-08.example.com>>
>> <http://iqn.2017-08.example.com
>> <http://iqn.2017-08.example.com>
>> <http://iqn.2017-08.example.com
>> <http://iqn.2017-08.example.com>>>:tgt1" op start
>> interval="0"
>> timeout="20" op stop interval="0"
>> timeout="20" op monitor
>> interval="120" timeout="30" clone meta
>> clone-max="3"
>> clone-node-max="1"
>> pcs resource create iSCSI2 ocf:heartbeat:iscsi
>> portal="10.0.0.1:3260 <http://10.0.0.1:3260>
>> <http://10.0.0.1:3260>
>> <http://10.0.0.1:3260/>"
>> target="iqn.2017-08.example.com
>> <http://iqn.2017-08.example.com>
>> <http://iqn.2017-08.example.com
>> <http://iqn.2017-08.example.com>>
>> <http://iqn.2017-08.example.com
>> <http://iqn.2017-08.example.com>
>> <http://iqn.2017-08.example.com
>> <http://iqn.2017-08.example.com>>>:tgt2" op start
>> interval="0"
>> timeout="20" op stop interval="0"
>> timeout="20" op monitor
>> interval="120" timeout="30" clone meta
>> clone-max="3"
>> clone-node-max="1"
>> pcs resource create iSCSI3 ocf:heartbeat:iscsi
>> portal="10.0.0.1:3260 <http://10.0.0.1:3260>
>> <http://10.0.0.1:3260>
>> <http://10.0.0.1:3260/>"
>> target="iqn.2017-08.example.com
>> <http://iqn.2017-08.example.com>
>> <http://iqn.2017-08.example.com
>> <http://iqn.2017-08.example.com>>
>> <http://iqn.2017-08.example.com
>> <http://iqn.2017-08.example.com>
>>
>> <http://iqn.2017-08.example.com
>> <http://iqn.2017-08.example.com>>>:tgt3" op start
>> interval="0"
>>
>> timeout="20" op stop interval="0"
>> timeout="20" op monitor
>> interval="120" timeout="30" clone meta
>> clone-max="3"
>> clone-node-max="1"
>> pcs resource create Mount1
>> ocf:heartbeat:Filesystem
>> device="/dev/disk/by-label/MyCluster:Data1"
>> directory="/mnt/data1"
>> fstype="gfs2"
>> options="noatime,nodiratime,rw" op monitor
>> interval="90" on-fail="fence" clone meta
>> clone-max="3"
>> clone-node-max="1" interleave="true"
>> pcs resource create Mount2
>> ocf:heartbeat:Filesystem
>> device="/dev/disk/by-label/MyCluster:Data2"
>> directory="/mnt/data2"
>> fstype="gfs2"
>> options="noatime,nodiratime,rw" op monitor
>> interval="90" on-fail="fence" clone meta
>> clone-max="3"
>> clone-node-max="1" interleave="true"
>> pcs resource create Mount3
>> ocf:heartbeat:Filesystem
>> device="/dev/disk/by-label/MyCluster:Data3"
>> directory="/mnt/data3"
>> fstype="gfs2"
>> options="noatime,nodiratime,rw" op monitor
>> interval="90" on-fail="fence" clone meta
>> clone-max="3"
>> clone-node-max="1" interleave="true"
>> pcs constraint order DLM-clone then
>> iSCSI1-clone
>> pcs constraint order DLM-clone then
>> iSCSI2-clone
>> pcs constraint order DLM-clone then
>> iSCSI3-clone
>> pcs constraint order iSCSI1-clone then
>> Mount1-clone
>> pcs constraint order iSCSI2-clone then
>> Mount2-clone
>> pcs constraint order iSCSI3-clone then
>> Mount3-clone
>>
>> If I issue the command "pcs cluster standby
>> node1" or
>> "pcs cluster
>> stop" on node 1 and after that I reboot the
>> node. When
>> the node
>> gets back online (unstandby if it was put in
>> standby
>> mode) all the
>> "MountX" resources get stopped on node 3 and
>> 4 and
>> started again.
>>
>> Can anyone help me figure out where and what
>> is the
>> mistake in my
>> configuration as I would like to keep the
>> started
>> resources on
>> active nodes (avoid stop and start of
>> resources)?
>>
>> Thank you in advance
>> Octavian Ciobanu
>> ______________________________
>> _________________
>> Users mailing list: Users at clusterlabs.org
>> <mailto:Users at clusterlabs.org>
>> <mailto:Users at clusterlabs.org
>> <mailto:Users at clusterlabs.org>>
>> <mailto:Users at clusterlabs.org
>> <mailto:Users at clusterlabs.org>
>> <mailto:Users at clusterlabs.org
>> <mailto:Users at clusterlabs.org>>>
>>
>> http://lists.clusterlabs.org/mailman/listinfo/users
>> <http://lists.clusterlabs.org/mailman/listinfo/users>
>>
>> <http://lists.clusterlabs.org/mailman/listinfo/users
>> <http://lists.clusterlabs.org/mailman/listinfo/users>>
>>
>> <http://lists.clusterlabs.org/mailman/listinfo/users
>> <http://lists.clusterlabs.org/mailman/listinfo/users>
>>
>> <http://lists.clusterlabs.org/mailman/listinfo/users
>> <http://lists.clusterlabs.org/mailman/listinfo/users>>>
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started:
>>
>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>
>>
>> <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>>
>>
>> <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>
>>
>> <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>>>
>> Bugs: http://bugs.clusterlabs.org
>>
>>
>>
>> _______________________________________________
>> Users mailing list: Users at clusterlabs.org
>> <mailto:Users at clusterlabs.org>
>> <mailto:Users at clusterlabs.org
>> <mailto:Users at clusterlabs.org>>
>> <mailto:Users at clusterlabs.org <mailto:Users at clusterlabs.org>
>>
>> <mailto:Users at clusterlabs.org
>> <mailto:Users at clusterlabs.org>>>
>>
>> http://lists.clusterlabs.org/mailman/listinfo/users
>> <http://lists.clusterlabs.org/mailman/listinfo/users>
>> <http://lists.clusterlabs.org/mailman/listinfo/users
>> <http://lists.clusterlabs.org/mailman/listinfo/users>>
>>
>> <http://lists.clusterlabs.org/mailman/listinfo/users
>> <http://lists.clusterlabs.org/mailman/listinfo/users>
>> <http://lists.clusterlabs.org/mailman/listinfo/users
>> <http://lists.clusterlabs.org/mailman/listinfo/users>>>
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started:
>>
>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>
>>
>> <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>>
>>
>> <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>
>>
>> <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>>>
>> Bugs: http://bugs.clusterlabs.org
>>
>>
>>
>>
>> _______________________________________________
>> Users mailing list: Users at clusterlabs.org
>> <mailto:Users at clusterlabs.org>
>> <mailto:Users at clusterlabs.org
>> <mailto:Users at clusterlabs.org>>
>> http://lists.clusterlabs.org/mailman/listinfo/users
>> <http://lists.clusterlabs.org/mailman/listinfo/users>
>> <http://lists.clusterlabs.org/mailman/listinfo/users
>> <http://lists.clusterlabs.org/mailman/listinfo/users>>
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started:
>>
>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>
>>
>> <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>>
>> Bugs: http://bugs.clusterlabs.org
>>
>>
>>
>> _______________________________________________
>> Users mailing list: Users at clusterlabs.org
>> <mailto:Users at clusterlabs.org> <mailto:Users at clusterlabs.org
>> <mailto:Users at clusterlabs.org>>
>> http://lists.clusterlabs.org/mailman/listinfo/users
>> <http://lists.clusterlabs.org/mailman/listinfo/users>
>> <http://lists.clusterlabs.org/mailman/listinfo/users
>> <http://lists.clusterlabs.org/mailman/listinfo/users>>
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started:
>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>
>> <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>>
>> Bugs: http://bugs.clusterlabs.org
>>
>>
>>
>>
>> _______________________________________________
>> Users mailing list: Users at clusterlabs.org
>> <mailto:Users at clusterlabs.org>
>> http://lists.clusterlabs.org/mailman/listinfo/users
>> <http://lists.clusterlabs.org/mailman/listinfo/users>
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started:
>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>
>> Bugs: http://bugs.clusterlabs.org
>>
>>
>>
>> _______________________________________________
>> Users mailing list: Users at clusterlabs.org
>> <mailto:Users at clusterlabs.org>
>> http://lists.clusterlabs.org/mailman/listinfo/users
>> <http://lists.clusterlabs.org/mailman/listinfo/users>
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started:
>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> <http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf>
>> Bugs: http://bugs.clusterlabs.org
>>
>>
>>
>>
>>
>> _______________________________________________
>> Users mailing list: Users at clusterlabs.org
>> http://lists.clusterlabs.org/mailman/listinfo/users
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>>
>>
>
> _______________________________________________
> Users mailing list: Users at clusterlabs.org
> http://lists.clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20170831/ee8b5526/attachment-0002.html>
More information about the Users
mailing list