[ClusterLabs] Antw: Re: single node fails to start the ocfs2 resource
Klaus Wenninger
kwenning at redhat.com
Tue Mar 13 11:18:24 EDT 2018
On 03/13/2018 03:43 PM, Muhammad Sharfuddin wrote:
> Thanks a lot for the explanation. But other then the ocfs2 resource
> group, this cluster starts all other resources
>
> on a single node, without any issue just because the use of
> "no-quorum-policy=ignore" option.
Yes I know. And what I tried to point out is that "no-quorum-policy=ignore"
is dangerous for services that do require a resource-manager. If you don't
have any of those go with a systemd startup.
Regards,
Klaus
>
> --
> Regards,
> Muhammad Sharfuddin
>
> On 3/13/2018 7:32 PM, Klaus Wenninger wrote:
>> On 03/13/2018 02:30 PM, Muhammad Sharfuddin wrote:
>>> Yes, by saying pacemaker, I meant to say corosync as well.
>>>
>>> Is there any fix ? or a two node cluster can't run ocfs2 resources
>>> when one node is offline ?
>> Actually there can't be a "fix" as 2 nodes are just not enough
>> for a partial-cluster to be quorate in the classical sense
>> (more votes than half of the cluster nodes).
>>
>> So to still be able to use it we have this 2-node config that
>> permanently sets quorum. But not to run into issues on
>> startup we need it to require both nodes seeing each
>> other once.
>>
>> So this is definitely nothing that is specific to ocfs2.
>> It just looks specific to ocfs2 because you've disabled
>> quorum for pacemaker.
>> To be honnest doing this you wouldn't need a resource-manager
>> at all and could just start up your services using systemd.
>>
>> If you don't want a full 3rd node, and still want to handle cases
>> where one node doesn't come up after a full shutdown of
>> all nodes, you probably could go for a setup with qdevice.
>>
>> Regards,
>> Klaus
>>
>>> --
>>> Regards,
>>> Muhammad Sharfuddin
>>>
>>> On 3/13/2018 6:16 PM, Klaus Wenninger wrote:
>>>> On 03/13/2018 02:03 PM, Muhammad Sharfuddin wrote:
>>>>> Hi,
>>>>>
>>>>> 1 - if I put a node(node2) offline; ocfs2 resources keep running on
>>>>> online node(node1)
>>>>>
>>>>> 2 - while node2 was offline, via cluster I stop/start the ocfs2
>>>>> resource group successfully so many times in a row.
>>>>>
>>>>> 3 - while node2 was offline; I restart the pacemaker service on the
>>>>> node1 and then tries to start the ocfs2 resource group, dlm started
>>>>> but ocfs2 file system resource does not start.
>>>>>
>>>>> Nutshell:
>>>>>
>>>>> a - both nodes must be online to start the ocfs2 resource.
>>>>>
>>>>> b - if one crashes or offline(gracefully) ocfs2 resource keeps
>>>>> running
>>>>> on the other/surviving node.
>>>>>
>>>>> c - while one node was offline, we can stop/start the ocfs2 resource
>>>>> group on the surviving node but if we stops the pacemaker service,
>>>>> then ocfs2 file system resource does not start with the following
>>>>> info
>>>>> in the logs:
>>>> >From the logs I would say startup of dlm_controld times out
>>>> because it
>>>> is waiting
>>>> for quorum - which doesn't happen because of wait-for-all.
>>>> Question is if you really just stopped pacemaker or if you stopped
>>>> corosync as well.
>>>> In the latter case I would say it is the expected behavior.
>>>>
>>>> Regards,
>>>> Klaus
>>>>
>>>>> lrmd[4317]: notice: executing - rsc:p-fssapmnt action:start
>>>>> call_id:53
>>>>> Filesystem(p-fssapmnt)[5139]: INFO: Running start for
>>>>> /dev/mapper/sapmnt on /sapmnt
>>>>> kernel: [ 706.162676] dlm: Using TCP for communications
>>>>> kernel: [ 706.162916] dlm: BFA9FF042AA045F4822C2A6A06020EE9: joining
>>>>> the lockspace group...
>>>>> dlm_controld[5105]: 759 fence work wait for quorum
>>>>> dlm_controld[5105]: 764 BFA9FF042AA045F4822C2A6A06020EE9 wait for
>>>>> quorum
>>>>> lrmd[4317]: warning: p-fssapmnt_start_0 process (PID 5139) timed out
>>>>> lrmd[4317]: warning: p-fssapmnt_start_0:5139 - timed out after
>>>>> 60000ms
>>>>> lrmd[4317]: notice: finished - rsc:p-fssapmnt action:start
>>>>> call_id:53 pid:5139 exit-code:1 exec-time:60002ms queue-time:0ms
>>>>> kernel: [ 766.056514] dlm: BFA9FF042AA045F4822C2A6A06020EE9: group
>>>>> event done -512 0
>>>>> kernel: [ 766.056528] dlm: BFA9FF042AA045F4822C2A6A06020EE9: group
>>>>> join failed -512 0
>>>>> crmd[4320]: notice: Result of stop operation for p-fssapmnt on
>>>>> pipci001: 0 (ok)
>>>>> crmd[4320]: notice: Initiating stop operation dlm_stop_0 locally on
>>>>> pipci001
>>>>> lrmd[4317]: notice: executing - rsc:dlm action:stop call_id:56
>>>>> dlm_controld[5105]: 766 shutdown ignored, active lockspaces
>>>>> lrmd[4317]: warning: dlm_stop_0 process (PID 5326) timed out
>>>>> lrmd[4317]: warning: dlm_stop_0:5326 - timed out after 100000ms
>>>>> lrmd[4317]: notice: finished - rsc:dlm action:stop call_id:56
>>>>> pid:5326 exit-code:1 exec-time:100003ms queue-time:0ms
>>>>> crmd[4320]: error: Result of stop operation for dlm on pipci001:
>>>>> Timed Out
>>>>> crmd[4320]: warning: Action 15 (dlm_stop_0) on pipci001 failed
>>>>> (target: 0 vs. rc: 1): Error
>>>>> crmd[4320]: notice: Transition aborted by operation dlm_stop_0
>>>>> 'modify' on pipci001: Event failed
>>>>> crmd[4320]: warning: Action 15 (dlm_stop_0) on pipci001 failed
>>>>> (target: 0 vs. rc: 1): Error
>>>>> pengine[4319]: notice: Watchdog will be used via SBD if fencing is
>>>>> required
>>>>> pengine[4319]: notice: On loss of CCM Quorum: Ignore
>>>>> pengine[4319]: warning: Processing failed op stop for dlm:0 on
>>>>> pipci001: unknown error (1)
>>>>> pengine[4319]: warning: Processing failed op stop for dlm:0 on
>>>>> pipci001: unknown error (1)
>>>>> pengine[4319]: warning: Cluster node pipci001 will be fenced: dlm:0
>>>>> failed there
>>>>> pengine[4319]: warning: Processing failed op start for p-fssapmnt:0
>>>>> on pipci001: unknown error (1)
>>>>> pengine[4319]: notice: Stop of failed resource dlm:0 is implicit
>>>>> after pipci001 is fenced
>>>>> pengine[4319]: notice: * Fence pipci001
>>>>> pengine[4319]: notice: Stop sbd-stonith#011(pipci001)
>>>>> pengine[4319]: notice: Stop dlm:0#011(pipci001)
>>>>> crmd[4320]: notice: Requesting fencing (reboot) of node pipci001
>>>>> stonith-ng[4316]: notice: Client crmd.4320.4c2f757b wants to fence
>>>>> (reboot) 'pipci001' with device '(any)'
>>>>> stonith-ng[4316]: notice: Requesting peer fencing (reboot) of
>>>>> pipci001
>>>>> stonith-ng[4316]: notice: sbd-stonith can fence (reboot) pipci001:
>>>>> dynamic-list
>>>>>
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Muhammad Sharfuddin | +923332144823 | nds.com.pk
>>>>>
>>>>> On 3/13/2018 1:04 PM, Ulrich Windl wrote:
>>>>>> Hi!
>>>>>>
>>>>>> I'd recommend this:
>>>>>> Cleanly boot your nodes, avoiding any manual operation with cluster
>>>>>> resources. Keep the logs.
>>>>>> Then start your tests, keeping the logs for each.
>>>>>> Try to fix issues by reading the logs and adjusting the cluster
>>>>>> configuration, and not by starting commands that the cluster should
>>>>>> start.
>>>>>>
>>>>>> We had an 2-node OCFS2 cluster running for quite some time with
>>>>>> SLES11, but now the cluster is three nodes. To me the output of
>>>>>> "crm_mon -1Arfj" combined with having set record-pending=true was
>>>>>> very valuable finding problems.
>>>>>>
>>>>>> Regards,
>>>>>> Ulrich
>>>>>>
>>>>>>
>>>>>>>>> Muhammad Sharfuddin <M.Sharfuddin at nds.com.pk> schrieb am
>>>>>>>>> 13.03.2018 um 08:43 in
>>>>>> Nachricht <7b773ae9-4209-d246-b5c0-2c8b67e623b3 at nds.com.pk>:
>>>>>>> Dear Klaus,
>>>>>>>
>>>>>>> If I understand you properly then, its a fencing issue, and
>>>>>>> whatever I
>>>>>>> am facing is "natural" or "by-design" in a two node cluster where
>>>>>>> quorum
>>>>>>> is incomplete.
>>>>>>>
>>>>>>> I am quite convinced that you have pointed out right because,
>>>>>>> when I
>>>>>>> start the dlm resource via cluster and then tries to start the
>>>>>>> ocfs2
>>>>>>> file system manually from command line, mount command remains
>>>>>>> hanged
>>>>>>> and
>>>>>>> following events are reported in the logs:
>>>>>>>
>>>>>>> kernel: [62622.864828] ocfs2: Registered cluster interface
>>>>>>> user
>>>>>>> kernel: [62622.884427] dlm: Using TCP for communications
>>>>>>> kernel: [62622.884750] dlm:
>>>>>>> BFA9FF042AA045F4822C2A6A06020EE9:
>>>>>>> joining the lockspace group...
>>>>>>> dlm_controld[17655]: 62627 fence work wait for quorum
>>>>>>> dlm_controld[17655]: 62680 BFA9FF042AA045F4822C2A6A06020EE9
>>>>>>> wait
>>>>>>> for quorum
>>>>>>>
>>>>>>> and then following messages keep reported every 5-10 minutes,
>>>>>>> till I
>>>>>>> kill the mount.ocfs2 process:
>>>>>>>
>>>>>>> dlm_controld[17655]: 62627 fence work wait for quorum
>>>>>>> dlm_controld[17655]: 62680 BFA9FF042AA045F4822C2A6A06020EE9
>>>>>>> wait
>>>>>>> for quorum
>>>>>>>
>>>>>>> I am also very much confused, because yesterday I did the same and
>>>>>>> was
>>>>>>> able to mount the ocfs2 file system manually from command line(at
>>>>>>> least
>>>>>>> once), and then unmount the file system manually stop the dlm
>>>>>>> resource
>>>>>>> from cluster and then complete ocfs2 resource stack(dlm, file
>>>>>>> systems)
>>>>>>> start/stop successfully via cluster even when only machine was
>>>>>>> online.
>>>>>>>
>>>>>>> In a two-node cluster, which have ocfs2 resources, we can't run the
>>>>>>> ocfs2 resources when quorum is incomplete(one node is offline) ?
>>>>>>>
>>>>>>> --
>>>>>>> Regards,
>>>>>>> Muhammad Sharfuddin
>>>>>>>
>>>>>>> On 3/12/2018 5:58 PM, Klaus Wenninger wrote:
>>>>>>>> On 03/12/2018 01:44 PM, Muhammad Sharfuddin wrote:
>>>>>>>>> Hi Klaus,
>>>>>>>>>
>>>>>>>>> primitive sbd-stonith stonith:external/sbd \
>>>>>>>>> op monitor interval=3000 timeout=20 \
>>>>>>>>> op start interval=0 timeout=240 \
>>>>>>>>> op stop interval=0 timeout=100 \
>>>>>>>>> params sbd_device="/dev/mapper/sbd" \
>>>>>>>>> meta target-role=Started
>>>>>>>> Makes more sense now.
>>>>>>>> Using pcmk_delay_max would probably be useful here
>>>>>>>> to prevent a fence-race.
>>>>>>>> That stonith-resource was not in your resource-list below ...
>>>>>>>>
>>>>>>>>> property cib-bootstrap-options: \
>>>>>>>>> have-watchdog=true \
>>>>>>>>> stonith-enabled=true \
>>>>>>>>> no-quorum-policy=ignore \
>>>>>>>>> stonith-timeout=90 \
>>>>>>>>> startup-fencing=true
>>>>>>>> You've set no-quorum-policy=ignore for pacemaker.
>>>>>>>> Whether this is a good idea or not in your setup is
>>>>>>>> written on another page.
>>>>>>>> But isn't dlm directly interfering with corosync so
>>>>>>>> that it would get the quorum state from there?
>>>>>>>> As you have 2-node set probably on a 2-node-cluster
>>>>>>>> this would - after both nodes down - wait for all
>>>>>>>> nodes up first.
>>>>>>>>
>>>>>>>> Regards,
>>>>>>>> Klaus
>>>>>>>>
>>>>>>>>> # ps -eaf |grep sbd
>>>>>>>>> root 6129 1 0 17:35 ? 00:00:00 sbd: inquisitor
>>>>>>>>> root 6133 6129 0 17:35 ? 00:00:00 sbd: watcher:
>>>>>>>>> /dev/mapper/sbd - slot: 1 - uuid:
>>>>>>>>> 6e80a337-95db-4608-bd62-d59517f39103
>>>>>>>>> root 6134 6129 0 17:35 ? 00:00:00 sbd: watcher:
>>>>>>>>> Pacemaker
>>>>>>>>> root 6135 6129 0 17:35 ? 00:00:00 sbd: watcher:
>>>>>>>>> Cluster
>>>>>>>>>
>>>>>>>>> This cluster does not start ocfs2 resources when I first
>>>>>>>>> intentionally
>>>>>>>>> crashed(reboot) both the nodes, then try to start ocfs2 resource
>>>>>>>>> while
>>>>>>>>> one node is offline.
>>>>>>>>>
>>>>>>>>> To fix the issue, I have one permanent solution, bring the other
>>>>>>>>> node(offline) online and things get fixed automatically, i.e
>>>>>>>>> ocfs2
>>>>>>>>> resources mounts.
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Regards,
>>>>>>>>> Muhammad Sharfuddin
>>>>>>>>>
>>>>>>>>> On 3/12/2018 5:25 PM, Klaus Wenninger wrote:
>>>>>>>>>> Hi Muhammad!
>>>>>>>>>>
>>>>>>>>>> Could you be a little bit more elaborate on your fencing-setup!
>>>>>>>>>> I read about you using SBD but I don't see any
>>>>>>>>>> sbd-fencing-resource.
>>>>>>>>>> For the case you wanted to use watchdog-fencing with SBD this
>>>>>>>>>> would require stonith-watchdog-timeout property to be set.
>>>>>>>>>> But watchdog-fencing relies on quorum (without 2-node trickery)
>>>>>>>>>> and thus wouldn't work on a 2-node-cluster anyway.
>>>>>>>>>>
>>>>>>>>>> Didn't read through the whole thread - so I might be missing
>>>>>>>>>> something ...
>>>>>>>>>>
>>>>>>>>>> Regards,
>>>>>>>>>> Klaus
>>>>>>>>>>
>>>>>>>>>> On 03/12/2018 12:51 PM, Muhammad Sharfuddin wrote:
>>>>>>>>>>> Hello Gang,
>>>>>>>>>>>
>>>>>>>>>>> as informed, previously cluster was fixed to start the ocfs2
>>>>>>>>>>> resources by
>>>>>>>>>>>
>>>>>>>>>>> a) crm resource start dlm
>>>>>>>>>>>
>>>>>>>>>>> b) mount/umount the ocfs2 file system manually. (this step was
>>>>>>>>>>> the
>>>>>>>>>>> fix)
>>>>>>>>>>>
>>>>>>>>>>> and then starting the clone group(which include dlm, ocfs2 file
>>>>>>>>>>> systems) worked fine:
>>>>>>>>>>>
>>>>>>>>>>> c) crm resource start base-clone.
>>>>>>>>>>>
>>>>>>>>>>> Now I crash the nodes intentionally and then keep only one node
>>>>>>>>>>> online, again cluster stopped starting the ocfs2 resources. I
>>>>>>>>>>> again
>>>>>>>>>>> tried to follow your instructions i.e
>>>>>>>>>>>
>>>>>>>>>>> i) crm resource start dlm
>>>>>>>>>>>
>>>>>>>>>>> then try to mount the ocfs2 file system manually which got
>>>>>>>>>>> hanged this
>>>>>>>>>>> time(previously manually mounting helped me):
>>>>>>>>>>>
>>>>>>>>>>> # cat /proc/3966/stack
>>>>>>>>>>> [<ffffffffa039f18e>] do_uevent+0x7e/0x200 [dlm]
>>>>>>>>>>> [<ffffffffa039fe0a>] new_lockspace+0x80a/0xa70 [dlm]
>>>>>>>>>>> [<ffffffffa03a02d9>] dlm_new_lockspace+0x69/0x160 [dlm]
>>>>>>>>>>> [<ffffffffa038e758>] user_cluster_connect+0xc8/0x350
>>>>>>>>>>> [ocfs2_stack_user]
>>>>>>>>>>> [<ffffffffa03c2872>] ocfs2_cluster_connect+0x192/0x240
>>>>>>>>>>> [ocfs2_stackglue]
>>>>>>>>>>> [<ffffffffa045eefc>] ocfs2_dlm_init+0x31c/0x570 [ocfs2]
>>>>>>>>>>> [<ffffffffa04a9983>] ocfs2_fill_super+0xb33/0x1200 [ocfs2]
>>>>>>>>>>> [<ffffffff8120e130>] mount_bdev+0x1a0/0x1e0
>>>>>>>>>>> [<ffffffff8120ea1a>] mount_fs+0x3a/0x170
>>>>>>>>>>> [<ffffffff81228bf2>] vfs_kern_mount+0x62/0x110
>>>>>>>>>>> [<ffffffff8122b123>] do_mount+0x213/0xcd0
>>>>>>>>>>> [<ffffffff8122bed5>] SyS_mount+0x85/0xd0
>>>>>>>>>>> [<ffffffff81614b0a>] entry_SYSCALL_64_fastpath+0x1e/0xb6
>>>>>>>>>>> [<ffffffffffffffff>] 0xffffffffffffffff
>>>>>>>>>>>
>>>>>>>>>>> I killed the mount.ocfs2 process stop(crm resource stop dlm)
>>>>>>>>>>> the
>>>>>>>>>>> dlm
>>>>>>>>>>> process, and then try to start(crm resource start dlm) the
>>>>>>>>>>> dlm(which
>>>>>>>>>>> previously always get started successfully), this time dlm
>>>>>>>>>>> didn't
>>>>>>>>>>> start and I checked the dlm_controld process
>>>>>>>>>>>
>>>>>>>>>>> cat /proc/3754/stack
>>>>>>>>>>> [<ffffffff8121dc55>] poll_schedule_timeout+0x45/0x60
>>>>>>>>>>> [<ffffffff8121f0bc>] do_sys_poll+0x38c/0x4f0
>>>>>>>>>>> [<ffffffff8121f2dd>] SyS_poll+0x5d/0xe0
>>>>>>>>>>> [<ffffffff81614b0a>] entry_SYSCALL_64_fastpath+0x1e/0xb6
>>>>>>>>>>> [<ffffffffffffffff>] 0xffffffffffffffff
>>>>>>>>>>>
>>>>>>>>>>> Nutshell:
>>>>>>>>>>>
>>>>>>>>>>> 1 - this cluster is configured to run when single node is
>>>>>>>>>>> online
>>>>>>>>>>>
>>>>>>>>>>> 2 - this cluster does not start the ocfs2 resources after a
>>>>>>>>>>> crash when
>>>>>>>>>>> only one node is online.
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Regards,
>>>>>>>>>>> Muhammad Sharfuddin | +923332144823 | nds.com.pk
>>>>>>>>>>>
>>>>>>>>>>> On 3/12/2018 12:41 PM, Gang He wrote:
>>>>>>>>>>>>> Hello Gang,
>>>>>>>>>>>>>
>>>>>>>>>>>>> to follow your instructions, I started the dlm resource via:
>>>>>>>>>>>>>
>>>>>>>>>>>>> crm resource start dlm
>>>>>>>>>>>>>
>>>>>>>>>>>>> then mount/unmount the ocfs2 file system manually..(which
>>>>>>>>>>>>> seems to be
>>>>>>>>>>>>> the fix of the situation).
>>>>>>>>>>>>>
>>>>>>>>>>>>> Now resources are getting started properly on a single node..
>>>>>>>>>>>>> I am
>>>>>>>>>>>>> happy
>>>>>>>>>>>>> as the issue is fixed, but at the same time I am lost because
>>>>>>>>>>>>> I have
>>>>>>>>>>>>> no idea
>>>>>>>>>>>>>
>>>>>>>>>>>>> how things get fixed here(merely by mounting/unmounting the
>>>>>>>>>>>>> ocfs2
>>>>>>>>>>>>> file
>>>>>>>>>>>>> systems)
>>>>>>>>>>>> >From your description.
>>>>>>>>>>>> I just wonder the DLM resource does not work normally under
>>>>>>>>>>>> that
>>>>>>>>>>>> situation.
>>>>>>>>>>>> Yan/Bin, do you have any comments about two-node cluster?
>>>>>>>>>>>> which
>>>>>>>>>>>> configuration settings will affect corosync quorum/DLM ?
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>> Gang
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Muhammad Sharfuddin
>>>>>>>>>>>>>
>>>>>>>>>>>>> On 3/12/2018 10:59 AM, Gang He wrote:
>>>>>>>>>>>>>> Hello Muhammad,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Usually, ocfs2 resource startup failure is caused by mount
>>>>>>>>>>>>>> command
>>>>>>>>>>>>>> timeout
>>>>>>>>>>>>> (or hanged).
>>>>>>>>>>>>>> The sample debugging method is,
>>>>>>>>>>>>>> remove ocfs2 resource from crm first,
>>>>>>>>>>>>>> then mount this file system manually, see if the mount
>>>>>>>>>>>>>> command
>>>>>>>>>>>>>> will be
>>>>>>>>>>>>> timeout or hanged.
>>>>>>>>>>>>>> If this command is hanged, please watch where is mount.ocfs2
>>>>>>>>>>>>>> process hanged
>>>>>>>>>>>>> via "cat /proc/xxx/stack" command.
>>>>>>>>>>>>>> If the back trace is stopped at DLM kernel module, usually
>>>>>>>>>>>>>> the root
>>>>>>>>>>>>>> cause is
>>>>>>>>>>>>> cluster configuration problem.
>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>> Gang
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On 3/12/2018 7:32 AM, Gang He wrote:
>>>>>>>>>>>>>>>> Hello Muhammad,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> I think this problem is not in ocfs2, the cause looks
>>>>>>>>>>>>>>>> like the
>>>>>>>>>>>>>>>> cluster
>>>>>>>>>>>>>>> quorum is missed.
>>>>>>>>>>>>>>>> For two-node cluster (does not three-node cluster), if one
>>>>>>>>>>>>>>>> node
>>>>>>>>>>>>>>>> is offline,
>>>>>>>>>>>>>>> the quorum will be missed by default.
>>>>>>>>>>>>>>>> So, you should configure two-node related quorum setting
>>>>>>>>>>>>>>>> according to the
>>>>>>>>>>>>>>> pacemaker manual.
>>>>>>>>>>>>>>>> Then, DLM can work normal, and ocfs2 resource can start
>>>>>>>>>>>>>>>> up.
>>>>>>>>>>>>>>> Yes its configured accordingly, no-quorum is set to
>>>>>>>>>>>>>>> "ignore".
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> property cib-bootstrap-options: \
>>>>>>>>>>>>>>> have-watchdog=true \
>>>>>>>>>>>>>>> stonith-enabled=true \
>>>>>>>>>>>>>>> stonith-timeout=80 \
>>>>>>>>>>>>>>> startup-fencing=true \
>>>>>>>>>>>>>>> no-quorum-policy=ignore
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>>> Gang
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> This two node cluster starts resources when both nodes
>>>>>>>>>>>>>>>>> are
>>>>>>>>>>>>>>>>> online but
>>>>>>>>>>>>>>>>> does not start the ocfs2 resources
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> when one node is offline. e.g if I gracefully stop the
>>>>>>>>>>>>>>>>> cluster
>>>>>>>>>>>>>>>>> resources
>>>>>>>>>>>>>>>>> then stop the pacemaker service on
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> either node, and try to start the ocfs2 resource on the
>>>>>>>>>>>>>>>>> online
>>>>>>>>>>>>>>>>> node, it
>>>>>>>>>>>>>>>>> fails.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> logs:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> pipci001 pengine[17732]: notice: Start
>>>>>>>>>>>>>>>>> dlm:0#011(pipci001)
>>>>>>>>>>>>>>>>> pengine[17732]: notice: Start
>>>>>>>>>>>>>>>>> p-fssapmnt:0#011(pipci001)
>>>>>>>>>>>>>>>>> pengine[17732]: notice: Start
>>>>>>>>>>>>>>>>> p-fsusrsap:0#011(pipci001)
>>>>>>>>>>>>>>>>> pipci001 pengine[17732]: notice: Calculated
>>>>>>>>>>>>>>>>> transition 2,
>>>>>>>>>>>>>>>>> saving
>>>>>>>>>>>>>>>>> inputs in /var/lib/pacemaker/pengine/pe-input-339.bz2
>>>>>>>>>>>>>>>>> pipci001 crmd[17733]: notice: Processing graph 2
>>>>>>>>>>>>>>>>> (ref=pe_calc-dc-1520613202-31) derived from
>>>>>>>>>>>>>>>>> /var/lib/pacemaker/pengine/pe-input-339.bz2
>>>>>>>>>>>>>>>>> crmd[17733]: notice: Initiating start operation
>>>>>>>>>>>>>>>>> dlm_start_0
>>>>>>>>>>>>>>>>> locally on
>>>>>>>>>>>>>>>>> pipci001
>>>>>>>>>>>>>>>>> lrmd[17730]: notice: executing - rsc:dlm action:start
>>>>>>>>>>>>>>>>> call_id:69
>>>>>>>>>>>>>>>>> dlm_controld[19019]: 4575 dlm_controld 4.0.7 started
>>>>>>>>>>>>>>>>> lrmd[17730]: notice: finished - rsc:dlm action:start
>>>>>>>>>>>>>>>>> call_id:69
>>>>>>>>>>>>>>>>> pid:18999 exit-code:0 exec-time:1082ms queue-time:1ms
>>>>>>>>>>>>>>>>> crmd[17733]: notice: Result of start operation for
>>>>>>>>>>>>>>>>> dlm on
>>>>>>>>>>>>>>>>> pipci001: 0 (ok)
>>>>>>>>>>>>>>>>> crmd[17733]: notice: Initiating monitor operation
>>>>>>>>>>>>>>>>> dlm_monitor_60000
>>>>>>>>>>>>>>>>> locally on pipci001
>>>>>>>>>>>>>>>>> crmd[17733]: notice: Initiating start operation
>>>>>>>>>>>>>>>>> p-fssapmnt_start_0
>>>>>>>>>>>>>>>>> locally on pipci001
>>>>>>>>>>>>>>>>> lrmd[17730]: notice: executing - rsc:p-fssapmnt
>>>>>>>>>>>>>>>>> action:start
>>>>>>>>>>>>>>>>> call_id:71
>>>>>>>>>>>>>>>>> Filesystem(p-fssapmnt)[19052]: INFO: Running start for
>>>>>>>>>>>>>>>>> /dev/mapper/sapmnt on /sapmnt
>>>>>>>>>>>>>>>>> kernel: [ 4576.529938] dlm: Using TCP for communications
>>>>>>>>>>>>>>>>> kernel: [ 4576.530233] dlm:
>>>>>>>>>>>>>>>>> BFA9FF042AA045F4822C2A6A06020EE9:
>>>>>>>>>>>>>>>>> joining
>>>>>>>>>>>>>>>>> the lockspace group.
>>>>>>>>>>>>>>>>> dlm_controld[19019]: 4629 fence work wait for quorum
>>>>>>>>>>>>>>>>> dlm_controld[19019]: 4634
>>>>>>>>>>>>>>>>> BFA9FF042AA045F4822C2A6A06020EE9
>>>>>>>>>>>>>>>>> wait
>>>>>>>>>>>>>>>>> for quorum
>>>>>>>>>>>>>>>>> lrmd[17730]: warning: p-fssapmnt_start_0 process (PID
>>>>>>>>>>>>>>>>> 19052)
>>>>>>>>>>>>>>>>> timed out
>>>>>>>>>>>>>>>>> kernel: [ 4636.418223] dlm:
>>>>>>>>>>>>>>>>> BFA9FF042AA045F4822C2A6A06020EE9:
>>>>>>>>>>>>>>>>> group
>>>>>>>>>>>>>>>>> event done -512 0
>>>>>>>>>>>>>>>>> kernel: [ 4636.418227] dlm:
>>>>>>>>>>>>>>>>> BFA9FF042AA045F4822C2A6A06020EE9:
>>>>>>>>>>>>>>>>> group join
>>>>>>>>>>>>>>>>> failed -512 0
>>>>>>>>>>>>>>>>> lrmd[17730]: warning: p-fssapmnt_start_0:19052 -
>>>>>>>>>>>>>>>>> timed out
>>>>>>>>>>>>>>>>> after 60000ms
>>>>>>>>>>>>>>>>> lrmd[17730]: notice: finished - rsc:p-fssapmnt
>>>>>>>>>>>>>>>>> action:start
>>>>>>>>>>>>>>>>> call_id:71
>>>>>>>>>>>>>>>>> pid:19052 exit-code:1 exec-time:60002ms queue-time:0ms
>>>>>>>>>>>>>>>>> kernel: [ 4636.420628] ocfs2: Unmounting device
>>>>>>>>>>>>>>>>> (254,1) on
>>>>>>>>>>>>>>>>> (node 0)
>>>>>>>>>>>>>>>>> crmd[17733]: error: Result of start operation for
>>>>>>>>>>>>>>>>> p-fssapmnt on
>>>>>>>>>>>>>>>>> pipci001: Timed Out
>>>>>>>>>>>>>>>>> crmd[17733]: warning: Action 11 (p-fssapmnt_start_0) on
>>>>>>>>>>>>>>>>> pipci001 failed
>>>>>>>>>>>>>>>>> (target: 0 vs. rc: 1): Error
>>>>>>>>>>>>>>>>> crmd[17733]: notice: Transition aborted by operation
>>>>>>>>>>>>>>>>> p-fssapmnt_start_0 'modify' on pipci001: Event failed
>>>>>>>>>>>>>>>>> crmd[17733]: warning: Action 11 (p-fssapmnt_start_0) on
>>>>>>>>>>>>>>>>> pipci001 failed
>>>>>>>>>>>>>>>>> (target: 0 vs. rc: 1): Error
>>>>>>>>>>>>>>>>> crmd[17733]: notice: Transition 2 (Complete=5,
>>>>>>>>>>>>>>>>> Pending=0,
>>>>>>>>>>>>>>>>> Fired=0,
>>>>>>>>>>>>>>>>> Skipped=0, Incomplete=6,
>>>>>>>>>>>>>>>>> Source=/var/lib/pacemaker/pengine/pe-input-339.bz2):
>>>>>>>>>>>>>>>>> Complete
>>>>>>>>>>>>>>>>> pengine[17732]: notice: Watchdog will be used via
>>>>>>>>>>>>>>>>> SBD if
>>>>>>>>>>>>>>>>> fencing is
>>>>>>>>>>>>>>>>> required
>>>>>>>>>>>>>>>>> pengine[17732]: notice: On loss of CCM Quorum: Ignore
>>>>>>>>>>>>>>>>> pengine[17732]: warning: Processing failed op start for
>>>>>>>>>>>>>>>>> p-fssapmnt:0 on
>>>>>>>>>>>>>>>>> pipci001: unknown error (1)
>>>>>>>>>>>>>>>>> pengine[17732]: warning: Processing failed op start for
>>>>>>>>>>>>>>>>> p-fssapmnt:0 on
>>>>>>>>>>>>>>>>> pipci001: unknown error (1)
>>>>>>>>>>>>>>>>> pengine[17732]: warning: Forcing base-clone away from
>>>>>>>>>>>>>>>>> pipci001
>>>>>>>>>>>>>>>>> after
>>>>>>>>>>>>>>>>> 1000000 failures (max=2)
>>>>>>>>>>>>>>>>> pengine[17732]: warning: Forcing base-clone away from
>>>>>>>>>>>>>>>>> pipci001
>>>>>>>>>>>>>>>>> after
>>>>>>>>>>>>>>>>> 1000000 failures (max=2)
>>>>>>>>>>>>>>>>> pengine[17732]: notice: Stop dlm:0#011(pipci001)
>>>>>>>>>>>>>>>>> pengine[17732]: notice: Stop
>>>>>>>>>>>>>>>>> p-fssapmnt:0#011(pipci001)
>>>>>>>>>>>>>>>>> pengine[17732]: notice: Calculated transition 3, saving
>>>>>>>>>>>>>>>>> inputs in
>>>>>>>>>>>>>>>>> /var/lib/pacemaker/pengine/pe-input-340.bz2
>>>>>>>>>>>>>>>>> pengine[17732]: notice: Watchdog will be used via
>>>>>>>>>>>>>>>>> SBD if
>>>>>>>>>>>>>>>>> fencing is
>>>>>>>>>>>>>>>>> required
>>>>>>>>>>>>>>>>> pengine[17732]: notice: On loss of CCM Quorum: Ignore
>>>>>>>>>>>>>>>>> pengine[17732]: warning: Processing failed op start for
>>>>>>>>>>>>>>>>> p-fssapmnt:0 on
>>>>>>>>>>>>>>>>> pipci001: unknown error (1)
>>>>>>>>>>>>>>>>> pengine[17732]: warning: Processing failed op start for
>>>>>>>>>>>>>>>>> p-fssapmnt:0 on
>>>>>>>>>>>>>>>>> pipci001: unknown error (1)
>>>>>>>>>>>>>>>>> pengine[17732]: warning: Forcing base-clone away from
>>>>>>>>>>>>>>>>> pipci001
>>>>>>>>>>>>>>>>> after
>>>>>>>>>>>>>>>>> 1000000 failures (max=2)
>>>>>>>>>>>>>>>>> pipci001 pengine[17732]: warning: Forcing base-clone
>>>>>>>>>>>>>>>>> away
>>>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>>> pipci001
>>>>>>>>>>>>>>>>> after 1000000 failures (max=2)
>>>>>>>>>>>>>>>>> pengine[17732]: notice: Stop dlm:0#011(pipci001)
>>>>>>>>>>>>>>>>> pengine[17732]: notice: Stop
>>>>>>>>>>>>>>>>> p-fssapmnt:0#011(pipci001)
>>>>>>>>>>>>>>>>> pengine[17732]: notice: Calculated transition 4, saving
>>>>>>>>>>>>>>>>> inputs in
>>>>>>>>>>>>>>>>> /var/lib/pacemaker/pengine/pe-input-341.bz2
>>>>>>>>>>>>>>>>> crmd[17733]: notice: Processing graph 4
>>>>>>>>>>>>>>>>> (ref=pe_calc-dc-1520613263-36)
>>>>>>>>>>>>>>>>> derived from /var/lib/pacemaker/pengine/pe-input-341.bz2
>>>>>>>>>>>>>>>>> crmd[17733]: notice: Initiating stop operation
>>>>>>>>>>>>>>>>> p-fssapmnt_stop_0
>>>>>>>>>>>>>>>>> locally on pipci001
>>>>>>>>>>>>>>>>> lrmd[17730]: notice: executing - rsc:p-fssapmnt
>>>>>>>>>>>>>>>>> action:stop
>>>>>>>>>>>>>>>>> call_id:72
>>>>>>>>>>>>>>>>> Filesystem(p-fssapmnt)[19189]: INFO: Running stop for
>>>>>>>>>>>>>>>>> /dev/mapper/sapmnt
>>>>>>>>>>>>>>>>> on /sapmnt
>>>>>>>>>>>>>>>>> pipci001 lrmd[17730]: notice: finished - rsc:p-fssapmnt
>>>>>>>>>>>>>>>>> action:stop
>>>>>>>>>>>>>>>>> call_id:72 pid:19189 exit-code:0 exec-time:83ms
>>>>>>>>>>>>>>>>> queue-time:0ms
>>>>>>>>>>>>>>>>> pipci001 crmd[17733]: notice: Result of stop operation
>>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>>> p-fssapmnt
>>>>>>>>>>>>>>>>> on pipci001: 0 (ok)
>>>>>>>>>>>>>>>>> crmd[17733]: notice: Initiating stop operation
>>>>>>>>>>>>>>>>> dlm_stop_0
>>>>>>>>>>>>>>>>> locally on
>>>>>>>>>>>>>>>>> pipci001
>>>>>>>>>>>>>>>>> pipci001 lrmd[17730]: notice: executing - rsc:dlm
>>>>>>>>>>>>>>>>> action:stop
>>>>>>>>>>>>>>>>> call_id:74
>>>>>>>>>>>>>>>>> pipci001 dlm_controld[19019]: 4636 shutdown ignored,
>>>>>>>>>>>>>>>>> active
>>>>>>>>>>>>>>>>> lockspaces
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> resource configuration:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> primitive p-fssapmnt Filesystem \
>>>>>>>>>>>>>>>>> params device="/dev/mapper/sapmnt"
>>>>>>>>>>>>>>>>> directory="/sapmnt"
>>>>>>>>>>>>>>>>> fstype=ocfs2 \
>>>>>>>>>>>>>>>>> op monitor interval=20 timeout=40 \
>>>>>>>>>>>>>>>>> op start timeout=60 interval=0 \
>>>>>>>>>>>>>>>>> op stop timeout=60 interval=0
>>>>>>>>>>>>>>>>> primitive dlm ocf:pacemaker:controld \
>>>>>>>>>>>>>>>>> op monitor interval=60 timeout=60 \
>>>>>>>>>>>>>>>>> op start interval=0 timeout=90 \
>>>>>>>>>>>>>>>>> op stop interval=0 timeout=100
>>>>>>>>>>>>>>>>> clone base-clone base-group \
>>>>>>>>>>>>>>>>> meta interleave=true target-role=Started
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> cluster properties:
>>>>>>>>>>>>>>>>> property cib-bootstrap-options: \
>>>>>>>>>>>>>>>>> have-watchdog=true \
>>>>>>>>>>>>>>>>> stonith-enabled=true \
>>>>>>>>>>>>>>>>> stonith-timeout=80 \
>>>>>>>>>>>>>>>>> startup-fencing=true \
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Software versions:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> kernel version: 4.4.114-94.11-default
>>>>>>>>>>>>>>>>> pacemaker-1.1.16-4.8.x86_64
>>>>>>>>>>>>>>>>> corosync-2.3.6-9.5.1.x86_64
>>>>>>>>>>>>>>>>> ocfs2-kmp-default-4.4.114-94.11.3.x86_64
>>>>>>>>>>>>>>>>> ocfs2-tools-1.8.5-1.35.x86_64
>>>>>>>>>>>>>>>>> dlm-kmp-default-4.4.114-94.11.3.x86_64
>>>>>>>>>>>>>>>>> libdlm3-4.0.7-1.28.x86_64
>>>>>>>>>>>>>>>>> libdlm-4.0.7-1.28.x86_64
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>> Muhammad Sharfuddin
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>>> This email has been checked for viruses by Avast
>>>>>>>>>>>>>>>>> antivirus
>>>>>>>>>>>>>>>>> software.
>>>>>>>>>>>>>>>>> https://www.avast.com/antivirus
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>>> Users mailing list: Users at clusterlabs.org
>>>>>>>>>>>>>>>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Project Home: http://www.clusterlabs.org
>>>>>>>>>>>>>>>>> Getting started:
>>>>>>>>>>>>>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>>>>>>>>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>> Users mailing list: Users at clusterlabs.org
>>>>>>>>>>>>>>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Project Home: http://www.clusterlabs.org
>>>>>>>>>>>>>>>> Getting started:
>>>>>>>>>>>>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>>>>>>>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>> Muhammad Sharfuddin
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>> Users mailing list: Users at clusterlabs.org
>>>>>>>>>>>>>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Project Home: http://www.clusterlabs.org
>>>>>>>>>>>>>>> Getting started:
>>>>>>>>>>>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>>>>>>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> Users mailing list: Users at clusterlabs.org
>>>>>>>>>>>>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Project Home: http://www.clusterlabs.org
>>>>>>>>>>>>>> Getting started:
>>>>>>>>>>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>>>>>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> Users mailing list: Users at clusterlabs.org
>>>>>>>>>>>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>>>>>>>>>>>
>>>>>>>>>>>>> Project Home: http://www.clusterlabs.org
>>>>>>>>>>>>> Getting started:
>>>>>>>>>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>>>>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> Users mailing list: Users at clusterlabs.org
>>>>>>>>>>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>>>>>>>>>>
>>>>>>>>>>>> Project Home: http://www.clusterlabs.org
>>>>>>>>>>>> Getting started:
>>>>>>>>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>>>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Users mailing list: Users at clusterlabs.org
>>>>>>>>>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>>>>>>>>>
>>>>>>>>>>> Project Home: http://www.clusterlabs.org
>>>>>>>>>>> Getting started:
>>>>>>>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>>>>> ---
>>>>>>>>> This email has been checked for viruses by Avast antivirus
>>>>>>>>> software.
>>>>>>>>> https://www.avast.com/antivirus
>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Users mailing list: Users at clusterlabs.org
>>>>>>>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>>>>>>>
>>>>>>>>> Project Home: http://www.clusterlabs.org
>>>>>>>>> Getting started:
>>>>>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>>> _______________________________________________
>>>>>>> Users mailing list: Users at clusterlabs.org
>>>>>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>>>>>
>>>>>>> Project Home: http://www.clusterlabs.org
>>>>>>> Getting started:
>>>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>> _______________________________________________
>>>>>> Users mailing list: Users at clusterlabs.org
>>>>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>>>>
>>>>>> Project Home: http://www.clusterlabs.org
>>>>>> Getting started:
>>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>>
>>>>> _______________________________________________
>>>>> Users mailing list: Users at clusterlabs.org
>>>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>>>
>>>>> Project Home: http://www.clusterlabs.org
>>>>> Getting started:
>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>> Bugs: http://bugs.clusterlabs.org
>>
>>
>
>
> ---
> This email has been checked for viruses by Avast antivirus software.
> https://www.avast.com/antivirus
>
--
Klaus Wenninger
Senior Software Engineer, EMEA ENG Base Operating Systems
Red Hat
kwenning at redhat.com
More information about the Users
mailing list