[ClusterLabs] Antw: Re: single node fails to start the ocfs2 resource
Klaus Wenninger
kwenning at redhat.com
Tue Mar 13 09:16:23 EDT 2018
On 03/13/2018 02:03 PM, Muhammad Sharfuddin wrote:
> Hi,
>
> 1 - if I put a node(node2) offline; ocfs2 resources keep running on
> online node(node1)
>
> 2 - while node2 was offline, via cluster I stop/start the ocfs2
> resource group successfully so many times in a row.
>
> 3 - while node2 was offline; I restart the pacemaker service on the
> node1 and then tries to start the ocfs2 resource group, dlm started
> but ocfs2 file system resource does not start.
>
> Nutshell:
>
> a - both nodes must be online to start the ocfs2 resource.
>
> b - if one crashes or offline(gracefully) ocfs2 resource keeps running
> on the other/surviving node.
>
> c - while one node was offline, we can stop/start the ocfs2 resource
> group on the surviving node but if we stops the pacemaker service,
> then ocfs2 file system resource does not start with the following info
> in the logs:
>From the logs I would say startup of dlm_controld times out because it
is waiting
for quorum - which doesn't happen because of wait-for-all.
Question is if you really just stopped pacemaker or if you stopped
corosync as well.
In the latter case I would say it is the expected behavior.
Regards,
Klaus
>
> lrmd[4317]: notice: executing - rsc:p-fssapmnt action:start call_id:53
> Filesystem(p-fssapmnt)[5139]: INFO: Running start for
> /dev/mapper/sapmnt on /sapmnt
> kernel: [ 706.162676] dlm: Using TCP for communications
> kernel: [ 706.162916] dlm: BFA9FF042AA045F4822C2A6A06020EE9: joining
> the lockspace group...
> dlm_controld[5105]: 759 fence work wait for quorum
> dlm_controld[5105]: 764 BFA9FF042AA045F4822C2A6A06020EE9 wait for quorum
> lrmd[4317]: warning: p-fssapmnt_start_0 process (PID 5139) timed out
> lrmd[4317]: warning: p-fssapmnt_start_0:5139 - timed out after 60000ms
> lrmd[4317]: notice: finished - rsc:p-fssapmnt action:start
> call_id:53 pid:5139 exit-code:1 exec-time:60002ms queue-time:0ms
> kernel: [ 766.056514] dlm: BFA9FF042AA045F4822C2A6A06020EE9: group
> event done -512 0
> kernel: [ 766.056528] dlm: BFA9FF042AA045F4822C2A6A06020EE9: group
> join failed -512 0
> crmd[4320]: notice: Result of stop operation for p-fssapmnt on
> pipci001: 0 (ok)
> crmd[4320]: notice: Initiating stop operation dlm_stop_0 locally on
> pipci001
> lrmd[4317]: notice: executing - rsc:dlm action:stop call_id:56
> dlm_controld[5105]: 766 shutdown ignored, active lockspaces
> lrmd[4317]: warning: dlm_stop_0 process (PID 5326) timed out
> lrmd[4317]: warning: dlm_stop_0:5326 - timed out after 100000ms
> lrmd[4317]: notice: finished - rsc:dlm action:stop call_id:56
> pid:5326 exit-code:1 exec-time:100003ms queue-time:0ms
> crmd[4320]: error: Result of stop operation for dlm on pipci001:
> Timed Out
> crmd[4320]: warning: Action 15 (dlm_stop_0) on pipci001 failed
> (target: 0 vs. rc: 1): Error
> crmd[4320]: notice: Transition aborted by operation dlm_stop_0
> 'modify' on pipci001: Event failed
> crmd[4320]: warning: Action 15 (dlm_stop_0) on pipci001 failed
> (target: 0 vs. rc: 1): Error
> pengine[4319]: notice: Watchdog will be used via SBD if fencing is
> required
> pengine[4319]: notice: On loss of CCM Quorum: Ignore
> pengine[4319]: warning: Processing failed op stop for dlm:0 on
> pipci001: unknown error (1)
> pengine[4319]: warning: Processing failed op stop for dlm:0 on
> pipci001: unknown error (1)
> pengine[4319]: warning: Cluster node pipci001 will be fenced: dlm:0
> failed there
> pengine[4319]: warning: Processing failed op start for p-fssapmnt:0
> on pipci001: unknown error (1)
> pengine[4319]: notice: Stop of failed resource dlm:0 is implicit
> after pipci001 is fenced
> pengine[4319]: notice: * Fence pipci001
> pengine[4319]: notice: Stop sbd-stonith#011(pipci001)
> pengine[4319]: notice: Stop dlm:0#011(pipci001)
> crmd[4320]: notice: Requesting fencing (reboot) of node pipci001
> stonith-ng[4316]: notice: Client crmd.4320.4c2f757b wants to fence
> (reboot) 'pipci001' with device '(any)'
> stonith-ng[4316]: notice: Requesting peer fencing (reboot) of pipci001
> stonith-ng[4316]: notice: sbd-stonith can fence (reboot) pipci001:
> dynamic-list
>
>
> --
> Regards,
> Muhammad Sharfuddin | +923332144823 | nds.com.pk
>
> On 3/13/2018 1:04 PM, Ulrich Windl wrote:
>> Hi!
>>
>> I'd recommend this:
>> Cleanly boot your nodes, avoiding any manual operation with cluster
>> resources. Keep the logs.
>> Then start your tests, keeping the logs for each.
>> Try to fix issues by reading the logs and adjusting the cluster
>> configuration, and not by starting commands that the cluster should
>> start.
>>
>> We had an 2-node OCFS2 cluster running for quite some time with
>> SLES11, but now the cluster is three nodes. To me the output of
>> "crm_mon -1Arfj" combined with having set record-pending=true was
>> very valuable finding problems.
>>
>> Regards,
>> Ulrich
>>
>>
>>>>> Muhammad Sharfuddin <M.Sharfuddin at nds.com.pk> schrieb am
>>>>> 13.03.2018 um 08:43 in
>> Nachricht <7b773ae9-4209-d246-b5c0-2c8b67e623b3 at nds.com.pk>:
>>> Dear Klaus,
>>>
>>> If I understand you properly then, its a fencing issue, and whatever I
>>> am facing is "natural" or "by-design" in a two node cluster where
>>> quorum
>>> is incomplete.
>>>
>>> I am quite convinced that you have pointed out right because, when I
>>> start the dlm resource via cluster and then tries to start the ocfs2
>>> file system manually from command line, mount command remains hanged
>>> and
>>> following events are reported in the logs:
>>>
>>> kernel: [62622.864828] ocfs2: Registered cluster interface user
>>> kernel: [62622.884427] dlm: Using TCP for communications
>>> kernel: [62622.884750] dlm: BFA9FF042AA045F4822C2A6A06020EE9:
>>> joining the lockspace group...
>>> dlm_controld[17655]: 62627 fence work wait for quorum
>>> dlm_controld[17655]: 62680 BFA9FF042AA045F4822C2A6A06020EE9 wait
>>> for quorum
>>>
>>> and then following messages keep reported every 5-10 minutes, till I
>>> kill the mount.ocfs2 process:
>>>
>>> dlm_controld[17655]: 62627 fence work wait for quorum
>>> dlm_controld[17655]: 62680 BFA9FF042AA045F4822C2A6A06020EE9 wait
>>> for quorum
>>>
>>> I am also very much confused, because yesterday I did the same and was
>>> able to mount the ocfs2 file system manually from command line(at least
>>> once), and then unmount the file system manually stop the dlm resource
>>> from cluster and then complete ocfs2 resource stack(dlm, file systems)
>>> start/stop successfully via cluster even when only machine was online.
>>>
>>> In a two-node cluster, which have ocfs2 resources, we can't run the
>>> ocfs2 resources when quorum is incomplete(one node is offline) ?
>>>
>>> --
>>> Regards,
>>> Muhammad Sharfuddin
>>>
>>> On 3/12/2018 5:58 PM, Klaus Wenninger wrote:
>>>> On 03/12/2018 01:44 PM, Muhammad Sharfuddin wrote:
>>>>> Hi Klaus,
>>>>>
>>>>> primitive sbd-stonith stonith:external/sbd \
>>>>> op monitor interval=3000 timeout=20 \
>>>>> op start interval=0 timeout=240 \
>>>>> op stop interval=0 timeout=100 \
>>>>> params sbd_device="/dev/mapper/sbd" \
>>>>> meta target-role=Started
>>>> Makes more sense now.
>>>> Using pcmk_delay_max would probably be useful here
>>>> to prevent a fence-race.
>>>> That stonith-resource was not in your resource-list below ...
>>>>
>>>>> property cib-bootstrap-options: \
>>>>> have-watchdog=true \
>>>>> stonith-enabled=true \
>>>>> no-quorum-policy=ignore \
>>>>> stonith-timeout=90 \
>>>>> startup-fencing=true
>>>> You've set no-quorum-policy=ignore for pacemaker.
>>>> Whether this is a good idea or not in your setup is
>>>> written on another page.
>>>> But isn't dlm directly interfering with corosync so
>>>> that it would get the quorum state from there?
>>>> As you have 2-node set probably on a 2-node-cluster
>>>> this would - after both nodes down - wait for all
>>>> nodes up first.
>>>>
>>>> Regards,
>>>> Klaus
>>>>
>>>>> # ps -eaf |grep sbd
>>>>> root 6129 1 0 17:35 ? 00:00:00 sbd: inquisitor
>>>>> root 6133 6129 0 17:35 ? 00:00:00 sbd: watcher:
>>>>> /dev/mapper/sbd - slot: 1 - uuid:
>>>>> 6e80a337-95db-4608-bd62-d59517f39103
>>>>> root 6134 6129 0 17:35 ? 00:00:00 sbd: watcher:
>>>>> Pacemaker
>>>>> root 6135 6129 0 17:35 ? 00:00:00 sbd: watcher: Cluster
>>>>>
>>>>> This cluster does not start ocfs2 resources when I first
>>>>> intentionally
>>>>> crashed(reboot) both the nodes, then try to start ocfs2 resource
>>>>> while
>>>>> one node is offline.
>>>>>
>>>>> To fix the issue, I have one permanent solution, bring the other
>>>>> node(offline) online and things get fixed automatically, i.e ocfs2
>>>>> resources mounts.
>>>>>
>>>>> --
>>>>> Regards,
>>>>> Muhammad Sharfuddin
>>>>>
>>>>> On 3/12/2018 5:25 PM, Klaus Wenninger wrote:
>>>>>> Hi Muhammad!
>>>>>>
>>>>>> Could you be a little bit more elaborate on your fencing-setup!
>>>>>> I read about you using SBD but I don't see any sbd-fencing-resource.
>>>>>> For the case you wanted to use watchdog-fencing with SBD this
>>>>>> would require stonith-watchdog-timeout property to be set.
>>>>>> But watchdog-fencing relies on quorum (without 2-node trickery)
>>>>>> and thus wouldn't work on a 2-node-cluster anyway.
>>>>>>
>>>>>> Didn't read through the whole thread - so I might be missing
>>>>>> something ...
>>>>>>
>>>>>> Regards,
>>>>>> Klaus
>>>>>>
>>>>>> On 03/12/2018 12:51 PM, Muhammad Sharfuddin wrote:
>>>>>>> Hello Gang,
>>>>>>>
>>>>>>> as informed, previously cluster was fixed to start the ocfs2
>>>>>>> resources by
>>>>>>>
>>>>>>> a) crm resource start dlm
>>>>>>>
>>>>>>> b) mount/umount the ocfs2 file system manually. (this step was the
>>>>>>> fix)
>>>>>>>
>>>>>>> and then starting the clone group(which include dlm, ocfs2 file
>>>>>>> systems) worked fine:
>>>>>>>
>>>>>>> c) crm resource start base-clone.
>>>>>>>
>>>>>>> Now I crash the nodes intentionally and then keep only one node
>>>>>>> online, again cluster stopped starting the ocfs2 resources. I again
>>>>>>> tried to follow your instructions i.e
>>>>>>>
>>>>>>> i) crm resource start dlm
>>>>>>>
>>>>>>> then try to mount the ocfs2 file system manually which got
>>>>>>> hanged this
>>>>>>> time(previously manually mounting helped me):
>>>>>>>
>>>>>>> # cat /proc/3966/stack
>>>>>>> [<ffffffffa039f18e>] do_uevent+0x7e/0x200 [dlm]
>>>>>>> [<ffffffffa039fe0a>] new_lockspace+0x80a/0xa70 [dlm]
>>>>>>> [<ffffffffa03a02d9>] dlm_new_lockspace+0x69/0x160 [dlm]
>>>>>>> [<ffffffffa038e758>] user_cluster_connect+0xc8/0x350
>>>>>>> [ocfs2_stack_user]
>>>>>>> [<ffffffffa03c2872>] ocfs2_cluster_connect+0x192/0x240
>>>>>>> [ocfs2_stackglue]
>>>>>>> [<ffffffffa045eefc>] ocfs2_dlm_init+0x31c/0x570 [ocfs2]
>>>>>>> [<ffffffffa04a9983>] ocfs2_fill_super+0xb33/0x1200 [ocfs2]
>>>>>>> [<ffffffff8120e130>] mount_bdev+0x1a0/0x1e0
>>>>>>> [<ffffffff8120ea1a>] mount_fs+0x3a/0x170
>>>>>>> [<ffffffff81228bf2>] vfs_kern_mount+0x62/0x110
>>>>>>> [<ffffffff8122b123>] do_mount+0x213/0xcd0
>>>>>>> [<ffffffff8122bed5>] SyS_mount+0x85/0xd0
>>>>>>> [<ffffffff81614b0a>] entry_SYSCALL_64_fastpath+0x1e/0xb6
>>>>>>> [<ffffffffffffffff>] 0xffffffffffffffff
>>>>>>>
>>>>>>> I killed the mount.ocfs2 process stop(crm resource stop dlm) the
>>>>>>> dlm
>>>>>>> process, and then try to start(crm resource start dlm) the
>>>>>>> dlm(which
>>>>>>> previously always get started successfully), this time dlm didn't
>>>>>>> start and I checked the dlm_controld process
>>>>>>>
>>>>>>> cat /proc/3754/stack
>>>>>>> [<ffffffff8121dc55>] poll_schedule_timeout+0x45/0x60
>>>>>>> [<ffffffff8121f0bc>] do_sys_poll+0x38c/0x4f0
>>>>>>> [<ffffffff8121f2dd>] SyS_poll+0x5d/0xe0
>>>>>>> [<ffffffff81614b0a>] entry_SYSCALL_64_fastpath+0x1e/0xb6
>>>>>>> [<ffffffffffffffff>] 0xffffffffffffffff
>>>>>>>
>>>>>>> Nutshell:
>>>>>>>
>>>>>>> 1 - this cluster is configured to run when single node is online
>>>>>>>
>>>>>>> 2 - this cluster does not start the ocfs2 resources after a
>>>>>>> crash when
>>>>>>> only one node is online.
>>>>>>>
>>>>>>> --
>>>>>>> Regards,
>>>>>>> Muhammad Sharfuddin | +923332144823 | nds.com.pk
>>>>>>>
>>>>>>> On 3/12/2018 12:41 PM, Gang He wrote:
>>>>>>>>> Hello Gang,
>>>>>>>>>
>>>>>>>>> to follow your instructions, I started the dlm resource via:
>>>>>>>>>
>>>>>>>>> crm resource start dlm
>>>>>>>>>
>>>>>>>>> then mount/unmount the ocfs2 file system manually..(which
>>>>>>>>> seems to be
>>>>>>>>> the fix of the situation).
>>>>>>>>>
>>>>>>>>> Now resources are getting started properly on a single node..
>>>>>>>>> I am
>>>>>>>>> happy
>>>>>>>>> as the issue is fixed, but at the same time I am lost because
>>>>>>>>> I have
>>>>>>>>> no idea
>>>>>>>>>
>>>>>>>>> how things get fixed here(merely by mounting/unmounting the ocfs2
>>>>>>>>> file
>>>>>>>>> systems)
>>>>>>>> >From your description.
>>>>>>>> I just wonder the DLM resource does not work normally under that
>>>>>>>> situation.
>>>>>>>> Yan/Bin, do you have any comments about two-node cluster? which
>>>>>>>> configuration settings will affect corosync quorum/DLM ?
>>>>>>>>
>>>>>>>>
>>>>>>>> Thanks
>>>>>>>> Gang
>>>>>>>>
>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Regards,
>>>>>>>>> Muhammad Sharfuddin
>>>>>>>>>
>>>>>>>>> On 3/12/2018 10:59 AM, Gang He wrote:
>>>>>>>>>> Hello Muhammad,
>>>>>>>>>>
>>>>>>>>>> Usually, ocfs2 resource startup failure is caused by mount
>>>>>>>>>> command
>>>>>>>>>> timeout
>>>>>>>>> (or hanged).
>>>>>>>>>> The sample debugging method is,
>>>>>>>>>> remove ocfs2 resource from crm first,
>>>>>>>>>> then mount this file system manually, see if the mount command
>>>>>>>>>> will be
>>>>>>>>> timeout or hanged.
>>>>>>>>>> If this command is hanged, please watch where is mount.ocfs2
>>>>>>>>>> process hanged
>>>>>>>>> via "cat /proc/xxx/stack" command.
>>>>>>>>>> If the back trace is stopped at DLM kernel module, usually
>>>>>>>>>> the root
>>>>>>>>>> cause is
>>>>>>>>> cluster configuration problem.
>>>>>>>>>> Thanks
>>>>>>>>>> Gang
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>> On 3/12/2018 7:32 AM, Gang He wrote:
>>>>>>>>>>>> Hello Muhammad,
>>>>>>>>>>>>
>>>>>>>>>>>> I think this problem is not in ocfs2, the cause looks like the
>>>>>>>>>>>> cluster
>>>>>>>>>>> quorum is missed.
>>>>>>>>>>>> For two-node cluster (does not three-node cluster), if one
>>>>>>>>>>>> node
>>>>>>>>>>>> is offline,
>>>>>>>>>>> the quorum will be missed by default.
>>>>>>>>>>>> So, you should configure two-node related quorum setting
>>>>>>>>>>>> according to the
>>>>>>>>>>> pacemaker manual.
>>>>>>>>>>>> Then, DLM can work normal, and ocfs2 resource can start up.
>>>>>>>>>>> Yes its configured accordingly, no-quorum is set to "ignore".
>>>>>>>>>>>
>>>>>>>>>>> property cib-bootstrap-options: \
>>>>>>>>>>> have-watchdog=true \
>>>>>>>>>>> stonith-enabled=true \
>>>>>>>>>>> stonith-timeout=80 \
>>>>>>>>>>> startup-fencing=true \
>>>>>>>>>>> no-quorum-policy=ignore
>>>>>>>>>>>
>>>>>>>>>>>> Thanks
>>>>>>>>>>>> Gang
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>
>>>>>>>>>>>>> This two node cluster starts resources when both nodes are
>>>>>>>>>>>>> online but
>>>>>>>>>>>>> does not start the ocfs2 resources
>>>>>>>>>>>>>
>>>>>>>>>>>>> when one node is offline. e.g if I gracefully stop the
>>>>>>>>>>>>> cluster
>>>>>>>>>>>>> resources
>>>>>>>>>>>>> then stop the pacemaker service on
>>>>>>>>>>>>>
>>>>>>>>>>>>> either node, and try to start the ocfs2 resource on the
>>>>>>>>>>>>> online
>>>>>>>>>>>>> node, it
>>>>>>>>>>>>> fails.
>>>>>>>>>>>>>
>>>>>>>>>>>>> logs:
>>>>>>>>>>>>>
>>>>>>>>>>>>> pipci001 pengine[17732]: notice: Start
>>>>>>>>>>>>> dlm:0#011(pipci001)
>>>>>>>>>>>>> pengine[17732]: notice: Start p-fssapmnt:0#011(pipci001)
>>>>>>>>>>>>> pengine[17732]: notice: Start p-fsusrsap:0#011(pipci001)
>>>>>>>>>>>>> pipci001 pengine[17732]: notice: Calculated transition 2,
>>>>>>>>>>>>> saving
>>>>>>>>>>>>> inputs in /var/lib/pacemaker/pengine/pe-input-339.bz2
>>>>>>>>>>>>> pipci001 crmd[17733]: notice: Processing graph 2
>>>>>>>>>>>>> (ref=pe_calc-dc-1520613202-31) derived from
>>>>>>>>>>>>> /var/lib/pacemaker/pengine/pe-input-339.bz2
>>>>>>>>>>>>> crmd[17733]: notice: Initiating start operation dlm_start_0
>>>>>>>>>>>>> locally on
>>>>>>>>>>>>> pipci001
>>>>>>>>>>>>> lrmd[17730]: notice: executing - rsc:dlm action:start
>>>>>>>>>>>>> call_id:69
>>>>>>>>>>>>> dlm_controld[19019]: 4575 dlm_controld 4.0.7 started
>>>>>>>>>>>>> lrmd[17730]: notice: finished - rsc:dlm action:start
>>>>>>>>>>>>> call_id:69
>>>>>>>>>>>>> pid:18999 exit-code:0 exec-time:1082ms queue-time:1ms
>>>>>>>>>>>>> crmd[17733]: notice: Result of start operation for dlm on
>>>>>>>>>>>>> pipci001: 0 (ok)
>>>>>>>>>>>>> crmd[17733]: notice: Initiating monitor operation
>>>>>>>>>>>>> dlm_monitor_60000
>>>>>>>>>>>>> locally on pipci001
>>>>>>>>>>>>> crmd[17733]: notice: Initiating start operation
>>>>>>>>>>>>> p-fssapmnt_start_0
>>>>>>>>>>>>> locally on pipci001
>>>>>>>>>>>>> lrmd[17730]: notice: executing - rsc:p-fssapmnt
>>>>>>>>>>>>> action:start
>>>>>>>>>>>>> call_id:71
>>>>>>>>>>>>> Filesystem(p-fssapmnt)[19052]: INFO: Running start for
>>>>>>>>>>>>> /dev/mapper/sapmnt on /sapmnt
>>>>>>>>>>>>> kernel: [ 4576.529938] dlm: Using TCP for communications
>>>>>>>>>>>>> kernel: [ 4576.530233] dlm: BFA9FF042AA045F4822C2A6A06020EE9:
>>>>>>>>>>>>> joining
>>>>>>>>>>>>> the lockspace group.
>>>>>>>>>>>>> dlm_controld[19019]: 4629 fence work wait for quorum
>>>>>>>>>>>>> dlm_controld[19019]: 4634 BFA9FF042AA045F4822C2A6A06020EE9
>>>>>>>>>>>>> wait
>>>>>>>>>>>>> for quorum
>>>>>>>>>>>>> lrmd[17730]: warning: p-fssapmnt_start_0 process (PID 19052)
>>>>>>>>>>>>> timed out
>>>>>>>>>>>>> kernel: [ 4636.418223] dlm: BFA9FF042AA045F4822C2A6A06020EE9:
>>>>>>>>>>>>> group
>>>>>>>>>>>>> event done -512 0
>>>>>>>>>>>>> kernel: [ 4636.418227] dlm: BFA9FF042AA045F4822C2A6A06020EE9:
>>>>>>>>>>>>> group join
>>>>>>>>>>>>> failed -512 0
>>>>>>>>>>>>> lrmd[17730]: warning: p-fssapmnt_start_0:19052 - timed out
>>>>>>>>>>>>> after 60000ms
>>>>>>>>>>>>> lrmd[17730]: notice: finished - rsc:p-fssapmnt action:start
>>>>>>>>>>>>> call_id:71
>>>>>>>>>>>>> pid:19052 exit-code:1 exec-time:60002ms queue-time:0ms
>>>>>>>>>>>>> kernel: [ 4636.420628] ocfs2: Unmounting device (254,1) on
>>>>>>>>>>>>> (node 0)
>>>>>>>>>>>>> crmd[17733]: error: Result of start operation for
>>>>>>>>>>>>> p-fssapmnt on
>>>>>>>>>>>>> pipci001: Timed Out
>>>>>>>>>>>>> crmd[17733]: warning: Action 11 (p-fssapmnt_start_0) on
>>>>>>>>>>>>> pipci001 failed
>>>>>>>>>>>>> (target: 0 vs. rc: 1): Error
>>>>>>>>>>>>> crmd[17733]: notice: Transition aborted by operation
>>>>>>>>>>>>> p-fssapmnt_start_0 'modify' on pipci001: Event failed
>>>>>>>>>>>>> crmd[17733]: warning: Action 11 (p-fssapmnt_start_0) on
>>>>>>>>>>>>> pipci001 failed
>>>>>>>>>>>>> (target: 0 vs. rc: 1): Error
>>>>>>>>>>>>> crmd[17733]: notice: Transition 2 (Complete=5, Pending=0,
>>>>>>>>>>>>> Fired=0,
>>>>>>>>>>>>> Skipped=0, Incomplete=6,
>>>>>>>>>>>>> Source=/var/lib/pacemaker/pengine/pe-input-339.bz2): Complete
>>>>>>>>>>>>> pengine[17732]: notice: Watchdog will be used via SBD if
>>>>>>>>>>>>> fencing is
>>>>>>>>>>>>> required
>>>>>>>>>>>>> pengine[17732]: notice: On loss of CCM Quorum: Ignore
>>>>>>>>>>>>> pengine[17732]: warning: Processing failed op start for
>>>>>>>>>>>>> p-fssapmnt:0 on
>>>>>>>>>>>>> pipci001: unknown error (1)
>>>>>>>>>>>>> pengine[17732]: warning: Processing failed op start for
>>>>>>>>>>>>> p-fssapmnt:0 on
>>>>>>>>>>>>> pipci001: unknown error (1)
>>>>>>>>>>>>> pengine[17732]: warning: Forcing base-clone away from
>>>>>>>>>>>>> pipci001
>>>>>>>>>>>>> after
>>>>>>>>>>>>> 1000000 failures (max=2)
>>>>>>>>>>>>> pengine[17732]: warning: Forcing base-clone away from
>>>>>>>>>>>>> pipci001
>>>>>>>>>>>>> after
>>>>>>>>>>>>> 1000000 failures (max=2)
>>>>>>>>>>>>> pengine[17732]: notice: Stop dlm:0#011(pipci001)
>>>>>>>>>>>>> pengine[17732]: notice: Stop p-fssapmnt:0#011(pipci001)
>>>>>>>>>>>>> pengine[17732]: notice: Calculated transition 3, saving
>>>>>>>>>>>>> inputs in
>>>>>>>>>>>>> /var/lib/pacemaker/pengine/pe-input-340.bz2
>>>>>>>>>>>>> pengine[17732]: notice: Watchdog will be used via SBD if
>>>>>>>>>>>>> fencing is
>>>>>>>>>>>>> required
>>>>>>>>>>>>> pengine[17732]: notice: On loss of CCM Quorum: Ignore
>>>>>>>>>>>>> pengine[17732]: warning: Processing failed op start for
>>>>>>>>>>>>> p-fssapmnt:0 on
>>>>>>>>>>>>> pipci001: unknown error (1)
>>>>>>>>>>>>> pengine[17732]: warning: Processing failed op start for
>>>>>>>>>>>>> p-fssapmnt:0 on
>>>>>>>>>>>>> pipci001: unknown error (1)
>>>>>>>>>>>>> pengine[17732]: warning: Forcing base-clone away from
>>>>>>>>>>>>> pipci001
>>>>>>>>>>>>> after
>>>>>>>>>>>>> 1000000 failures (max=2)
>>>>>>>>>>>>> pipci001 pengine[17732]: warning: Forcing base-clone away
>>>>>>>>>>>>> from
>>>>>>>>>>>>> pipci001
>>>>>>>>>>>>> after 1000000 failures (max=2)
>>>>>>>>>>>>> pengine[17732]: notice: Stop dlm:0#011(pipci001)
>>>>>>>>>>>>> pengine[17732]: notice: Stop p-fssapmnt:0#011(pipci001)
>>>>>>>>>>>>> pengine[17732]: notice: Calculated transition 4, saving
>>>>>>>>>>>>> inputs in
>>>>>>>>>>>>> /var/lib/pacemaker/pengine/pe-input-341.bz2
>>>>>>>>>>>>> crmd[17733]: notice: Processing graph 4
>>>>>>>>>>>>> (ref=pe_calc-dc-1520613263-36)
>>>>>>>>>>>>> derived from /var/lib/pacemaker/pengine/pe-input-341.bz2
>>>>>>>>>>>>> crmd[17733]: notice: Initiating stop operation
>>>>>>>>>>>>> p-fssapmnt_stop_0
>>>>>>>>>>>>> locally on pipci001
>>>>>>>>>>>>> lrmd[17730]: notice: executing - rsc:p-fssapmnt action:stop
>>>>>>>>>>>>> call_id:72
>>>>>>>>>>>>> Filesystem(p-fssapmnt)[19189]: INFO: Running stop for
>>>>>>>>>>>>> /dev/mapper/sapmnt
>>>>>>>>>>>>> on /sapmnt
>>>>>>>>>>>>> pipci001 lrmd[17730]: notice: finished - rsc:p-fssapmnt
>>>>>>>>>>>>> action:stop
>>>>>>>>>>>>> call_id:72 pid:19189 exit-code:0 exec-time:83ms
>>>>>>>>>>>>> queue-time:0ms
>>>>>>>>>>>>> pipci001 crmd[17733]: notice: Result of stop operation for
>>>>>>>>>>>>> p-fssapmnt
>>>>>>>>>>>>> on pipci001: 0 (ok)
>>>>>>>>>>>>> crmd[17733]: notice: Initiating stop operation dlm_stop_0
>>>>>>>>>>>>> locally on
>>>>>>>>>>>>> pipci001
>>>>>>>>>>>>> pipci001 lrmd[17730]: notice: executing - rsc:dlm
>>>>>>>>>>>>> action:stop
>>>>>>>>>>>>> call_id:74
>>>>>>>>>>>>> pipci001 dlm_controld[19019]: 4636 shutdown ignored, active
>>>>>>>>>>>>> lockspaces
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> resource configuration:
>>>>>>>>>>>>>
>>>>>>>>>>>>> primitive p-fssapmnt Filesystem \
>>>>>>>>>>>>> params device="/dev/mapper/sapmnt"
>>>>>>>>>>>>> directory="/sapmnt"
>>>>>>>>>>>>> fstype=ocfs2 \
>>>>>>>>>>>>> op monitor interval=20 timeout=40 \
>>>>>>>>>>>>> op start timeout=60 interval=0 \
>>>>>>>>>>>>> op stop timeout=60 interval=0
>>>>>>>>>>>>> primitive dlm ocf:pacemaker:controld \
>>>>>>>>>>>>> op monitor interval=60 timeout=60 \
>>>>>>>>>>>>> op start interval=0 timeout=90 \
>>>>>>>>>>>>> op stop interval=0 timeout=100
>>>>>>>>>>>>> clone base-clone base-group \
>>>>>>>>>>>>> meta interleave=true target-role=Started
>>>>>>>>>>>>>
>>>>>>>>>>>>> cluster properties:
>>>>>>>>>>>>> property cib-bootstrap-options: \
>>>>>>>>>>>>> have-watchdog=true \
>>>>>>>>>>>>> stonith-enabled=true \
>>>>>>>>>>>>> stonith-timeout=80 \
>>>>>>>>>>>>> startup-fencing=true \
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> Software versions:
>>>>>>>>>>>>>
>>>>>>>>>>>>> kernel version: 4.4.114-94.11-default
>>>>>>>>>>>>> pacemaker-1.1.16-4.8.x86_64
>>>>>>>>>>>>> corosync-2.3.6-9.5.1.x86_64
>>>>>>>>>>>>> ocfs2-kmp-default-4.4.114-94.11.3.x86_64
>>>>>>>>>>>>> ocfs2-tools-1.8.5-1.35.x86_64
>>>>>>>>>>>>> dlm-kmp-default-4.4.114-94.11.3.x86_64
>>>>>>>>>>>>> libdlm3-4.0.7-1.28.x86_64
>>>>>>>>>>>>> libdlm-4.0.7-1.28.x86_64
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Muhammad Sharfuddin
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> ---
>>>>>>>>>>>>> This email has been checked for viruses by Avast antivirus
>>>>>>>>>>>>> software.
>>>>>>>>>>>>> https://www.avast.com/antivirus
>>>>>>>>>>>>>
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> Users mailing list: Users at clusterlabs.org
>>>>>>>>>>>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>>>>>>>>>>>
>>>>>>>>>>>>> Project Home: http://www.clusterlabs.org
>>>>>>>>>>>>> Getting started:
>>>>>>>>>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>>>>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> Users mailing list: Users at clusterlabs.org
>>>>>>>>>>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>>>>>>>>>>
>>>>>>>>>>>> Project Home: http://www.clusterlabs.org
>>>>>>>>>>>> Getting started:
>>>>>>>>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>>>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Regards,
>>>>>>>>>>> Muhammad Sharfuddin
>>>>>>>>>>>
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Users mailing list: Users at clusterlabs.org
>>>>>>>>>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>>>>>>>>>
>>>>>>>>>>> Project Home: http://www.clusterlabs.org
>>>>>>>>>>> Getting started:
>>>>>>>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Users mailing list: Users at clusterlabs.org
>>>>>>>>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>>>>>>>>
>>>>>>>>>> Project Home: http://www.clusterlabs.org
>>>>>>>>>> Getting started:
>>>>>>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>>>>>>
>>>>>>>>> _______________________________________________
>>>>>>>>> Users mailing list: Users at clusterlabs.org
>>>>>>>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>>>>>>>
>>>>>>>>> Project Home: http://www.clusterlabs.org
>>>>>>>>> Getting started:
>>>>>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>>>> _______________________________________________
>>>>>>>> Users mailing list: Users at clusterlabs.org
>>>>>>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>>>>>>
>>>>>>>> Project Home: http://www.clusterlabs.org
>>>>>>>> Getting started:
>>>>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>>>>
>>>>>>> _______________________________________________
>>>>>>> Users mailing list: Users at clusterlabs.org
>>>>>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>>>>>
>>>>>>> Project Home: http://www.clusterlabs.org
>>>>>>> Getting started:
>>>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>> Bugs: http://bugs.clusterlabs.org
>>>>> ---
>>>>> This email has been checked for viruses by Avast antivirus software.
>>>>> https://www.avast.com/antivirus
>>>>>
>>>>> _______________________________________________
>>>>> Users mailing list: Users at clusterlabs.org
>>>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>>>
>>>>> Project Home: http://www.clusterlabs.org
>>>>> Getting started:
>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>> Bugs: http://bugs.clusterlabs.org
>>> _______________________________________________
>>> Users mailing list: Users at clusterlabs.org
>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>
>>> Project Home: http://www.clusterlabs.org
>>> Getting started:
>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>> Bugs: http://bugs.clusterlabs.org
>> _______________________________________________
>> Users mailing list: Users at clusterlabs.org
>> https://lists.clusterlabs.org/mailman/listinfo/users
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>>
>
> _______________________________________________
> Users mailing list: Users at clusterlabs.org
> https://lists.clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
More information about the Users
mailing list