[ClusterLabs] Antw: Re: single node fails to start the ocfs2 resource

Muhammad Sharfuddin M.Sharfuddin at nds.com.pk
Tue Mar 13 10:43:50 EDT 2018 

Thanks a lot for the explanation. But other then the ocfs2 resource 
group, this cluster starts all other resources

on a single node, without any issue just because the use of 
"no-quorum-policy=ignore" option.

--
Regards,
Muhammad Sharfuddin

On 3/13/2018 7:32 PM, Klaus Wenninger wrote:
> On 03/13/2018 02:30 PM, Muhammad Sharfuddin wrote:
>> Yes, by saying pacemaker,  I meant to say corosync as well.
>>
>> Is there any fix ? or a two node cluster can't run ocfs2 resources
>> when one node is offline ?
> Actually there can't be a "fix" as 2 nodes are just not enough
> for a partial-cluster to be quorate in the classical sense
> (more votes than half of the cluster nodes).
>
> So to still be able to use it we have this 2-node config that
> permanently sets quorum. But not to run into issues on
> startup we need it to require both nodes seeing each
> other once.
>
> So this is definitely nothing that is specific to ocfs2.
> It just looks specific to ocfs2 because you've disabled
> quorum for pacemaker.
> To be honnest doing this you wouldn't need a resource-manager
> at all and could just start up your services using systemd.
>
> If you don't want a full 3rd node, and still want to handle cases
> where one node doesn't come up after a full shutdown of
> all nodes, you probably could go for a setup with qdevice.
>
> Regards,
> Klaus
>
>> -- 
>> Regards,
>> Muhammad Sharfuddin
>>
>> On 3/13/2018 6:16 PM, Klaus Wenninger wrote:
>>> On 03/13/2018 02:03 PM, Muhammad Sharfuddin wrote:
>>>> Hi,
>>>>
>>>> 1 - if I put a node(node2) offline; ocfs2 resources keep running on
>>>> online node(node1)
>>>>
>>>> 2 - while node2 was offline, via cluster I stop/start the ocfs2
>>>> resource group successfully so many times in a row.
>>>>
>>>> 3 - while node2 was offline; I restart the pacemaker service on the
>>>> node1 and then tries to start the ocfs2 resource group, dlm started
>>>> but ocfs2 file system resource does not start.
>>>>
>>>> Nutshell:
>>>>
>>>> a - both nodes must be online to start the ocfs2 resource.
>>>>
>>>> b - if one crashes or offline(gracefully) ocfs2 resource keeps running
>>>> on the other/surviving node.
>>>>
>>>> c - while one node was offline, we can stop/start the ocfs2 resource
>>>> group on the surviving node but if we stops the pacemaker service,
>>>> then ocfs2 file system resource does not start with the following info
>>>> in the logs:
>>> >From the logs I would say startup of dlm_controld times out because it
>>> is waiting
>>> for quorum - which doesn't happen because of wait-for-all.
>>> Question is if you really just stopped pacemaker or if you stopped
>>> corosync as well.
>>> In the latter case I would say it is the expected behavior.
>>>
>>> Regards,
>>> Klaus
>>>   
>>>> lrmd[4317]:   notice: executing - rsc:p-fssapmnt action:start
>>>> call_id:53
>>>> Filesystem(p-fssapmnt)[5139]: INFO: Running start for
>>>> /dev/mapper/sapmnt on /sapmnt
>>>> kernel: [  706.162676] dlm: Using TCP for communications
>>>> kernel: [  706.162916] dlm: BFA9FF042AA045F4822C2A6A06020EE9: joining
>>>> the lockspace group...
>>>> dlm_controld[5105]: 759 fence work wait for quorum
>>>> dlm_controld[5105]: 764 BFA9FF042AA045F4822C2A6A06020EE9 wait for
>>>> quorum
>>>> lrmd[4317]:  warning: p-fssapmnt_start_0 process (PID 5139) timed out
>>>> lrmd[4317]:  warning: p-fssapmnt_start_0:5139 - timed out after 60000ms
>>>> lrmd[4317]:   notice: finished - rsc:p-fssapmnt action:start
>>>> call_id:53 pid:5139 exit-code:1 exec-time:60002ms queue-time:0ms
>>>> kernel: [  766.056514] dlm: BFA9FF042AA045F4822C2A6A06020EE9: group
>>>> event done -512 0
>>>> kernel: [  766.056528] dlm: BFA9FF042AA045F4822C2A6A06020EE9: group
>>>> join failed -512 0
>>>> crmd[4320]:   notice: Result of stop operation for p-fssapmnt on
>>>> pipci001: 0 (ok)
>>>> crmd[4320]:   notice: Initiating stop operation dlm_stop_0 locally on
>>>> pipci001
>>>> lrmd[4317]:   notice: executing - rsc:dlm action:stop call_id:56
>>>> dlm_controld[5105]: 766 shutdown ignored, active lockspaces
>>>> lrmd[4317]:  warning: dlm_stop_0 process (PID 5326) timed out
>>>> lrmd[4317]:  warning: dlm_stop_0:5326 - timed out after 100000ms
>>>> lrmd[4317]:   notice: finished - rsc:dlm action:stop call_id:56
>>>> pid:5326 exit-code:1 exec-time:100003ms queue-time:0ms
>>>> crmd[4320]:    error: Result of stop operation for dlm on pipci001:
>>>> Timed Out
>>>> crmd[4320]:  warning: Action 15 (dlm_stop_0) on pipci001 failed
>>>> (target: 0 vs. rc: 1): Error
>>>> crmd[4320]:   notice: Transition aborted by operation dlm_stop_0
>>>> 'modify' on pipci001: Event failed
>>>> crmd[4320]:  warning: Action 15 (dlm_stop_0) on pipci001 failed
>>>> (target: 0 vs. rc: 1): Error
>>>> pengine[4319]:   notice: Watchdog will be used via SBD if fencing is
>>>> required
>>>> pengine[4319]:   notice: On loss of CCM Quorum: Ignore
>>>> pengine[4319]:  warning: Processing failed op stop for dlm:0 on
>>>> pipci001: unknown error (1)
>>>> pengine[4319]:  warning: Processing failed op stop for dlm:0 on
>>>> pipci001: unknown error (1)
>>>> pengine[4319]:  warning: Cluster node pipci001 will be fenced: dlm:0
>>>> failed there
>>>> pengine[4319]:  warning: Processing failed op start for p-fssapmnt:0
>>>> on pipci001: unknown error (1)
>>>> pengine[4319]:   notice: Stop of failed resource dlm:0 is implicit
>>>> after pipci001 is fenced
>>>> pengine[4319]:   notice:  * Fence pipci001
>>>> pengine[4319]:   notice: Stop    sbd-stonith#011(pipci001)
>>>> pengine[4319]:   notice: Stop    dlm:0#011(pipci001)
>>>> crmd[4320]:   notice: Requesting fencing (reboot) of node pipci001
>>>> stonith-ng[4316]:   notice: Client crmd.4320.4c2f757b wants to fence
>>>> (reboot) 'pipci001' with device '(any)'
>>>> stonith-ng[4316]:   notice: Requesting peer fencing (reboot) of
>>>> pipci001
>>>> stonith-ng[4316]:   notice: sbd-stonith can fence (reboot) pipci001:
>>>> dynamic-list
>>>>
>>>>
>>>> -- 
>>>> Regards,
>>>> Muhammad Sharfuddin | +923332144823 | nds.com.pk
>>>>
>>>> On 3/13/2018 1:04 PM, Ulrich Windl wrote:
>>>>> Hi!
>>>>>
>>>>> I'd recommend this:
>>>>> Cleanly boot your nodes, avoiding any manual operation with cluster
>>>>> resources. Keep the logs.
>>>>> Then start your tests, keeping the logs for each.
>>>>> Try to fix issues by reading the logs and adjusting the cluster
>>>>> configuration, and not by starting commands that the cluster should
>>>>> start.
>>>>>
>>>>> We had an 2-node OCFS2 cluster running for quite some time with
>>>>> SLES11, but now the cluster is three nodes. To me the output of
>>>>> "crm_mon -1Arfj" combined with having set record-pending=true was
>>>>> very valuable finding problems.
>>>>>
>>>>> Regards,
>>>>> Ulrich
>>>>>
>>>>>
>>>>>>>> Muhammad Sharfuddin <M.Sharfuddin at nds.com.pk> schrieb am
>>>>>>>> 13.03.2018 um 08:43 in
>>>>> Nachricht <7b773ae9-4209-d246-b5c0-2c8b67e623b3 at nds.com.pk>:
>>>>>> Dear Klaus,
>>>>>>
>>>>>> If I understand you properly then, its a fencing issue, and
>>>>>> whatever I
>>>>>> am facing is "natural" or "by-design" in a two node cluster where
>>>>>> quorum
>>>>>> is incomplete.
>>>>>>
>>>>>> I am quite convinced that you have pointed out right because, when I
>>>>>> start the dlm resource via cluster and then tries to start the ocfs2
>>>>>> file system manually from command line, mount command remains hanged
>>>>>> and
>>>>>> following events are reported in the logs:
>>>>>>
>>>>>>         kernel: [62622.864828] ocfs2: Registered cluster interface
>>>>>> user
>>>>>>         kernel: [62622.884427] dlm: Using TCP for communications
>>>>>>         kernel: [62622.884750] dlm: BFA9FF042AA045F4822C2A6A06020EE9:
>>>>>> joining the lockspace group...
>>>>>>         dlm_controld[17655]: 62627 fence work wait for quorum
>>>>>>         dlm_controld[17655]: 62680 BFA9FF042AA045F4822C2A6A06020EE9
>>>>>> wait
>>>>>> for quorum
>>>>>>
>>>>>> and then following messages keep reported every 5-10 minutes, till I
>>>>>> kill the mount.ocfs2 process:
>>>>>>
>>>>>>         dlm_controld[17655]: 62627 fence work wait for quorum
>>>>>>         dlm_controld[17655]: 62680 BFA9FF042AA045F4822C2A6A06020EE9
>>>>>> wait
>>>>>> for quorum
>>>>>>
>>>>>> I am also very much confused, because yesterday I did the same and
>>>>>> was
>>>>>> able to mount the ocfs2 file system manually from command line(at
>>>>>> least
>>>>>> once), and then unmount the file system manually stop the dlm
>>>>>> resource
>>>>>> from cluster and then complete ocfs2 resource stack(dlm, file
>>>>>> systems)
>>>>>> start/stop successfully via cluster even when only machine was
>>>>>> online.
>>>>>>
>>>>>> In a two-node cluster, which have ocfs2 resources, we can't run the
>>>>>> ocfs2 resources when quorum is incomplete(one node is offline) ?
>>>>>>
>>>>>> -- 
>>>>>> Regards,
>>>>>> Muhammad Sharfuddin
>>>>>>
>>>>>> On 3/12/2018 5:58 PM, Klaus Wenninger wrote:
>>>>>>> On 03/12/2018 01:44 PM, Muhammad Sharfuddin wrote:
>>>>>>>> Hi Klaus,
>>>>>>>>
>>>>>>>> primitive sbd-stonith stonith:external/sbd \
>>>>>>>>             op monitor interval=3000 timeout=20 \
>>>>>>>>             op start interval=0 timeout=240 \
>>>>>>>>             op stop interval=0 timeout=100 \
>>>>>>>>             params sbd_device="/dev/mapper/sbd" \
>>>>>>>>             meta target-role=Started
>>>>>>> Makes more sense now.
>>>>>>> Using pcmk_delay_max would probably be useful here
>>>>>>> to prevent a fence-race.
>>>>>>> That stonith-resource was not in your resource-list below ...
>>>>>>>
>>>>>>>> property cib-bootstrap-options: \
>>>>>>>>             have-watchdog=true \
>>>>>>>>             stonith-enabled=true \
>>>>>>>>             no-quorum-policy=ignore \
>>>>>>>>             stonith-timeout=90 \
>>>>>>>>             startup-fencing=true
>>>>>>> You've set no-quorum-policy=ignore for pacemaker.
>>>>>>> Whether this is a good idea or not in your setup is
>>>>>>> written on another page.
>>>>>>> But isn't dlm directly interfering with corosync so
>>>>>>> that it would get the quorum state from there?
>>>>>>> As you have 2-node set probably on a 2-node-cluster
>>>>>>> this would - after both nodes down - wait for all
>>>>>>> nodes up first.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Klaus
>>>>>>>
>>>>>>>> # ps -eaf |grep sbd
>>>>>>>> root      6129     1  0 17:35 ?        00:00:00 sbd: inquisitor
>>>>>>>> root      6133  6129  0 17:35 ?        00:00:00 sbd: watcher:
>>>>>>>> /dev/mapper/sbd - slot: 1 - uuid:
>>>>>>>> 6e80a337-95db-4608-bd62-d59517f39103
>>>>>>>> root      6134  6129  0 17:35 ?        00:00:00 sbd: watcher:
>>>>>>>> Pacemaker
>>>>>>>> root      6135  6129  0 17:35 ?        00:00:00 sbd: watcher:
>>>>>>>> Cluster
>>>>>>>>
>>>>>>>> This cluster does not start ocfs2 resources when I first
>>>>>>>> intentionally
>>>>>>>> crashed(reboot) both the nodes, then try to start ocfs2 resource
>>>>>>>> while
>>>>>>>> one node is  offline.
>>>>>>>>
>>>>>>>> To fix the issue, I have one permanent solution, bring the other
>>>>>>>> node(offline) online and things get fixed automatically, i.e ocfs2
>>>>>>>> resources mounts.
>>>>>>>>
>>>>>>>> -- 
>>>>>>>> Regards,
>>>>>>>> Muhammad Sharfuddin
>>>>>>>>
>>>>>>>> On 3/12/2018 5:25 PM, Klaus Wenninger wrote:
>>>>>>>>> Hi Muhammad!
>>>>>>>>>
>>>>>>>>> Could you be a little bit more elaborate on your fencing-setup!
>>>>>>>>> I read about you using SBD but I don't see any
>>>>>>>>> sbd-fencing-resource.
>>>>>>>>> For the case you wanted to use watchdog-fencing with SBD this
>>>>>>>>> would require stonith-watchdog-timeout property to be set.
>>>>>>>>> But watchdog-fencing relies on quorum (without 2-node trickery)
>>>>>>>>> and thus wouldn't work on a 2-node-cluster anyway.
>>>>>>>>>
>>>>>>>>> Didn't read through the whole thread - so I might be missing
>>>>>>>>> something ...
>>>>>>>>>
>>>>>>>>> Regards,
>>>>>>>>> Klaus
>>>>>>>>>
>>>>>>>>> On 03/12/2018 12:51 PM, Muhammad Sharfuddin wrote:
>>>>>>>>>> Hello Gang,
>>>>>>>>>>
>>>>>>>>>> as informed, previously cluster was fixed to start the ocfs2
>>>>>>>>>> resources by
>>>>>>>>>>
>>>>>>>>>> a) crm resource start dlm
>>>>>>>>>>
>>>>>>>>>> b) mount/umount the ocfs2 file system manually. (this step was
>>>>>>>>>> the
>>>>>>>>>> fix)
>>>>>>>>>>
>>>>>>>>>> and then starting the clone group(which include dlm, ocfs2 file
>>>>>>>>>> systems) worked fine:
>>>>>>>>>>
>>>>>>>>>> c) crm resource start base-clone.
>>>>>>>>>>
>>>>>>>>>> Now I crash the nodes intentionally and then keep only one node
>>>>>>>>>> online, again cluster stopped starting the ocfs2 resources. I
>>>>>>>>>> again
>>>>>>>>>> tried to follow your instructions i.e
>>>>>>>>>>
>>>>>>>>>> i) crm resource start dlm
>>>>>>>>>>
>>>>>>>>>> then try to mount the ocfs2 file system manually which got
>>>>>>>>>> hanged this
>>>>>>>>>> time(previously manually mounting helped me):
>>>>>>>>>>
>>>>>>>>>> # cat /proc/3966/stack
>>>>>>>>>> [<ffffffffa039f18e>] do_uevent+0x7e/0x200 [dlm]
>>>>>>>>>> [<ffffffffa039fe0a>] new_lockspace+0x80a/0xa70 [dlm]
>>>>>>>>>> [<ffffffffa03a02d9>] dlm_new_lockspace+0x69/0x160 [dlm]
>>>>>>>>>> [<ffffffffa038e758>] user_cluster_connect+0xc8/0x350
>>>>>>>>>> [ocfs2_stack_user]
>>>>>>>>>> [<ffffffffa03c2872>] ocfs2_cluster_connect+0x192/0x240
>>>>>>>>>> [ocfs2_stackglue]
>>>>>>>>>> [<ffffffffa045eefc>] ocfs2_dlm_init+0x31c/0x570 [ocfs2]
>>>>>>>>>> [<ffffffffa04a9983>] ocfs2_fill_super+0xb33/0x1200 [ocfs2]
>>>>>>>>>> [<ffffffff8120e130>] mount_bdev+0x1a0/0x1e0
>>>>>>>>>> [<ffffffff8120ea1a>] mount_fs+0x3a/0x170
>>>>>>>>>> [<ffffffff81228bf2>] vfs_kern_mount+0x62/0x110
>>>>>>>>>> [<ffffffff8122b123>] do_mount+0x213/0xcd0
>>>>>>>>>> [<ffffffff8122bed5>] SyS_mount+0x85/0xd0
>>>>>>>>>> [<ffffffff81614b0a>] entry_SYSCALL_64_fastpath+0x1e/0xb6
>>>>>>>>>> [<ffffffffffffffff>] 0xffffffffffffffff
>>>>>>>>>>
>>>>>>>>>> I killed the mount.ocfs2 process stop(crm resource stop dlm) the
>>>>>>>>>> dlm
>>>>>>>>>> process, and then try to start(crm resource start dlm) the
>>>>>>>>>> dlm(which
>>>>>>>>>> previously always get started successfully), this time dlm didn't
>>>>>>>>>> start and I checked the dlm_controld process
>>>>>>>>>>
>>>>>>>>>> cat /proc/3754/stack
>>>>>>>>>> [<ffffffff8121dc55>] poll_schedule_timeout+0x45/0x60
>>>>>>>>>> [<ffffffff8121f0bc>] do_sys_poll+0x38c/0x4f0
>>>>>>>>>> [<ffffffff8121f2dd>] SyS_poll+0x5d/0xe0
>>>>>>>>>> [<ffffffff81614b0a>] entry_SYSCALL_64_fastpath+0x1e/0xb6
>>>>>>>>>> [<ffffffffffffffff>] 0xffffffffffffffff
>>>>>>>>>>
>>>>>>>>>> Nutshell:
>>>>>>>>>>
>>>>>>>>>> 1 - this cluster is configured to run when single node is online
>>>>>>>>>>
>>>>>>>>>> 2 - this cluster does not start the ocfs2 resources after a
>>>>>>>>>> crash when
>>>>>>>>>> only one node is online.
>>>>>>>>>>
>>>>>>>>>> -- 
>>>>>>>>>> Regards,
>>>>>>>>>> Muhammad Sharfuddin | +923332144823 | nds.com.pk
>>>>>>>>>>
>>>>>>>>>> On 3/12/2018 12:41 PM, Gang He wrote:
>>>>>>>>>>>> Hello Gang,
>>>>>>>>>>>>
>>>>>>>>>>>> to follow your instructions, I started the dlm resource via:
>>>>>>>>>>>>
>>>>>>>>>>>>            crm resource start dlm
>>>>>>>>>>>>
>>>>>>>>>>>> then mount/unmount the ocfs2 file system manually..(which
>>>>>>>>>>>> seems to be
>>>>>>>>>>>> the fix of the situation).
>>>>>>>>>>>>
>>>>>>>>>>>> Now resources are getting started properly on a single node..
>>>>>>>>>>>> I am
>>>>>>>>>>>> happy
>>>>>>>>>>>> as the issue is fixed, but at the same time I am lost because
>>>>>>>>>>>> I have
>>>>>>>>>>>> no idea
>>>>>>>>>>>>
>>>>>>>>>>>> how things get fixed here(merely by mounting/unmounting the
>>>>>>>>>>>> ocfs2
>>>>>>>>>>>> file
>>>>>>>>>>>> systems)
>>>>>>>>>>> >From your description.
>>>>>>>>>>> I just wonder  the DLM resource does not work normally under
>>>>>>>>>>> that
>>>>>>>>>>> situation.
>>>>>>>>>>> Yan/Bin, do you have any comments about two-node cluster? which
>>>>>>>>>>> configuration settings will affect corosync quorum/DLM ?
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Thanks
>>>>>>>>>>> Gang
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>> -- 
>>>>>>>>>>>> Regards,
>>>>>>>>>>>> Muhammad Sharfuddin
>>>>>>>>>>>>
>>>>>>>>>>>> On 3/12/2018 10:59 AM, Gang He wrote:
>>>>>>>>>>>>> Hello Muhammad,
>>>>>>>>>>>>>
>>>>>>>>>>>>> Usually, ocfs2 resource startup failure is caused by mount
>>>>>>>>>>>>> command
>>>>>>>>>>>>> timeout
>>>>>>>>>>>> (or hanged).
>>>>>>>>>>>>> The sample debugging method is,
>>>>>>>>>>>>> remove ocfs2 resource from crm first,
>>>>>>>>>>>>> then mount this file system manually, see if the mount command
>>>>>>>>>>>>> will be
>>>>>>>>>>>> timeout or hanged.
>>>>>>>>>>>>> If this command is hanged, please watch where is mount.ocfs2
>>>>>>>>>>>>> process hanged
>>>>>>>>>>>> via "cat /proc/xxx/stack" command.
>>>>>>>>>>>>> If the back trace is stopped at DLM kernel module, usually
>>>>>>>>>>>>> the root
>>>>>>>>>>>>> cause is
>>>>>>>>>>>> cluster configuration problem.
>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>> Gang
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>> On 3/12/2018 7:32 AM, Gang He wrote:
>>>>>>>>>>>>>>> Hello Muhammad,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I think this problem is not in ocfs2, the cause looks
>>>>>>>>>>>>>>> like the
>>>>>>>>>>>>>>> cluster
>>>>>>>>>>>>>> quorum is missed.
>>>>>>>>>>>>>>> For two-node cluster (does not three-node cluster), if one
>>>>>>>>>>>>>>> node
>>>>>>>>>>>>>>> is offline,
>>>>>>>>>>>>>> the quorum will be missed by default.
>>>>>>>>>>>>>>> So, you should configure two-node related quorum setting
>>>>>>>>>>>>>>> according to the
>>>>>>>>>>>>>> pacemaker manual.
>>>>>>>>>>>>>>> Then, DLM can work normal, and ocfs2 resource can start up.
>>>>>>>>>>>>>> Yes its configured accordingly, no-quorum is set to "ignore".
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> property cib-bootstrap-options: \
>>>>>>>>>>>>>>                  have-watchdog=true \
>>>>>>>>>>>>>>                  stonith-enabled=true \
>>>>>>>>>>>>>>                  stonith-timeout=80 \
>>>>>>>>>>>>>>                  startup-fencing=true \
>>>>>>>>>>>>>>                  no-quorum-policy=ignore
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Thanks
>>>>>>>>>>>>>>> Gang
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> This two node cluster starts resources when both nodes are
>>>>>>>>>>>>>>>> online but
>>>>>>>>>>>>>>>> does not start the ocfs2 resources
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> when one node is offline. e.g if I gracefully stop the
>>>>>>>>>>>>>>>> cluster
>>>>>>>>>>>>>>>> resources
>>>>>>>>>>>>>>>> then stop the pacemaker service on
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> either node, and try to start the ocfs2 resource on the
>>>>>>>>>>>>>>>> online
>>>>>>>>>>>>>>>> node, it
>>>>>>>>>>>>>>>> fails.
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> logs:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> pipci001 pengine[17732]:   notice: Start
>>>>>>>>>>>>>>>> dlm:0#011(pipci001)
>>>>>>>>>>>>>>>> pengine[17732]:   notice: Start
>>>>>>>>>>>>>>>> p-fssapmnt:0#011(pipci001)
>>>>>>>>>>>>>>>> pengine[17732]:   notice: Start
>>>>>>>>>>>>>>>> p-fsusrsap:0#011(pipci001)
>>>>>>>>>>>>>>>> pipci001 pengine[17732]:   notice: Calculated transition 2,
>>>>>>>>>>>>>>>> saving
>>>>>>>>>>>>>>>> inputs in /var/lib/pacemaker/pengine/pe-input-339.bz2
>>>>>>>>>>>>>>>> pipci001 crmd[17733]:   notice: Processing graph 2
>>>>>>>>>>>>>>>> (ref=pe_calc-dc-1520613202-31) derived from
>>>>>>>>>>>>>>>> /var/lib/pacemaker/pengine/pe-input-339.bz2
>>>>>>>>>>>>>>>> crmd[17733]:   notice: Initiating start operation
>>>>>>>>>>>>>>>> dlm_start_0
>>>>>>>>>>>>>>>> locally on
>>>>>>>>>>>>>>>> pipci001
>>>>>>>>>>>>>>>> lrmd[17730]:   notice: executing - rsc:dlm action:start
>>>>>>>>>>>>>>>> call_id:69
>>>>>>>>>>>>>>>> dlm_controld[19019]: 4575 dlm_controld 4.0.7 started
>>>>>>>>>>>>>>>> lrmd[17730]:   notice: finished - rsc:dlm action:start
>>>>>>>>>>>>>>>> call_id:69
>>>>>>>>>>>>>>>> pid:18999 exit-code:0 exec-time:1082ms queue-time:1ms
>>>>>>>>>>>>>>>> crmd[17733]:   notice: Result of start operation for dlm on
>>>>>>>>>>>>>>>> pipci001: 0 (ok)
>>>>>>>>>>>>>>>> crmd[17733]:   notice: Initiating monitor operation
>>>>>>>>>>>>>>>> dlm_monitor_60000
>>>>>>>>>>>>>>>> locally on pipci001
>>>>>>>>>>>>>>>> crmd[17733]:   notice: Initiating start operation
>>>>>>>>>>>>>>>> p-fssapmnt_start_0
>>>>>>>>>>>>>>>> locally on pipci001
>>>>>>>>>>>>>>>> lrmd[17730]:   notice: executing - rsc:p-fssapmnt
>>>>>>>>>>>>>>>> action:start
>>>>>>>>>>>>>>>> call_id:71
>>>>>>>>>>>>>>>> Filesystem(p-fssapmnt)[19052]: INFO: Running start for
>>>>>>>>>>>>>>>> /dev/mapper/sapmnt on /sapmnt
>>>>>>>>>>>>>>>> kernel: [ 4576.529938] dlm: Using TCP for communications
>>>>>>>>>>>>>>>> kernel: [ 4576.530233] dlm:
>>>>>>>>>>>>>>>> BFA9FF042AA045F4822C2A6A06020EE9:
>>>>>>>>>>>>>>>> joining
>>>>>>>>>>>>>>>> the lockspace group.
>>>>>>>>>>>>>>>> dlm_controld[19019]: 4629 fence work wait for quorum
>>>>>>>>>>>>>>>> dlm_controld[19019]: 4634 BFA9FF042AA045F4822C2A6A06020EE9
>>>>>>>>>>>>>>>> wait
>>>>>>>>>>>>>>>> for quorum
>>>>>>>>>>>>>>>> lrmd[17730]:  warning: p-fssapmnt_start_0 process (PID
>>>>>>>>>>>>>>>> 19052)
>>>>>>>>>>>>>>>> timed out
>>>>>>>>>>>>>>>> kernel: [ 4636.418223] dlm:
>>>>>>>>>>>>>>>> BFA9FF042AA045F4822C2A6A06020EE9:
>>>>>>>>>>>>>>>> group
>>>>>>>>>>>>>>>> event done -512 0
>>>>>>>>>>>>>>>> kernel: [ 4636.418227] dlm:
>>>>>>>>>>>>>>>> BFA9FF042AA045F4822C2A6A06020EE9:
>>>>>>>>>>>>>>>> group join
>>>>>>>>>>>>>>>> failed -512 0
>>>>>>>>>>>>>>>> lrmd[17730]:  warning: p-fssapmnt_start_0:19052 - timed out
>>>>>>>>>>>>>>>> after 60000ms
>>>>>>>>>>>>>>>> lrmd[17730]:   notice: finished - rsc:p-fssapmnt
>>>>>>>>>>>>>>>> action:start
>>>>>>>>>>>>>>>> call_id:71
>>>>>>>>>>>>>>>> pid:19052 exit-code:1 exec-time:60002ms queue-time:0ms
>>>>>>>>>>>>>>>> kernel: [ 4636.420628] ocfs2: Unmounting device (254,1) on
>>>>>>>>>>>>>>>> (node 0)
>>>>>>>>>>>>>>>> crmd[17733]:    error: Result of start operation for
>>>>>>>>>>>>>>>> p-fssapmnt on
>>>>>>>>>>>>>>>> pipci001: Timed Out
>>>>>>>>>>>>>>>> crmd[17733]:  warning: Action 11 (p-fssapmnt_start_0) on
>>>>>>>>>>>>>>>> pipci001 failed
>>>>>>>>>>>>>>>> (target: 0 vs. rc: 1): Error
>>>>>>>>>>>>>>>> crmd[17733]:   notice: Transition aborted by operation
>>>>>>>>>>>>>>>> p-fssapmnt_start_0 'modify' on pipci001: Event failed
>>>>>>>>>>>>>>>> crmd[17733]:  warning: Action 11 (p-fssapmnt_start_0) on
>>>>>>>>>>>>>>>> pipci001 failed
>>>>>>>>>>>>>>>> (target: 0 vs. rc: 1): Error
>>>>>>>>>>>>>>>> crmd[17733]:   notice: Transition 2 (Complete=5, Pending=0,
>>>>>>>>>>>>>>>> Fired=0,
>>>>>>>>>>>>>>>> Skipped=0, Incomplete=6,
>>>>>>>>>>>>>>>> Source=/var/lib/pacemaker/pengine/pe-input-339.bz2):
>>>>>>>>>>>>>>>> Complete
>>>>>>>>>>>>>>>> pengine[17732]:   notice: Watchdog will be used via SBD if
>>>>>>>>>>>>>>>> fencing is
>>>>>>>>>>>>>>>> required
>>>>>>>>>>>>>>>> pengine[17732]:   notice: On loss of CCM Quorum: Ignore
>>>>>>>>>>>>>>>> pengine[17732]:  warning: Processing failed op start for
>>>>>>>>>>>>>>>> p-fssapmnt:0 on
>>>>>>>>>>>>>>>> pipci001: unknown error (1)
>>>>>>>>>>>>>>>> pengine[17732]:  warning: Processing failed op start for
>>>>>>>>>>>>>>>> p-fssapmnt:0 on
>>>>>>>>>>>>>>>> pipci001: unknown error (1)
>>>>>>>>>>>>>>>> pengine[17732]:  warning: Forcing base-clone away from
>>>>>>>>>>>>>>>> pipci001
>>>>>>>>>>>>>>>> after
>>>>>>>>>>>>>>>> 1000000 failures (max=2)
>>>>>>>>>>>>>>>> pengine[17732]:  warning: Forcing base-clone away from
>>>>>>>>>>>>>>>> pipci001
>>>>>>>>>>>>>>>> after
>>>>>>>>>>>>>>>> 1000000 failures (max=2)
>>>>>>>>>>>>>>>> pengine[17732]:   notice: Stop    dlm:0#011(pipci001)
>>>>>>>>>>>>>>>> pengine[17732]:   notice: Stop
>>>>>>>>>>>>>>>> p-fssapmnt:0#011(pipci001)
>>>>>>>>>>>>>>>> pengine[17732]:   notice: Calculated transition 3, saving
>>>>>>>>>>>>>>>> inputs in
>>>>>>>>>>>>>>>> /var/lib/pacemaker/pengine/pe-input-340.bz2
>>>>>>>>>>>>>>>> pengine[17732]:   notice: Watchdog will be used via SBD if
>>>>>>>>>>>>>>>> fencing is
>>>>>>>>>>>>>>>> required
>>>>>>>>>>>>>>>> pengine[17732]:   notice: On loss of CCM Quorum: Ignore
>>>>>>>>>>>>>>>> pengine[17732]:  warning: Processing failed op start for
>>>>>>>>>>>>>>>> p-fssapmnt:0 on
>>>>>>>>>>>>>>>> pipci001: unknown error (1)
>>>>>>>>>>>>>>>> pengine[17732]:  warning: Processing failed op start for
>>>>>>>>>>>>>>>> p-fssapmnt:0 on
>>>>>>>>>>>>>>>> pipci001: unknown error (1)
>>>>>>>>>>>>>>>> pengine[17732]:  warning: Forcing base-clone away from
>>>>>>>>>>>>>>>> pipci001
>>>>>>>>>>>>>>>> after
>>>>>>>>>>>>>>>> 1000000 failures (max=2)
>>>>>>>>>>>>>>>> pipci001 pengine[17732]:  warning: Forcing base-clone away
>>>>>>>>>>>>>>>> from
>>>>>>>>>>>>>>>> pipci001
>>>>>>>>>>>>>>>> after 1000000 failures (max=2)
>>>>>>>>>>>>>>>> pengine[17732]:   notice: Stop    dlm:0#011(pipci001)
>>>>>>>>>>>>>>>> pengine[17732]:   notice: Stop
>>>>>>>>>>>>>>>> p-fssapmnt:0#011(pipci001)
>>>>>>>>>>>>>>>> pengine[17732]:   notice: Calculated transition 4, saving
>>>>>>>>>>>>>>>> inputs in
>>>>>>>>>>>>>>>> /var/lib/pacemaker/pengine/pe-input-341.bz2
>>>>>>>>>>>>>>>> crmd[17733]:   notice: Processing graph 4
>>>>>>>>>>>>>>>> (ref=pe_calc-dc-1520613263-36)
>>>>>>>>>>>>>>>> derived from /var/lib/pacemaker/pengine/pe-input-341.bz2
>>>>>>>>>>>>>>>> crmd[17733]:   notice: Initiating stop operation
>>>>>>>>>>>>>>>> p-fssapmnt_stop_0
>>>>>>>>>>>>>>>> locally on pipci001
>>>>>>>>>>>>>>>> lrmd[17730]:   notice: executing - rsc:p-fssapmnt
>>>>>>>>>>>>>>>> action:stop
>>>>>>>>>>>>>>>> call_id:72
>>>>>>>>>>>>>>>> Filesystem(p-fssapmnt)[19189]: INFO: Running stop for
>>>>>>>>>>>>>>>> /dev/mapper/sapmnt
>>>>>>>>>>>>>>>> on /sapmnt
>>>>>>>>>>>>>>>> pipci001 lrmd[17730]:   notice: finished - rsc:p-fssapmnt
>>>>>>>>>>>>>>>> action:stop
>>>>>>>>>>>>>>>> call_id:72 pid:19189 exit-code:0 exec-time:83ms
>>>>>>>>>>>>>>>> queue-time:0ms
>>>>>>>>>>>>>>>> pipci001 crmd[17733]:   notice: Result of stop operation
>>>>>>>>>>>>>>>> for
>>>>>>>>>>>>>>>> p-fssapmnt
>>>>>>>>>>>>>>>> on pipci001: 0 (ok)
>>>>>>>>>>>>>>>> crmd[17733]:   notice: Initiating stop operation dlm_stop_0
>>>>>>>>>>>>>>>> locally on
>>>>>>>>>>>>>>>> pipci001
>>>>>>>>>>>>>>>> pipci001 lrmd[17730]:   notice: executing - rsc:dlm
>>>>>>>>>>>>>>>> action:stop
>>>>>>>>>>>>>>>> call_id:74
>>>>>>>>>>>>>>>> pipci001 dlm_controld[19019]: 4636 shutdown ignored, active
>>>>>>>>>>>>>>>> lockspaces
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> resource configuration:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> primitive p-fssapmnt Filesystem \
>>>>>>>>>>>>>>>>                  params device="/dev/mapper/sapmnt"
>>>>>>>>>>>>>>>> directory="/sapmnt"
>>>>>>>>>>>>>>>> fstype=ocfs2 \
>>>>>>>>>>>>>>>>                  op monitor interval=20 timeout=40 \
>>>>>>>>>>>>>>>>                  op start timeout=60 interval=0 \
>>>>>>>>>>>>>>>>                  op stop timeout=60 interval=0
>>>>>>>>>>>>>>>> primitive dlm ocf:pacemaker:controld \
>>>>>>>>>>>>>>>>                  op monitor interval=60 timeout=60 \
>>>>>>>>>>>>>>>>                  op start interval=0 timeout=90 \
>>>>>>>>>>>>>>>>                  op stop interval=0 timeout=100
>>>>>>>>>>>>>>>> clone base-clone base-group \
>>>>>>>>>>>>>>>>                  meta interleave=true target-role=Started
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> cluster properties:
>>>>>>>>>>>>>>>> property cib-bootstrap-options: \
>>>>>>>>>>>>>>>>                  have-watchdog=true \
>>>>>>>>>>>>>>>>                  stonith-enabled=true \
>>>>>>>>>>>>>>>>                  stonith-timeout=80 \
>>>>>>>>>>>>>>>>                  startup-fencing=true \
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Software versions:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> kernel version: 4.4.114-94.11-default
>>>>>>>>>>>>>>>> pacemaker-1.1.16-4.8.x86_64
>>>>>>>>>>>>>>>> corosync-2.3.6-9.5.1.x86_64
>>>>>>>>>>>>>>>> ocfs2-kmp-default-4.4.114-94.11.3.x86_64
>>>>>>>>>>>>>>>> ocfs2-tools-1.8.5-1.35.x86_64
>>>>>>>>>>>>>>>> dlm-kmp-default-4.4.114-94.11.3.x86_64
>>>>>>>>>>>>>>>> libdlm3-4.0.7-1.28.x86_64
>>>>>>>>>>>>>>>> libdlm-4.0.7-1.28.x86_64
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>> Muhammad Sharfuddin
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> ---
>>>>>>>>>>>>>>>> This email has been checked for viruses by Avast antivirus
>>>>>>>>>>>>>>>> software.
>>>>>>>>>>>>>>>> https://www.avast.com/antivirus
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>>> Users mailing list: Users at clusterlabs.org
>>>>>>>>>>>>>>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Project Home: http://www.clusterlabs.org
>>>>>>>>>>>>>>>> Getting started:
>>>>>>>>>>>>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>>>>>>>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>>> Users mailing list: Users at clusterlabs.org
>>>>>>>>>>>>>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Project Home: http://www.clusterlabs.org
>>>>>>>>>>>>>>> Getting started:
>>>>>>>>>>>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>>>>>>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>> -- 
>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>> Muhammad Sharfuddin
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>>> Users mailing list: Users at clusterlabs.org
>>>>>>>>>>>>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Project Home: http://www.clusterlabs.org
>>>>>>>>>>>>>> Getting started:
>>>>>>>>>>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>>>>>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>>> Users mailing list: Users at clusterlabs.org
>>>>>>>>>>>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>>>>>>>>>>>
>>>>>>>>>>>>> Project Home: http://www.clusterlabs.org
>>>>>>>>>>>>> Getting started:
>>>>>>>>>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>>>>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>>>>>>>>>
>>>>>>>>>>>> _______________________________________________
>>>>>>>>>>>> Users mailing list: Users at clusterlabs.org
>>>>>>>>>>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>>>>>>>>>>
>>>>>>>>>>>> Project Home: http://www.clusterlabs.org
>>>>>>>>>>>> Getting started:
>>>>>>>>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>>>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>>>>>>> _______________________________________________
>>>>>>>>>>> Users mailing list: Users at clusterlabs.org
>>>>>>>>>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>>>>>>>>>
>>>>>>>>>>> Project Home: http://www.clusterlabs.org
>>>>>>>>>>> Getting started:
>>>>>>>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>>>>>>>
>>>>>>>>>> _______________________________________________
>>>>>>>>>> Users mailing list: Users at clusterlabs.org
>>>>>>>>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>>>>>>>>
>>>>>>>>>> Project Home: http://www.clusterlabs.org
>>>>>>>>>> Getting started:
>>>>>>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>>>> ---
>>>>>>>> This email has been checked for viruses by Avast antivirus
>>>>>>>> software.
>>>>>>>> https://www.avast.com/antivirus
>>>>>>>>
>>>>>>>> _______________________________________________
>>>>>>>> Users mailing list: Users at clusterlabs.org
>>>>>>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>>>>>>
>>>>>>>> Project Home: http://www.clusterlabs.org
>>>>>>>> Getting started:
>>>>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>> _______________________________________________
>>>>>> Users mailing list: Users at clusterlabs.org
>>>>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>>>>
>>>>>> Project Home: http://www.clusterlabs.org
>>>>>> Getting started:
>>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>> Bugs: http://bugs.clusterlabs.org
>>>>> _______________________________________________
>>>>> Users mailing list: Users at clusterlabs.org
>>>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>>>
>>>>> Project Home: http://www.clusterlabs.org
>>>>> Getting started:
>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>
>>>> _______________________________________________
>>>> Users mailing list: Users at clusterlabs.org
>>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>>
>>>> Project Home: http://www.clusterlabs.org
>>>> Getting started:
>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>> Bugs: http://bugs.clusterlabs.org
>   
>
>


---
This email has been checked for viruses by Avast antivirus software.
https://www.avast.com/antivirus

More information about the Users mailing list