[ClusterLabs] Antw: [EXT] Multiple nfsserver resource groups

Mon Mar 9 06:37:08 EDT 2020

>>> "Christoforos Christoforou" <christoforos at globalreach.com> schrieb am
09.03.2020 um 10:38 in Nachricht
<02ff01d5f5f6$7024af70$506e0e50$@globalreach.com>:
> Thanks for the advice.
> We haven’t had any issues with the time it takes to prepare the exportfs 
> resources so far, and we've been running this setup for 2 years now, but I 
> will keep it in mind as we increase the number of exportfs resources. I have

> already implemented the solution discussed and merged all filesystems and 
> exports into one resource group and everything looks good.
> 
> For your problem, what is the order in which your resources 
> startup/shutdown?

Start: RAID1, LVM, fs, nsfsserver, exportfs, IP address (stop is the other way
'round)

> Is your nfs info dir a filesystem resource or an LVM resource?

We have everything as LVs (see above)

> Do you have an nfsnotify resource in place?

Not explicitly. We are using NFSv3 only.

> We have found that the order in which resources startup/shutdown without any

> problems is to have the nfsnotify resource stop first, then stop the
exportfs 
> resources, nfsserver after that, filesystems (one of which is the nfs shared

> info dir) and finally the virtual IP resource. Startup is the reverse of 
> that.

Regards,
Ulrich

> 
> -----Original Message-----
> From: Ulrich Windl <Ulrich.Windl at rz.uni-regensburg.de> 
> Sent: Monday, March 9, 2020 9:26 AM
> To: users at clusterlabs.org; christoforos at globalreach.com 
> Subject: Antw: [EXT] [ClusterLabs] Multiple nfsserver resource groups
> 
>>>> "Christoforos Christoforou" <christoforos at globalreach.com> schrieb 
>>>> am
> 06.03.2020 um 18:56 in Nachricht
>
<25205_1583517421_5E628EE5_25205_75_1_01e001d5f3e0$804a6240$80df26c0$@globalr
> eac
> .com>:
>> Hello,
>> 
>>  
>> 
>> We have a PCS cluster running on 2 CentOS 7 nodes, exposing 2 NFSv3 
>> volumes which are then mounted to multiple servers (around 8).
>> 
>> We want to have 2 more sets of additional shared NFS volumes, for a 
>> total
> of
>> 6.
>> 
>>  
>> 
>> I have successfully configured 3 resource groups, with each group 
>> having
> the
>> following resources:
>> 
>> *	1x ocf_heartbeat_IPaddr2 resource for the Virtual IP that exposes
>> the NFS share assigned to its own NIC.
>> *	3x ocf_heartbeat_Filesystem resources (1 is for the
>> nfs_shared_infodir and the other 2 are the ones exposed via the NFS
server)
>> *	1x ocf_heartbeat_nfsserver resource that uses the aforementioned
>> nfs_shared_infodir.
>> *	2x ocf_heartbeat_exportfs resources that expose the other 2
>> filesystems as NFS shares.
>> *	1x ocf_heartbeat_nfsnotify resource that has the Virtual IP set as
>> its own source_host.
>> 
>>  
>> 
>> All 9 filesystem volumes are mounted via iSCSI to the PCS nodes in 
>> /dev/mapper/mpathX
>> 
>> So the structure is like so:
>> 
>> Resource group 1:
>> 
>> *	/dev/mapper/mpatha ‑ shared volume 1
>> *	/dev/mapper/mpathb ‑ shared volume 2
>> *	/dev/mapper/mpathc ‑ nfs_shared_infodir for resource group 1
>> 
>> Resource group 2:
>> 
>> *	/dev/mapper/mpathd ‑ shared volume 3
>> *	/dev/mapper/mpathe ‑ shared volume 4
>> *	/dev/mapper/mpathf ‑ nfs_shared_infodir for resource group 2
>> 
>> Resource group 3:
>> 
>> *	/dev/mapper/mpathg ‑ shared volume 5
>> *	/dev/mapper/mpathh ‑ shared volume 6
>> *	/dev/mapper/mpathi ‑ nfs_shared_infodir for resource group 3
>> 
>>  
>> 
>> My concern is that when I run a df command on the active node, the 
>> last ocf_heartbeat_nfsserver volume (/dev/mapper/mpathi) mounted to
> /var/lib/nfs.
>> I understand that I cannot change this, but I can change the location 
>> of
> the
>> rpc_pipefs folder.
>> 
>>  
>> 
>> I have had this setup running with 2 resource groups in our 
>> development environment, and have not noticed any issues, but since 
>> we're planning to move to production and add a 3rd resource group, I 
>> want to make sure that this setup will not cause any issues. I am by 
>> no means an expert on NFS, so some insight is appreciated.
>> 
>>  
>> 
>> If this kind of setup is not supported or recommended, I have 2 
>> alternate plans in mind:
>> 
>> 1.	Have all resources in the same resource group, in a setup that will
>> look like this:
>> 
>> a.	1x ocf_heartbeat_IPaddr2 resource for the Virtual IP that exposes
>> the NFS share.
>> b.	7x ocf_heartbeat_Filesystem resources (1 is for the
>> nfs_shared_infodir and 6 exposed via the NFS server)
>> c.	1x ocf_heartbeat_nfsserver resource that uses the aforementioned
>> nfs_shared_infodir.
>> d.	6x ocf_heartbeat_exportfs resources that expose the other 6
>> filesystems as NFS shares. Use the clientspec option to restrict to 
>> IPs and prevent unwanted mounts.
>> e.	1x ocf_heartbeat_nfsnotify resource that has the Virtual IP set as
>> its own source_host.
>> 
>> 2.	Setup 2 more clusters to accommodate our needs
>> 
>>  
>> 
>> I really want to avoid #2, due to the fact that it will be overkill 
>> for our case.
> 
> Things you might consider is to get reid of the groups and use explicit 
> colocation and orderings. The advantages will be that you can execute
several 
> agents in parallel (e.g. prepare all fileysstems in parallel). In the past
we 
> had made the experience that exportfs resources can take quite some time and

> if you have like 20 or more of them, it delays the shutdown/startup 
> significatly.
> So we moved to using netgroups provided by LDAP instead, and we could reduce

> the number of exportfs statements drastically.
> However we have one odd problem (SLES12 SP4): The NFS resource using systemd

> does not shut down clearly due some unmount issue related the shared info 
> dir.
> 
>> 
>> Thanks
>> 
>>  
>> 
>> Christoforos Christoforou
>> 
>> Senior Systems Administrator
>> 
>> Global Reach Internet Productions
>> 
>>  <http://www.twitter.com/globalreach> Twitter | 
>> <http://www.facebook.com/globalreach> Facebook | 
>> <https://www.linkedin.com/company/global‑reach‑internet‑productions>
>> LinkedIn
>> 
>> p (515) 996‑0996 |  <http://www.globalreach.com/> globalreach.com
>> 
>>  