[ClusterLabs] Antw: [EXT] Multiple nfsserver resource groups

Mon Mar 9 05:38:04 EDT 2020

Thanks for the advice.
We haven’t had any issues with the time it takes to prepare the exportfs resources so far, and we've been running this setup for 2 years now, but I will keep it in mind as we increase the number of exportfs resources. I have already implemented the solution discussed and merged all filesystems and exports into one resource group and everything looks good.

For your problem, what is the order in which your resources startup/shutdown?
Is your nfs info dir a filesystem resource or an LVM resource?
Do you have an nfsnotify resource in place?
We have found that the order in which resources startup/shutdown without any problems is to have the nfsnotify resource stop first, then stop the exportfs resources, nfsserver after that, filesystems (one of which is the nfs shared info dir) and finally the virtual IP resource. Startup is the reverse of that.

-----Original Message-----
From: Ulrich Windl <Ulrich.Windl at rz.uni-regensburg.de> 
Sent: Monday, March 9, 2020 9:26 AM
To: users at clusterlabs.org; christoforos at globalreach.com
Subject: Antw: [EXT] [ClusterLabs] Multiple nfsserver resource groups

>>> "Christoforos Christoforou" <christoforos at globalreach.com> schrieb 
>>> am
06.03.2020 um 18:56 in Nachricht
<25205_1583517421_5E628EE5_25205_75_1_01e001d5f3e0$804a6240$80df26c0$@globalreac
.com>:
> Hello,
> 
>  
> 
> We have a PCS cluster running on 2 CentOS 7 nodes, exposing 2 NFSv3 
> volumes which are then mounted to multiple servers (around 8).
> 
> We want to have 2 more sets of additional shared NFS volumes, for a 
> total
of
> 6.
> 
>  
> 
> I have successfully configured 3 resource groups, with each group 
> having
the
> following resources:
> 
> *	1x ocf_heartbeat_IPaddr2 resource for the Virtual IP that exposes
> the NFS share assigned to its own NIC.
> *	3x ocf_heartbeat_Filesystem resources (1 is for the
> nfs_shared_infodir and the other 2 are the ones exposed via the NFS server)
> *	1x ocf_heartbeat_nfsserver resource that uses the aforementioned
> nfs_shared_infodir.
> *	2x ocf_heartbeat_exportfs resources that expose the other 2
> filesystems as NFS shares.
> *	1x ocf_heartbeat_nfsnotify resource that has the Virtual IP set as
> its own source_host.
> 
>  
> 
> All 9 filesystem volumes are mounted via iSCSI to the PCS nodes in 
> /dev/mapper/mpathX
> 
> So the structure is like so:
> 
> Resource group 1:
> 
> *	/dev/mapper/mpatha ‑ shared volume 1
> *	/dev/mapper/mpathb ‑ shared volume 2
> *	/dev/mapper/mpathc ‑ nfs_shared_infodir for resource group 1
> 
> Resource group 2:
> 
> *	/dev/mapper/mpathd ‑ shared volume 3
> *	/dev/mapper/mpathe ‑ shared volume 4
> *	/dev/mapper/mpathf ‑ nfs_shared_infodir for resource group 2
> 
> Resource group 3:
> 
> *	/dev/mapper/mpathg ‑ shared volume 5
> *	/dev/mapper/mpathh ‑ shared volume 6
> *	/dev/mapper/mpathi ‑ nfs_shared_infodir for resource group 3
> 
>  
> 
> My concern is that when I run a df command on the active node, the 
> last ocf_heartbeat_nfsserver volume (/dev/mapper/mpathi) mounted to
/var/lib/nfs.
> I understand that I cannot change this, but I can change the location 
> of
the
> rpc_pipefs folder.
> 
>  
> 
> I have had this setup running with 2 resource groups in our 
> development environment, and have not noticed any issues, but since 
> we're planning to move to production and add a 3rd resource group, I 
> want to make sure that this setup will not cause any issues. I am by 
> no means an expert on NFS, so some insight is appreciated.
> 
>  
> 
> If this kind of setup is not supported or recommended, I have 2 
> alternate plans in mind:
> 
> 1.	Have all resources in the same resource group, in a setup that will
> look like this:
> 
> a.	1x ocf_heartbeat_IPaddr2 resource for the Virtual IP that exposes
> the NFS share.
> b.	7x ocf_heartbeat_Filesystem resources (1 is for the
> nfs_shared_infodir and 6 exposed via the NFS server)
> c.	1x ocf_heartbeat_nfsserver resource that uses the aforementioned
> nfs_shared_infodir.
> d.	6x ocf_heartbeat_exportfs resources that expose the other 6
> filesystems as NFS shares. Use the clientspec option to restrict to 
> IPs and prevent unwanted mounts.
> e.	1x ocf_heartbeat_nfsnotify resource that has the Virtual IP set as
> its own source_host.
> 
> 2.	Setup 2 more clusters to accommodate our needs
> 
>  
> 
> I really want to avoid #2, due to the fact that it will be overkill 
> for our case.

Things you might consider is to get reid of the groups and use explicit colocation and orderings. The advantages will be that you can execute several agents in parallel (e.g. prepare all fileysstems in parallel). In the past we had made the experience that exportfs resources can take quite some time and if you have like 20 or more of them, it delays the shutdown/startup significatly.
So we moved to using netgroups provided by LDAP instead, and we could reduce the number of exportfs statements drastically.
However we have one odd problem (SLES12 SP4): The NFS resource using systemd does not shut down clearly due some unmount issue related the shared info dir.

> 
> Thanks
> 
>  
> 
> Christoforos Christoforou
> 
> Senior Systems Administrator
> 
> Global Reach Internet Productions
> 
>  <http://www.twitter.com/globalreach> Twitter | 
> <http://www.facebook.com/globalreach> Facebook | 
> <https://www.linkedin.com/company/global‑reach‑internet‑productions>
> LinkedIn
> 
> p (515) 996‑0996 |  <http://www.globalreach.com/> globalreach.com
> 
>  