[ClusterLabs] Multiple nfsserver resource groups

Fri Mar 6 17:09:55 EST 2020

On March 6, 2020 11:58:33 PM GMT+02:00, Christoforos Christoforou <christoforos at globalreach.com> wrote:
>>I don't get it.
>>Why the last nfs_shared_infodir  will be mounted  on /var/lib/nfs  ?
>>As  far as I remember, you set that dir in any location you want as a
>Filesystem resource. 
>The dir you set in the resource options is indeed the dir used, but
>apparently the nfsserver daemon will mount that dir in /var/lib/nfs. I
>believe it's the last one because it's the last instance of the
>nfsdaemon that starts, so it "steals" the /var/lib/nfs mount from the
>other 2 that started before.
>
>From the ocf_heartbeat_nfsserver manpage (
>https://manpages.debian.org/testing/resource-agents/ocf_heartbeat_nfsserver.7.en.html
>) :
>"nfs_shared_infodir
>The nfsserver resource agent will save nfs related information in this
>specific directory. And this directory must be able to fail-over before
>nfsserver itself.
>(optional, string, no default)
>rpcpipefs_dir
>The mount point for the sunrpc file system. Default is
>/var/lib/nfs/rpc_pipefs. This script will mount (bind)
>nfs_shared_infodir on /var/lib/nfs/ (cannot be changed), and this
>script will mount the sunrpc file system on /var/lib/nfs/rpc_pipefs
>(default, can be changed by this parameter). If you want to move only
>rpc_pipefs/ (e.g. to keep rpc_pipefs/ local) from default, please set
>this value.
>(optional, string, default "/var/lib/nfs/rpc_pipefs")"
>
>>Note: You do realize  that mpathi on 1 host can be turned into mpathZ
>on the other. Either use alias for each wwid, or do not use  friendly
>names  on those cluster nodes.
>I know, wwid bindings are set to be identical on both nodes, so it's
>not an issue.
>>Also, I would use LVM resource with 'exclusive=true' in order to avoid
>accidental activation of the LV on the other node.
>I remember trying out LVM resource when I was doing initial testing but
>that was 2 years ago, so I can't remember why I went with a filesystem
>resource instead.
>I will revisit that, thanks.
>
>Any other insight on the nfs_shared_infodir situation?
>
>-----Original Message-----
>From: Strahil Nikolov <hunter86_bg at yahoo.com> 
>Sent: Friday, March 6, 2020 11:20 PM
>To: christoforos at globalreach.com; Cluster Labs - All topics related to
>open-source clustering welcomed <users at clusterlabs.org>
>Subject: Re: [ClusterLabs] Multiple nfsserver resource groups
>
>On March 6, 2020 7:56:00 PM GMT+02:00, Christoforos Christoforou
><christoforos at globalreach.com> wrote:
>>Hello,
>>
>> 
>>
>>We have a PCS cluster running on 2 CentOS 7 nodes, exposing 2 NFSv3 
>>volumes which are then mounted to multiple servers (around 8).
>>
>>We want to have 2 more sets of additional shared NFS volumes, for a 
>>total of 6.
>>
>> 
>>
>>I have successfully configured 3 resource groups, with each group 
>>having the following resources:
>>
>>*	1x ocf_heartbeat_IPaddr2 resource for the Virtual IP that exposes
>>the NFS share assigned to its own NIC.
>>*	3x ocf_heartbeat_Filesystem resources (1 is for the
>>nfs_shared_infodir and the other 2 are the ones exposed via the NFS
>>server)
>>*	1x ocf_heartbeat_nfsserver resource that uses the aforementioned
>>nfs_shared_infodir.
>>*	2x ocf_heartbeat_exportfs resources that expose the other 2
>>filesystems as NFS shares.
>>*	1x ocf_heartbeat_nfsnotify resource that has the Virtual IP set as
>>its own source_host.
>>
>> 
>>
>>All 9 filesystem volumes are mounted via iSCSI to the PCS nodes in 
>>/dev/mapper/mpathX
>>
>>So the structure is like so:
>>
>>Resource group 1:
>>
>>*	/dev/mapper/mpatha - shared volume 1
>>*	/dev/mapper/mpathb - shared volume 2
>>*	/dev/mapper/mpathc - nfs_shared_infodir for resource group 1
>>
>>Resource group 2:
>>
>>*	/dev/mapper/mpathd - shared volume 3
>>*	/dev/mapper/mpathe - shared volume 4
>>*	/dev/mapper/mpathf - nfs_shared_infodir for resource group 2
>>
>>Resource group 3:
>>
>>*	/dev/mapper/mpathg - shared volume 5
>>*	/dev/mapper/mpathh - shared volume 6
>>*	/dev/mapper/mpathi - nfs_shared_infodir for resource group 3
>>
>> 
>>
>>My concern is that when I run a df command on the active node, the
>last 
>>ocf_heartbeat_nfsserver volume (/dev/mapper/mpathi) mounted to 
>>/var/lib/nfs.
>>I understand that I cannot change this, but I can change the location 
>>of the rpc_pipefs folder.
>>
>> 
>>
>>I have had this setup running with 2 resource groups in our
>development 
>>environment, and have not noticed any issues, but since we're planning
>
>>to move to production and add a 3rd resource group, I want to make
>sure 
>>that this setup will not cause any issues. I am by no means an expert 
>>on NFS, so some insight is appreciated.
>>
>> 
>>
>>If this kind of setup is not supported or recommended, I have 2 
>>alternate plans in mind:
>>
>>1.	Have all resources in the same resource group, in a setup that will
>>look like this:
>>
>>a.	1x ocf_heartbeat_IPaddr2 resource for the Virtual IP that exposes
>>the NFS share.
>>b.	7x ocf_heartbeat_Filesystem resources (1 is for the
>>nfs_shared_infodir and 6 exposed via the NFS server)
>>c.	1x ocf_heartbeat_nfsserver resource that uses the aforementioned
>>nfs_shared_infodir.
>>d.	6x ocf_heartbeat_exportfs resources that expose the other 6
>>filesystems as NFS shares. Use the clientspec option to restrict to
>IPs 
>>and prevent unwanted mounts.
>>e.	1x ocf_heartbeat_nfsnotify resource that has the Virtual IP set as
>>its own source_host.
>>
>>2.	Setup 2 more clusters to accommodate our needs
>>
>> 
>>
>>I really want to avoid #2, due to the fact that it will be overkill
>for 
>>our case.
>>
>>Thanks
>>
>> 
>>
>>Christoforos Christoforou
>>
>>Senior Systems Administrator
>>
>>Global Reach Internet Productions
>>
>> <http://www.twitter.com/globalreach> Twitter | 
>><http://www.facebook.com/globalreach> Facebook | 
>><https://www.linkedin.com/company/global-reach-internet-productions>
>>LinkedIn
>>
>>p (515) 996-0996 |  <http://www.globalreach.com/> globalreach.com
>>
>> 
>
>I don't get it.
>Why the last nfs_shared_infodir  will be mounted  on /var/lib/nfs  ?
>As  far as I remember, you set that dir in any location you want as a
>Filesystem resource.
>
>Note: You do realize  that mpathi on 1 host can be turned into mpathZ
>on the other. Either use alias for each wwid, or do not use  friendly
>names  on those cluster nodes.
>Also, I would use LVM resource with 'exclusive=true' in order to avoid
>accidental activation of the LV on the other node.
>
>Best Regards,
>Strahil Nikolov

Hm...
I never noticed that behaviour, but it make sense.
When you don't use a cluster - there  is only 1 NFS server on the system and everything else is controlled via the exports.

I guess  you will need to try to combine all 3 groups into a single resource group with 1   'nfssserver' and multiple  exports .

Best Regards,
Strahil Nikolov