[Pacemaker] nfsv4 grace period

[Dennis Jacobfeuerborn]

On 09.02.2014 13:58, Dennis Jacobfeuerborn wrote:
> On 09.02.2014 08:33, Michael Schwartzkopff wrote:
>> Am Sonntag, 9. Februar 2014, 02:56:55 schrieb Dennis Jacobfeuerborn:
>>> Hi,
>>> i have setup a nfsv3 HA cluster before and that works fine but now I'm
>>> trying to move to v4 and run into problems with the lease grace period.
>>> The grace period on CentOS 6 is 90 seconds and that limits how quickly
>>> the fail-over can happen. The file /etc/sysconfig/nfs contains a
>>> variable NFSD_V4_GRACE to control this but it doesn't get applied. The
>>> reason is that the file /proc/fs/nfsd/nfsv4gracetime is not writable:
>>>
>>> [root at nfs1 init.d]# echo 10 > /proc/fs/nfsd/nfsv4gracetime
>>> -bash: echo: write error: Device or resource busy

Apparently the nfs daemon is the culprit here. If it is running you get 
this error. The reason for why the value is not set probably even right 
after the kernel module is loaded seems to have a different cause (see 
below).

>>
>> You can write the paramerter only after the nfsd kernel module is
>> loaded. As
>> far as I can remember the init script sets the leasetime according to the
>> config. You have to add a config option and patch your init script to
>> set the
>> gracetime at the same point in the script.
>>
>>> Does anyone know what the proper way is to reduce this value?
>>
>> In RHEL 6.5 it is a option in the config file.
>
> Yes this is the NFSD_V4_GRACE option in /etc/sysconfig/nfs I was
> mentioning above and in the init script this gets set right after the
> kernel module is loaded but this doesn't seem to work. When I uncomment
> this option and set it to 10 seconds after the service is started the
> value for the grace time is still 90 for /proc/fs/nfsd/nfsv4gracetime.
> I've also looked for a parameter for the kernel module itself to set
> this when the module is loaded but that doesn't exist and now I'm
> wondering how to set this value at all.
>
> Do you know what the criterion is that determines when this value is
> settable? The module is loaded the whole time so that alone can't be the
> only factor here and there must be some additional constraint that now
> prevents me from updating this value.

So I investigated this further and there seem to be two distinct issues:

1) NFSD_V4_GRACE only gets applied the second time the nfs service is 
started

Check out the following sequence that I executed right after booting the 
system (pacemaker is not running at this point):

[root at nfs2 ~]# cat /etc/sysconfig/nfs |grep GRACE
NFSD_V4_GRACE=10
[root at nfs2 ~]# service nfs start
Starting NFS services:                                     [  OK  ]
Starting NFS quotas:                                       [  OK  ]
Starting NFS mountd:                                       [  OK  ]
Starting NFS daemon:                                       [  OK  ]
Starting RPC idmapd:                                       [  OK  ]
[root at nfs2 ~]# cat /proc/fs/nfsd/nfsv4gracetime
90
[root at nfs2 ~]# service nfs stop
Shutting down NFS daemon:                                  [  OK  ]
Shutting down NFS mountd:                                  [  OK  ]
Shutting down NFS quotas:                                  [  OK  ]
Shutting down RPC idmapd:                                  [  OK  ]
[root at nfs2 ~]# cat /proc/fs/nfsd/nfsv4gracetime
90
[root at nfs2 ~]# service nfs start
Starting NFS services:                                     [  OK  ]
Starting NFS quotas:                                       [  OK  ]
Starting NFS mountd:                                       [  OK  ]
Starting NFS daemon:                                       [  OK  ]
Starting RPC idmapd:                                       [  OK  ]
[root at nfs2 ~]# cat /proc/fs/nfsd/nfsv4gracetime
10

As you can see the grace time is only applied after I stopped and 
started the service the second time. Not sure why this is the case.

2) When started via Pacemaker the value of /proc/fs/nfsd/nfsv4gracetime 
is always set to the default value of 90.

It appears that starting the nfs service from the shell vs. starting it 
using Pacemaker makes a difference and when started with pacemaker the 
value from /etc/sysconfig/nfs is not applied and even when the value has 
been set manually to e.g. 20 after pacemaker starts the service it will 
be reset to the default 90.
What is strange about this is that the init script simply sources the 
sysconfig file so I'm not sure how the environment could prevent the 
value from getting applied.

Regards,
   Dennis