[ClusterLabs] Regression in Filesystem RA
Dejan Muhamedagic
dejanmm at fastmail.fm
Tue Oct 24 02:59:17 EDT 2017
Hi,
On Thu, Oct 19, 2017 at 10:23:52AM +0900, Christian Balzer wrote:
>
> Hello Dejan,
>
> On Tue, 17 Oct 2017 13:13:11 +0200 Dejan Muhamedagic wrote:
>
> > Hi Lars,
> >
> > On Mon, Oct 16, 2017 at 08:52:04PM +0200, Lars Ellenberg wrote:
> > > On Mon, Oct 16, 2017 at 08:09:21PM +0200, Dejan Muhamedagic wrote:
> > > > Hi,
> > > >
> > > > On Thu, Oct 12, 2017 at 03:30:30PM +0900, Christian Balzer wrote:
> > > > >
> > > > > Hello,
> > > > >
> > > > > 2nd post in 10 years, lets see if this one gets an answer unlike the first
> > > > > one...
> > >
> > > Do you want to make me check for the old one? ;-)
> > >
> > > > > One of the main use cases for pacemaker here are DRBD replicated
> > > > > active/active mailbox servers (dovecot/exim) on Debian machines.
> > > > > We've been doing this for a loong time, as evidenced by the oldest pair
> > > > > still running Wheezy with heartbeat and pacemaker 1.1.7.
> > > > >
> > > > > The majority of cluster pairs is on Jessie with corosync and backported
> > > > > pacemaker 1.1.16.
> > > > >
> > > > > Yesterday we had a hiccup, resulting in half the machines loosing
> > > > > their upstream router for 50 seconds which in turn caused the pingd RA to
> > > > > trigger a fail-over of the DRBD RA and associated resource group
> > > > > (filesystem/IP) to the other node.
> > > > >
> > > > > The old cluster performed flawlessly, the newer clusters all wound up with
> > > > > DRBD and FS resource being BLOCKED as the processes holding open the
> > > > > filesystem didn't get killed fast enough.
> > > > >
> > > > > Comparing the 2 RAs (no versioning T_T) reveals a large change in the
> > > > > "signal_processes" routine.
> > > > >
> > > > > So with the old Filesystem RA using fuser we get something like this and
> > > > > thousands of processes killed per second:
> > > > > ---
> > > > > Oct 11 15:06:35 mbx07 lrmd: [4731]: info: RA output: (res_Filesystem_mb07:stop:stdout) 3478 3593 ...
> > > > > Oct 11 15:06:35 mbx07 lrmd: [4731]: info: RA output: (res_Filesystem_mb07:stop:stderr) cmccmccmccmcmcmcmcmccmccmcmcmcmcmcmcmcmcmcmcmcmccmcm
> > > > > Oct 11 15:06:35 mbx07 lrmd: [4731]: info: RA output: (res_Filesystem_mb07:stop:stdout) 4032 4058 ...
> > > > > ---
> > > > >
> > > > > Whereas the new RA (newer isn't better) that goes around killing processes
> > > > > individually with beautiful logging was a total fail at about 4 processes
> > > > > per second killed...
> > > > > ---
> > > > > Oct 11 15:06:46 mbx10 Filesystem(res_Filesystem_mb10)[288712]: INFO: sending signal TERM to: mail 4226 4909 0 09:43 ? S 0:00 dovecot/imap
> > > > > Oct 11 15:06:46 mbx10 Filesystem(res_Filesystem_mb10)[288712]: INFO: sending signal TERM to: mail 4229 4909 0 09:43 ? S 0:00 dovecot/imap [idling]
> > > > > Oct 11 15:06:46 mbx10 Filesystem(res_Filesystem_mb10)[288712]: INFO: sending signal TERM to: mail 4238 4909 0 09:43 ? S 0:00 dovecot/imap
> > > > > Oct 11 15:06:46 mbx10 Filesystem(res_Filesystem_mb10)[288712]: INFO: sending signal TERM to: mail 4239 4909 0 09:43 ? S 0:00 dovecot/imap
> > > > > ---
> > > > >
> > > > > So my questions are:
> > > > >
> > > > > 1. Am I the only one with more than a handful of processes per FS who
> > > > > can't afford to wait hours the new routine to finish?
> > > >
> > > > The change was introduced about five years ago.
> > >
> Yeah, that was thanks to Debian Jessie not having pacemaker at all from
> the start and when the backport arrived it was corosync only w/o a
> graceful transition from heartbeat option, so quite a few machines stayed
> at wheezy (thanks to the LTS efforts).
>
> > > Also, usually there should be no process anymore,
> > > because whatever is using the Filesystem should have it's own RA,
> > > which should have appropriate constraints,
> > > which means that should have been called and "stop"ped first,
> > > before the Filesystem stop and umount, and only the "accidental,
> > > stray, abandoned, idle since three weeks, operator shell session,
> > > that happend to cd into that file system" is supposed to be around
> > > *unexpectedly* and in need of killing, and not "thousands of service
> > > processes, expectedly".
> >
> > Indeed, but obviously one can never tell ;-)
> >
> > > So arguably your setup is broken,
> >
> > Or the other RA didn't/couldn't stop the resource ...
> >
> See my previous mail, there is no good/right way to solve this with a RA
> for dovecot, which would essentially mimic what the FS RA should be doing,
> since stopping dovecot entirely is not what is called for.
>
> > > relying on a fall-back workaround
> > > which used to "perform" better.
> > >
> > > The bug is not that this fall-back workaround now
> > > has pretty printing and is much slower (and eventually times out),
> > > the bug is that you don't properly kill the service first.
> > > [and that you don't have fencing].
> >
> > ... and didn't exit with an appropriate exit code (i.e. fail).
> >
> Could somebody elaborate on this, especially the fencing part?
If a stop action of any cluster resource fails then the node is
fenced.
> Because DRBD fencing is configured and working as expected.
>
> > > > > 2. Can we have the old FUSER (kill) mode back?
> > > >
> > > > Yes. I'll make a pull request.
> > >
> > > Still, that's a sane thing to do,
> > > thanks, dejanm.
> >
> > Right. We probably cannot fix all issues coming from various RAs
> > or configurations, but we should at least try a bit harder.
> >
> Indeed, your response is much appreciated.
>
> And while we're at it, I'd like to point out that while FUSER is faster
> than this it still slows down exponentially with the number of processes.
>
> To wit:
> ---
> # time lsof -n |grep mb07.dentaku |awk '{print $2}' |uniq |wc
> 1381 1381 7777
>
> real 0m0.874s
> user 0m0.396s
> sys 0m0.496s
>
> # time fuser -m /mail/spool/mb07.dentaku.gol.com >/dev/null
> real 0m0.426s
> user 0m0.112s
> sys 0m0.308s
> ---
> Snappy, fuser beats my crappy pipe for 1400 processes.
>
> Alas on a busier server:
> ---
> # time lsof -n |grep mb10.dentaku |awk '{print $2}' |uniq |wc
> 8319 8319 56849
>
> real 0m4.767s
> user 0m2.176s
> sys 0m2.676s
>
> # time fuser -m /mail/spool/mb10.dentaku.gol.com >/dev/null
> real 0m11.414s
> user 0m10.292s
> sys 0m1.112s
> ---
> So with 8K processes fuser is clearly falling behind.
>
>
> And finally on my largest machines it turns into molasses:
> ---
> # time lsof -n |grep mb11.dentaku |awk '{print $2}' |uniq |wc
> 33319 33319 231577
>
> real 0m26.349s
> user 0m15.920s
> sys 0m10.688s
>
> time fuser -m /mail/spool/mb11.dentaku.gol.com >/dev/null
> real 2m32.957s
> user 2m28.556s
> sys 0m4.376s
> ---
Heh, somebody should take a look at fuser.
> While the 2.5 minutes would still be acceptable of sorts (with a properly
> set stop-timeout), clearly the 26s version is preferable.
>
> So what I'm suggesting here is looking at the lsof approach to speed
> things up, no matter how many processes there are.
I just made a pull request:
https://github.com/ClusterLabs/resource-agents/pull/1042
It would be great if you could test it!
Cheers,
Dejan
> Regards,
>
> Christian
>
> > > Maybe we can even come up with a way
> > > to both "pretty print" and kill fast?
> >
> > My best guess right now is no ;-) But we could log nicely for the
> > usual case of a small number of stray processes ... maybe
> > something like this:
> >
> > i=""
> > get_pids | tr '\n' ' ' | fold -s |
> > while read procs; do
> > if [ -z "$i" ]; then
> > killnlog $procs
> > i="nolog"
> > else
> > justkill $procs
> > fi
> > done
> >
> > Cheers,
> >
> > Dejan
> >
> > > --
> > > : Lars Ellenberg
> > > : LINBIT | Keeping the Digital World Running
> > > : DRBD -- Heartbeat -- Corosync -- Pacemaker
> > > : R&D, Integration, Ops, Consulting, Support
> > >
> > > DRBD® and LINBIT® are registered trademarks of LINBIT
> > >
> > > _______________________________________________
> > > Users mailing list: Users at clusterlabs.org
> > > http://lists.clusterlabs.org/mailman/listinfo/users
> > >
> > > Project Home: http://www.clusterlabs.org
> > > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > > Bugs: http://bugs.clusterlabs.org
> >
> > _______________________________________________
> > Users mailing list: Users at clusterlabs.org
> > http://lists.clusterlabs.org/mailman/listinfo/users
> >
> > Project Home: http://www.clusterlabs.org
> > Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > Bugs: http://bugs.clusterlabs.org
> >
>
>
> --
> Christian Balzer Network/Systems Engineer
> chibi at gol.com Rakuten Communications
More information about the Users
mailing list