[ClusterLabs] [EXT] [EXT] Re: pacemaker&corosync services shutting off after the network cable is disconnected

[Chris Lumens]

The easiest way to get to the bottom of this problem would be if I could 
reproduce it.  I could then `git bisect` between the known good and 
known bad releases of pacemaker and track down the offending commit.

I could also tell you how to do that, but it's more work than I'd want 
to push off onto you.

Would you be willing to send me your CIB file, after editing out any 
sensitive information like passwords, internal hostnames, etc.?  You can 
just email it to me directly.  There's no need to send it to the list 
where it'll be archived.

> The test is simply to watch the behavior of the cluster when the cable 
> is removed from the hypervisor which runs the VM with the VIP resource 
> started on it. When the cable is pulled, the remaining nodes behave as 
> expected, meaning, the resources successfully start on other nodes.  The 
> problem I saw is the unexpected exit of the pacemaker service, which you 
> can see in the log file I attached.

Unfortunately, I don't have a setup here where I have enough physical 
hardware.  Pretty much everything I do on clusters involves VMs, which 
means however I'm bringing down the network is totally different from 
what you're doing.  Maybe I can find someone who's got such an 
environment, however.

> So, my first question.  Based on the log file, can anyone tell me why 
> the pacemaker service exiting?
> 
> Should pacemaker be exiting every time it loses quorum?  I ask because 
> I've performed several more tests and the behavior of pacemaker service 
> is inconsistent.  Sometimes, it doesn't exit.
> 
> I am very confused by this inconsistency.  Any insight would be 
> appreciated.
> 
> An interesting environmental issue.  When the network cable is removed 
> from the hypervisor, the load average, according to "top", start going 
> up very quickly.  I mention it because it may be affecting what I see.
I've gotta say, whatever is going on in your log file is just utterly 
bizarre.  pacemakerd repeatedly detects that pengine (the old name for 
pacemakerd-schedulerd) is unresponsive to IPC, and then tries to kill 
it.  Apparently that doesn't work because it tries again and again, 
something like 130 times.

I also don't see what pacemaker-schedulerd could possibly be up to. 
After it stops all resources (this is due to no-quorum-policy), it's 
just never heard from again.

It's possible the load average has something to do with it.  One thing 
you could do is set 
PCMK_trace_files="controld_throttle.c,controld_messages.c" in 
/etc/sysconfig/pacemaker and restart.  This will likely result in a lot 
of logs.  Maybe that will turn up something interesting.

Another thought... is there anything in the system log about OOM killing 
processes, or anything like that?

- Chris