[ClusterLabs] Cluster Stopped, No Messages?

[Andrei Borzenkov]

On 01.06.2021 19:21, Eric Robinson wrote:
> 
>> -----Original Message-----
>> From: Users <users-bounces at clusterlabs.org> On Behalf Of Klaus
>> Wenninger
>> Sent: Monday, May 31, 2021 12:54 AM
>> To: users at clusterlabs.org
>> Subject: Re: [ClusterLabs] Cluster Stopped, No Messages?
>>
>> On 5/29/21 12:21 AM, Strahil Nikolov wrote:
>>> I agree -> fencing is mandatory.
>> Agreed that with proper fencing setup the cluster wouldn'thave run into that
>> state.
>> But still it might be interesting to find out what has happened.
> 
> Thank you for looking past the fencing issue to the real question. Regardless of whether or not fencing was enabled, there should still be some indication of what actions the cluster took and why, but it appears that cluster services just terminated silently.
> 
>> Not seeing anything in the log snippet either.
> 
> Me neither.
> 
>> Assuming you are running something systemd-based. Centos 7.
> 
> Yes. CentOS Linux release 7.5.1804.
> 
>> Did you check the journal for pacemaker to see what systemd is thinking?
>> With the standard unit-file systemd should observe pacemakerd and restart
>> it if it goes away ungracefully.
> 
> The only log entry showing Pacemaker startup that I found in any of the messages files (current and several days of history) was the one when I started the cluster manually (see below).
> 

If cluster processes stopped or crashed you obviously won't see any logs
from them until they are restarted. You need to look at other system
logs - may be they record something unusual around this time? Any crash
dumps?