<div dir="ltr">Hi,<div><br></div><div>I have checked node ha-apex2. <br>The log on that machine from /var/log/messages says "systemd: Power button pressed" and "Shutting down...." but this message appeared just when the ha-apex1 node scheduled the shutdown with difference in seconds.<br><div><div class="gmail_extra"><br>It seems like the peer node (ha-apex1) has sent some kind of power off request and it obeyed to the request.<br> <br>On node ha-apex1 it clearly says "Scheduling Node ha-apex2 for shutdown" which seems like it has scheduled this task to be executed on peer node.<br><br>My servers are running in production, please help me out. I really do not want anything to happen to any of node. I hope you understand the seriousness of this issue.<br><br>NOTE: This didn't only happen on this cluster group of nodes. It also happened few times on another cluster group of machines as well.<br><br>Look at this two messages from ha-apex1 node.<br><br></div><div class="gmail_extra">Jun 14 15:52:23 apex1 pengine[18732]: notice: Scheduling Node ha-apex2 for shutdown<br></div><div class="gmail_extra"><br></div><div class="gmail_extra">Jun 14 15:52:27 apex1 crmd[18733]: notice: do_shutdown of peer ha-apex2 is complete</div><div class="gmail_extra"><br><br>Best regards,</div><div class="gmail_extra">Jaz</div><div class="gmail_extra"><br></div><div class="gmail_extra"><br></div><div class="gmail_extra"><br><div class="gmail_quote"><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><br>
<br>
Message: 1<br>
Date: Thu, 15 Jun 2017 13:53:00 -0500<br>
From: Ken Gaillot <<a href="mailto:kgaillot@redhat.com">kgaillot@redhat.com</a>><br>
To: <a href="mailto:users@clusterlabs.org">users@clusterlabs.org</a><br>
Subject: Re: [ClusterLabs] Pacemaker shutting down peer node<br>
Message-ID: <<a href="mailto:5d122183-2030-050d-3a8e-9c158fa5fb5d@redhat.com">5d122183-2030-050d-3a8e-<wbr>9c158fa5fb5d@redhat.com</a>><br>
Content-Type: text/plain; charset=utf-8<br>
<br>
On 06/15/2017 12:38 AM, Jaz Khan wrote:<br>
> Hi,<br>
><br>
> I have been encountering this serious issue from past couple of months.<br>
> I really have no idea that why pacemaker sends shutdown signal to peer<br>
> node and it goes down. This is very strange and I am too much worried .<br>
><br>
> This is not happening daily, but it surely does this kind of behavior<br>
> after every few days.<br>
><br>
> Version:<br>
> Pacemaker 1.1.16<br>
> Corosync 2.4.2<br>
><br>
> Please help me out with this bug! Below is the log message.<br>
><br>
><br>
><br>
> Jun 14 15:52:23 apex1 crmd[18733]: notice: State transition S_IDLE -><br>
> S_POLICY_ENGINE<br>
> Jun 14 15:52:23 apex1 pengine[18732]: notice: On loss of CCM Quorum: Ignore<br>
><br>
> Jun 14 15:52:23 apex1 pengine[18732]: notice: Scheduling Node ha-apex2<br>
> for shutdown<br>
<br>
This is not a fencing, but a clean shutdown. Normally this only happens<br>
in response to a user request.<br>
<br>
Check the logs on both nodes before this point, to try to see what was<br>
the first indication that it would shut down.<br>
<br>
><br>
> Jun 14 15:52:23 apex1 pengine[18732]: notice: Move vip#011(Started<br>
> ha-apex2 -> ha-apex1)<br>
> Jun 14 15:52:23 apex1 pengine[18732]: notice: Move<br>
> filesystem#011(Started ha-apex2 -> ha-apex1)<br>
> Jun 14 15:52:23 apex1 pengine[18732]: notice: Move samba#011(Started<br>
> ha-apex2 -> ha-apex1)<br>
> Jun 14 15:52:23 apex1 pengine[18732]: notice: Move<br>
> database#011(Started ha-apex2 -> ha-apex1)<br>
> Jun 14 15:52:23 apex1 pengine[18732]: notice: Calculated transition<br>
> 1744, saving inputs in /var/lib/pacemaker/pengine/pe-<wbr>input-123.bz2<br>
> Jun 14 15:52:23 apex1 crmd[18733]: notice: Initiating stop operation<br>
> vip_stop_0 on ha-apex2<br>
> Jun 14 15:52:23 apex1 crmd[18733]: notice: Initiating stop operation<br>
> samba_stop_0 on ha-apex2<br>
> Jun 14 15:52:23 apex1 crmd[18733]: notice: Initiating stop operation<br>
> database_stop_0 on ha-apex2<br>
> Jun 14 15:52:26 apex1 crmd[18733]: notice: Initiating stop operation<br>
> filesystem_stop_0 on ha-apex2<br>
> Jun 14 15:52:27 apex1 kernel: drbd apexdata <a href="http://apex2.br" rel="noreferrer" target="_blank">apex2.br</a> <<a href="http://apex2.br" rel="noreferrer" target="_blank">http://apex2.br</a>>:<br>
> peer( Primary -> Secondary )<br>
> Jun 14 15:52:27 apex1 crmd[18733]: notice: Initiating start operation<br>
> filesystem_start_0 locally on ha-apex1<br>
><br>
> Jun 14 15:52:27 apex1 crmd[18733]: notice: do_shutdown of peer ha-apex2<br>
> is complete<br>
><br>
> Jun 14 15:52:27 apex1 attrd[18731]: notice: Node ha-apex2 state is now lost<br>
> Jun 14 15:52:27 apex1 attrd[18731]: notice: Removing all ha-apex2<br>
> attributes for peer loss<br>
> Jun 14 15:52:27 apex1 attrd[18731]: notice: Lost attribute writer ha-apex2<br>
> Jun 14 15:52:27 apex1 attrd[18731]: notice: Purged 1 peers with id=2<br>
> and/or uname=ha-apex2 from the membership cache<br>
> Jun 14 15:52:27 apex1 stonith-ng[18729]: notice: Node ha-apex2 state is<br>
> now lost<br>
> Jun 14 15:52:27 apex1 stonith-ng[18729]: notice: Purged 1 peers with<br>
> id=2 and/or uname=ha-apex2 from the membership cache<br>
> Jun 14 15:52:27 apex1 cib[18728]: notice: Node ha-apex2 state is now lost<br>
> Jun 14 15:52:27 apex1 cib[18728]: notice: Purged 1 peers with id=2<br>
> and/or uname=ha-apex2 from the membership cache<br>
><br>
><br>
><br>
> Best regards,<br>
> Jaz. K<br>
<br>
<br>
<br></blockquote></div></div></div></div></div>