[ClusterLabs] Pending Fencing Actions shown in pcs status
Klaus Wenninger
kwenning at redhat.com
Thu Jan 7 05:14:57 EST 2021
Hi Steffen,
If you just see the leftover pending-action on one node
it would be interesting if restarting of pacemaker on
one of the other nodes does sync it to all of the
nodes.
Regards,
Klaus
On 1/7/21 9:54 AM, renayama19661014 at ybb.ne.jp wrote:
> Hi Steffen,
>
>> Unfortunately not sure about the exact scenario. But I have been doing
>> some recent experiments with node standby/unstandby stop/start. This
>> to get procedures right for updating node rpms etc.
>>
>> Later I noticed the uncomforting "pending fencing actions" status msg.
> Okay!
>
> Repeat the standby and unstandby steps in the same way to check.
> We will start checking after tomorrow, so I think it will take some time until next week.
>
>
> Many thanks,
> Hideo Yamauchi.
>
>
>
> ----- Original Message -----
>> From: "renayama19661014 at ybb.ne.jp" <renayama19661014 at ybb.ne.jp>
>> To: Reid Wahl <nwahl at redhat.com>; Cluster Labs - All topics related to open-source clustering welcomed <users at clusterlabs.org>
>> Cc:
>> Date: 2021/1/7, Thu 17:51
>> Subject: Re: [ClusterLabs] Pending Fencing Actions shown in pcs status
>>
>> Hi Steffen,
>> Hi Reid,
>>
>> The fencing history is kept inside stonith-ng and is not written to cib.
>> However, getting the entire cib and getting it sent will help you to reproduce
>> the problem.
>>
>> Best Regards,
>> Hideo Yamauchi.
>>
>>
>> ----- Original Message -----
>>> From: Reid Wahl <nwahl at redhat.com>
>>> To: renayama19661014 at ybb.ne.jp; Cluster Labs - All topics related to
>> open-source clustering welcomed <users at clusterlabs.org>
>>> Date: 2021/1/7, Thu 17:39
>>> Subject: Re: [ClusterLabs] Pending Fencing Actions shown in pcs status
>>>
>>>
>>> Hi, Steffen. Those attachments don't contain the CIB. They contain the
>> `pcs config` output. You can get the cib with `pcs cluster cib >
>> $(hostname).cib.xml`.
>>>
>>> Granted, it's possible that this fence action information wouldn't
>> be in the CIB at all. It might be stored in fencer memory.
>>>
>>> On Thu, Jan 7, 2021 at 12:26 AM <renayama19661014 at ybb.ne.jp> wrote:
>>>
>>> Hi Steffen,
>>>>> Here CIB settings attached (pcs config show) for all 3 of my nodes
>>>>> (all 3 seems 100% identical), node03 is the DC.
>>>>
>>>> Thank you for the attachment.
>>>>
>>>> What is the scenario when this situation occurs?
>>>> In what steps did the problem appear when fencing was performed (or
>> failed)?
>>>>
>>>> Best Regards,
>>>> Hideo Yamauchi.
>>>>
>>>>
>>>> ----- Original Message -----
>>>>> From: Steffen Vinther Sørensen <svinther at gmail.com>
>>>>> To: renayama19661014 at ybb.ne.jp; Cluster Labs - All topics related
>> to open-source clustering welcomed <users at clusterlabs.org>
>>>>> Cc:
>>>>> Date: 2021/1/7, Thu 17:05
>>>>> Subject: Re: [ClusterLabs] Pending Fencing Actions shown in pcs
>> status
>>>>> Hi Hideo,
>>>>>
>>>>> Here CIB settings attached (pcs config show) for all 3 of my nodes
>>>>> (all 3 seems 100% identical), node03 is the DC.
>>>>>
>>>>> Regards
>>>>> Steffen
>>>>>
>>>>> On Thu, Jan 7, 2021 at 8:06 AM <renayama19661014 at ybb.ne.jp>
>> wrote:
>>>>>> Hi Steffen,
>>>>>> Hi Reid,
>>>>>>
>>>>>> I also checked the Centos source rpm and it seems to include a
>> fix for the
>>>>> problem.
>>>>>> As Steffen suggested, if you share your CIB settings, I might
>> know
>>>>> something.
>>>>>> If this issue is the same as the fix, the display will only be
>> displayed on
>>>>> the DC node and will not affect the operation.
>>>>>> The pending actions shown will remain for a long time, but
>> will not have a
>>>>> negative impact on the cluster.
>>>>>> Best Regards,
>>>>>> Hideo Yamauchi.
>>>>>>
>>>>>>
>>>>>> ----- Original Message -----
>>>>>> > From: Reid Wahl <nwahl at redhat.com>
>>>>>> > To: Cluster Labs - All topics related to open-source
>> clustering
>>>>> welcomed <users at clusterlabs.org>
>>>>>> > Cc:
>>>>>> > Date: 2021/1/7, Thu 15:58
>>>>>> > Subject: Re: [ClusterLabs] Pending Fencing Actions shown
>> in pcs status
>>>>>> >
>>>>>> > It's supposedly fixed in that version.
>>>>>> > - https://bugzilla.redhat.com/show_bug.cgi?id=1787749
>>>>>> > - https://access.redhat.com/solutions/4713471
>>>>>> >
>>>>>> > So you may be hitting a different issue (unless
>> there's a bug in
>>>>> the
>>>>>> > pcmk 1.1 backport of the fix).
>>>>>> >
>>>>>> > I may be a little bit out of my area of knowledge here,
>> but can you
>>>>>> > share the CIBs from nodes 1 and 3? Maybe Hideo, Klaus, or
>> Ken has some
>>>>>> > insight.
>>>>>> >
>>>>>> > On Wed, Jan 6, 2021 at 10:53 PM Steffen Vinther Sørensen
>>>>>> > <svinther at gmail.com> wrote:
>>>>>> >>
>>>>>> >> Hi Hideo,
>>>>>> >>
>>>>>> >> If the fix is not going to make it into the CentOS7
>> pacemaker
>>>>> version,
>>>>>> >> I guess the stable approach to take advantage of it
>> is to build
>>>>> the
>>>>>> >> cluster on another OS than CentOS7 ? A little late
>> for that in
>>>>> this
>>>>>> >> case though :)
>>>>>> >>
>>>>>> >> Regards
>>>>>> >> Steffen
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >>
>>>>>> >> On Thu, Jan 7, 2021 at 7:27 AM
>> <renayama19661014 at ybb.ne.jp>
>>>>> wrote:
>>>>>> >> >
>>>>>> >> > Hi Steffen,
>>>>>> >> >
>>>>>> >> > The fix pointed out by Reid is affecting it.
>>>>>> >> >
>>>>>> >> > Since the fencing action requested by the DC
>> node exists
>>>>> only in the
>>>>>> > DC node, such an event occurs.
>>>>>> >> > You will need to take advantage of the modified
>> pacemaker to
>>>>> resolve
>>>>>> > the issue.
>>>>>> >> >
>>>>>> >> > Best Regards,
>>>>>> >> > Hideo Yamauchi.
>>>>>> >> >
>>>>>> >> >
>>>>>> >> >
>>>>>> >> > ----- Original Message -----
>>>>>> >> > > From: Reid Wahl <nwahl at redhat.com>
>>>>>> >> > > To: Cluster Labs - All topics related to
>> open-source
>>>>> clustering
>>>>>> > welcomed <users at clusterlabs.org>
>>>>>> >> > > Cc:
>>>>>> >> > > Date: 2021/1/7, Thu 15:07
>>>>>> >> > > Subject: Re: [ClusterLabs] Pending Fencing
>> Actions
>>>>> shown in pcs
>>>>>> > status
>>>>>> >> > >
>>>>>> >> > > Hi, Steffen. Are your cluster nodes all
>> running the
>>>>> same
>>>>>> > Pacemaker
>>>>>> >> > > versions? This looks like Bug 5401[1],
>> which is fixed
>>>>> by upstream
>>>>>> >> > > commit df71a07[2]. I'm a little bit
>> confused about
>>>>> why it
>>>>>> > only shows
>>>>>> >> > > up on one out of three nodes though.
>>>>>> >> > >
>>>>>> >> > > [1]
>> https://bugs.clusterlabs.org/show_bug.cgi?id=5401
>>>>>> >> > > [2]
>>>>> https://github.com/ClusterLabs/pacemaker/commit/df71a07
>>>>>> >> > >
>>>>>> >> > > On Tue, Jan 5, 2021 at 8:31 AM Steffen
>> Vinther Sørensen
>>>>>> >> > > <svinther at gmail.com> wrote:
>>>>>> >> > >>
>>>>>> >> > >> Hello
>>>>>> >> > >>
>>>>>> >> > >> node 1 is showing this in 'pcs
>> status'
>>>>>> >> > >>
>>>>>> >> > >> Pending Fencing Actions:
>>>>>> >> > >> * reboot of
>> kvm03-node02.avigol-gcs.dk pending:
>>>>>> > client=crmd.37819,
>>>>>> >> > >> origin=kvm03-node03.avigol-gcs.dk
>>>>>> >> > >>
>>>>>> >> > >> node 2 and node 3 outputs no such
>> thing (node 3 is
>>>>> DC)
>>>>>> >> > >>
>>>>>> >> > >> Google is not much help, how to
>> investigate this
>>>>> further and
>>>>>> > get rid
>>>>>> >> > >> of such terrifying status message ?
>>>>>> >> > >>
>>>>>> >> > >> Regards
>>>>>> >> > >> Steffen
>>>>>> >> > >>
>> _______________________________________________
>>>>>> >> > >> Manage your subscription:
>>>>>> >> > >>
>>>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>>>> >> > >>
>>>>>> >> > >> ClusterLabs home:
>> https://www.clusterlabs.org/
>>>>>> >> > >>
>>>>>> >> > >
>>>>>> >> > >
>>>>>> >> > > --
>>>>>> >> > > Regards,
>>>>>> >> > >
>>>>>> >> > > Reid Wahl, RHCA
>>>>>> >> > > Senior Software Maintenance Engineer, Red
>> Hat
>>>>>> >> > > CEE - Platform Support Delivery -
>> ClusterHA
>>>>>> >> > >
>>>>>> >> > >
>> _______________________________________________
>>>>>> >> > > Manage your subscription:
>>>>>> >> > >
>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>>>> >> > >
>>>>>> >> > > ClusterLabs home:
>> https://www.clusterlabs.org/
>>>>>> >> > >
>>>>>> >> >
>>>>>> >> > _______________________________________________
>>>>>> >> > Manage your subscription:
>>>>>> >> >
>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>>>> >> >
>>>>>> >> > ClusterLabs home: https://www.clusterlabs.org/
>>>>>> >> _______________________________________________
>>>>>> >> Manage your subscription:
>>>>>> >> https://lists.clusterlabs.org/mailman/listinfo/users
>>>>>> >>
>>>>>> >> ClusterLabs home: https://www.clusterlabs.org/
>>>>>> >
>>>>>> >
>>>>>> >
>>>>>> > --
>>>>>> > Regards,
>>>>>> >
>>>>>> > Reid Wahl, RHCA
>>>>>> > Senior Software Maintenance Engineer, Red Hat
>>>>>> > CEE - Platform Support Delivery - ClusterHA
>>>>>> >
>>>>>> > _______________________________________________
>>>>>> > Manage your subscription:
>>>>>> > https://lists.clusterlabs.org/mailman/listinfo/users
>>>>>> >
>>>>>> > ClusterLabs home: https://www.clusterlabs.org/
>>>>>> >
>>>>>>
>>>>>> _______________________________________________
>>>>>> Manage your subscription:
>>>>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>>>>
>>>>>> ClusterLabs home: https://www.clusterlabs.org/
>>>> _______________________________________________
>>>> Manage your subscription:
>>>> https://lists.clusterlabs.org/mailman/listinfo/users
>>>>
>>>> ClusterLabs home: https://www.clusterlabs.org/
>>>>
>>> --
>>>
>>> Regards,
>>>
>>> Reid Wahl, RHCA
>>>
>>> Senior Software Maintenance Engineer, Red Hat
>>> CEE - Platform Support Delivery - ClusterHA
>>>
>>>
>> _______________________________________________
>> Manage your subscription:
>> https://lists.clusterlabs.org/mailman/listinfo/users
>>
>> ClusterLabs home: https://www.clusterlabs.org/
>>
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
>
> ClusterLabs home: https://www.clusterlabs.org/
More information about the Users
mailing list