[ClusterLabs] Pending Fencing Actions shown in pcs status

Klaus Wenninger kwenning at redhat.com
Thu Jan 7 05:14:57 EST 2021


Hi Steffen,

If you just see the leftover pending-action on one node
it would be interesting if restarting of pacemaker on
one of the other nodes does sync it to all of the
nodes.

Regards,
Klaus

On 1/7/21 9:54 AM, renayama19661014 at ybb.ne.jp wrote:
> Hi Steffen,
>
>> Unfortunately not sure about the exact scenario. But I have been doing
>> some recent experiments with node standby/unstandby stop/start. This
>> to get procedures right for updating node rpms etc.
>>  
>> Later I noticed the uncomforting "pending fencing actions" status msg.
> Okay!
>
> Repeat the standby and unstandby steps in the same way to check.
> We will start checking after tomorrow, so I think it will take some time until next week.
>
>
> Many thanks,
> Hideo Yamauchi.
>
>
>
> ----- Original Message -----
>> From: "renayama19661014 at ybb.ne.jp" <renayama19661014 at ybb.ne.jp>
>> To: Reid Wahl <nwahl at redhat.com>; Cluster Labs - All topics related to open-source clustering welcomed <users at clusterlabs.org>
>> Cc: 
>> Date: 2021/1/7, Thu 17:51
>> Subject: Re: [ClusterLabs] Pending Fencing Actions shown in pcs status
>>
>> Hi Steffen,
>> Hi Reid,
>>
>> The fencing history is kept inside stonith-ng and is not written to cib.
>> However, getting the entire cib and getting it sent will help you to reproduce 
>> the problem.
>>
>> Best Regards,
>> Hideo Yamauchi.
>>
>>
>> ----- Original Message -----
>>> From: Reid Wahl <nwahl at redhat.com>
>>> To: renayama19661014 at ybb.ne.jp; Cluster Labs - All topics related to 
>> open-source clustering welcomed <users at clusterlabs.org> 
>>> Date: 2021/1/7, Thu 17:39
>>> Subject: Re: [ClusterLabs] Pending Fencing Actions shown in pcs status
>>>
>>>
>>> Hi, Steffen. Those attachments don't contain the CIB. They contain the 
>> `pcs config` output. You can get the cib with `pcs cluster cib > 
>> $(hostname).cib.xml`.
>>>
>>> Granted, it's possible that this fence action information wouldn't 
>> be in the CIB at all. It might be stored in fencer memory.
>>>
>>> On Thu, Jan 7, 2021 at 12:26 AM <renayama19661014 at ybb.ne.jp> wrote:
>>>
>>> Hi Steffen,
>>>>>  Here CIB settings attached (pcs config show) for all 3 of my nodes
>>>>>  (all 3 seems 100% identical), node03 is the DC.
>>>>
>>>> Thank you for the attachment.
>>>>
>>>> What is the scenario when this situation occurs?
>>>> In what steps did the problem appear when fencing was performed (or 
>> failed)?
>>>>
>>>> Best Regards,
>>>> Hideo Yamauchi.
>>>>
>>>>
>>>> ----- Original Message -----
>>>>>  From: Steffen Vinther Sørensen <svinther at gmail.com>
>>>>>  To: renayama19661014 at ybb.ne.jp; Cluster Labs - All topics related 
>> to open-source clustering welcomed <users at clusterlabs.org>
>>>>>  Cc: 
>>>>>  Date: 2021/1/7, Thu 17:05
>>>>>  Subject: Re: [ClusterLabs] Pending Fencing Actions shown in pcs 
>> status
>>>>>  Hi Hideo,
>>>>>
>>>>>  Here CIB settings attached (pcs config show) for all 3 of my nodes
>>>>>  (all 3 seems 100% identical), node03 is the DC.
>>>>>
>>>>>  Regards
>>>>>  Steffen
>>>>>
>>>>>  On Thu, Jan 7, 2021 at 8:06 AM <renayama19661014 at ybb.ne.jp> 
>> wrote:
>>>>>>   Hi Steffen,
>>>>>>   Hi Reid,
>>>>>>
>>>>>>   I also checked the Centos source rpm and it seems to include a 
>> fix for the 
>>>>>  problem.
>>>>>>   As Steffen suggested, if you share your CIB settings, I might 
>> know 
>>>>>  something.
>>>>>>   If this issue is the same as the fix, the display will only be 
>> displayed on 
>>>>>  the DC node and will not affect the operation.
>>>>>>   The pending actions shown will remain for a long time, but 
>> will not have a 
>>>>>  negative impact on the cluster.
>>>>>>   Best Regards,
>>>>>>   Hideo Yamauchi.
>>>>>>
>>>>>>
>>>>>>   ----- Original Message -----
>>>>>>   > From: Reid Wahl <nwahl at redhat.com>
>>>>>>   > To: Cluster Labs - All topics related to open-source 
>> clustering 
>>>>>  welcomed <users at clusterlabs.org>
>>>>>>   > Cc:
>>>>>>   > Date: 2021/1/7, Thu 15:58
>>>>>>   > Subject: Re: [ClusterLabs] Pending Fencing Actions shown 
>> in pcs status
>>>>>>   >
>>>>>>   > It's supposedly fixed in that version.
>>>>>>   >   - https://bugzilla.redhat.com/show_bug.cgi?id=1787749 
>>>>>>   >   - https://access.redhat.com/solutions/4713471 
>>>>>>   >
>>>>>>   > So you may be hitting a different issue (unless 
>> there's a bug in 
>>>>>  the
>>>>>>   > pcmk 1.1 backport of the fix).
>>>>>>   >
>>>>>>   > I may be a little bit out of my area of knowledge here, 
>> but can you
>>>>>>   > share the CIBs from nodes 1 and 3? Maybe Hideo, Klaus, or 
>> Ken has some
>>>>>>   > insight.
>>>>>>   >
>>>>>>   > On Wed, Jan 6, 2021 at 10:53 PM Steffen Vinther Sørensen
>>>>>>   > <svinther at gmail.com> wrote:
>>>>>>   >>
>>>>>>   >>  Hi Hideo,
>>>>>>   >>
>>>>>>   >>  If the fix is not going to make it into the CentOS7 
>> pacemaker 
>>>>>  version,
>>>>>>   >>  I guess the stable approach to take advantage of it 
>> is to build 
>>>>>  the
>>>>>>   >>  cluster on another OS than CentOS7 ? A little late 
>> for that in 
>>>>>  this
>>>>>>   >>  case though :)
>>>>>>   >>
>>>>>>   >>  Regards
>>>>>>   >>  Steffen
>>>>>>   >>
>>>>>>   >>
>>>>>>   >>
>>>>>>   >>
>>>>>>   >>  On Thu, Jan 7, 2021 at 7:27 AM 
>> <renayama19661014 at ybb.ne.jp> 
>>>>>  wrote:
>>>>>>   >>  >
>>>>>>   >>  > Hi Steffen,
>>>>>>   >>  >
>>>>>>   >>  > The fix pointed out by Reid is affecting it.
>>>>>>   >>  >
>>>>>>   >>  > Since the fencing action requested by the DC 
>> node exists 
>>>>>  only in the
>>>>>>   > DC node, such an event occurs.
>>>>>>   >>  > You will need to take advantage of the modified 
>> pacemaker to 
>>>>>  resolve
>>>>>>   > the issue.
>>>>>>   >>  >
>>>>>>   >>  > Best Regards,
>>>>>>   >>  > Hideo Yamauchi.
>>>>>>   >>  >
>>>>>>   >>  >
>>>>>>   >>  >
>>>>>>   >>  > ----- Original Message -----
>>>>>>   >>  > > From: Reid Wahl <nwahl at redhat.com>
>>>>>>   >>  > > To: Cluster Labs - All topics related to 
>> open-source 
>>>>>  clustering
>>>>>>   > welcomed <users at clusterlabs.org>
>>>>>>   >>  > > Cc:
>>>>>>   >>  > > Date: 2021/1/7, Thu 15:07
>>>>>>   >>  > > Subject: Re: [ClusterLabs] Pending Fencing 
>> Actions 
>>>>>  shown in pcs
>>>>>>   > status
>>>>>>   >>  > >
>>>>>>   >>  > > Hi, Steffen. Are your cluster nodes all 
>> running the 
>>>>>  same
>>>>>>   > Pacemaker
>>>>>>   >>  > > versions? This looks like Bug 5401[1], 
>> which is fixed 
>>>>>  by upstream
>>>>>>   >>  > > commit df71a07[2]. I'm a little bit 
>> confused about 
>>>>>  why it
>>>>>>   > only shows
>>>>>>   >>  > > up on one out of three nodes though.
>>>>>>   >>  > >
>>>>>>   >>  > > [1] 
>> https://bugs.clusterlabs.org/show_bug.cgi?id=5401 
>>>>>>   >>  > > [2] 
>>>>>  https://github.com/ClusterLabs/pacemaker/commit/df71a07 
>>>>>>   >>  > >
>>>>>>   >>  > > On Tue, Jan 5, 2021 at 8:31 AM Steffen 
>> Vinther Sørensen
>>>>>>   >>  > > <svinther at gmail.com> wrote:
>>>>>>   >>  > >>
>>>>>>   >>  > >>  Hello
>>>>>>   >>  > >>
>>>>>>   >>  > >>  node 1 is showing this in 'pcs 
>> status'
>>>>>>   >>  > >>
>>>>>>   >>  > >>  Pending Fencing Actions:
>>>>>>   >>  > >>  * reboot of 
>> kvm03-node02.avigol-gcs.dk pending:
>>>>>>   > client=crmd.37819,
>>>>>>   >>  > >>  origin=kvm03-node03.avigol-gcs.dk
>>>>>>   >>  > >>
>>>>>>   >>  > >>  node 2 and node 3 outputs no such 
>> thing (node 3 is 
>>>>>  DC)
>>>>>>   >>  > >>
>>>>>>   >>  > >>  Google is not much help, how to 
>> investigate this 
>>>>>  further and
>>>>>>   > get rid
>>>>>>   >>  > >>  of such terrifying status message ?
>>>>>>   >>  > >>
>>>>>>   >>  > >>  Regards
>>>>>>   >>  > >>  Steffen
>>>>>>   >>  > >>  
>> _______________________________________________
>>>>>>   >>  > >>  Manage your subscription:
>>>>>>   >>  > >>  
>>>>>  https://lists.clusterlabs.org/mailman/listinfo/users 
>>>>>>   >>  > >>
>>>>>>   >>  > >>  ClusterLabs home: 
>> https://www.clusterlabs.org/ 
>>>>>>   >>  > >>
>>>>>>   >>  > >
>>>>>>   >>  > >
>>>>>>   >>  > > --
>>>>>>   >>  > > Regards,
>>>>>>   >>  > >
>>>>>>   >>  > > Reid Wahl, RHCA
>>>>>>   >>  > > Senior Software Maintenance Engineer, Red 
>> Hat
>>>>>>   >>  > > CEE - Platform Support Delivery - 
>> ClusterHA
>>>>>>   >>  > >
>>>>>>   >>  > > 
>> _______________________________________________
>>>>>>   >>  > > Manage your subscription:
>>>>>>   >>  > > 
>> https://lists.clusterlabs.org/mailman/listinfo/users 
>>>>>>   >>  > >
>>>>>>   >>  > > ClusterLabs home: 
>> https://www.clusterlabs.org/ 
>>>>>>   >>  > >
>>>>>>   >>  >
>>>>>>   >>  > _______________________________________________
>>>>>>   >>  > Manage your subscription:
>>>>>>   >>  > 
>> https://lists.clusterlabs.org/mailman/listinfo/users 
>>>>>>   >>  >
>>>>>>   >>  > ClusterLabs home: https://www.clusterlabs.org/ 
>>>>>>   >>  _______________________________________________
>>>>>>   >>  Manage your subscription:
>>>>>>   >>  https://lists.clusterlabs.org/mailman/listinfo/users 
>>>>>>   >>
>>>>>>   >>  ClusterLabs home: https://www.clusterlabs.org/ 
>>>>>>   >
>>>>>>   >
>>>>>>   >
>>>>>>   > --
>>>>>>   > Regards,
>>>>>>   >
>>>>>>   > Reid Wahl, RHCA
>>>>>>   > Senior Software Maintenance Engineer, Red Hat
>>>>>>   > CEE - Platform Support Delivery - ClusterHA
>>>>>>   >
>>>>>>   > _______________________________________________
>>>>>>   > Manage your subscription:
>>>>>>   > https://lists.clusterlabs.org/mailman/listinfo/users 
>>>>>>   >
>>>>>>   > ClusterLabs home: https://www.clusterlabs.org/ 
>>>>>>   >
>>>>>>
>>>>>>   _______________________________________________
>>>>>>   Manage your subscription:
>>>>>>   https://lists.clusterlabs.org/mailman/listinfo/users 
>>>>>>
>>>>>>   ClusterLabs home: https://www.clusterlabs.org/ 
>>>> _______________________________________________
>>>> Manage your subscription:
>>>> https://lists.clusterlabs.org/mailman/listinfo/users 
>>>>
>>>> ClusterLabs home: https://www.clusterlabs.org/ 
>>>>
>>> -- 
>>>
>>> Regards,
>>>
>>> Reid Wahl, RHCA
>>>
>>> Senior Software Maintenance Engineer, Red Hat
>>> CEE - Platform Support Delivery - ClusterHA
>>>
>>>
>> _______________________________________________
>> Manage your subscription:
>> https://lists.clusterlabs.org/mailman/listinfo/users 
>>
>> ClusterLabs home: https://www.clusterlabs.org/ 
>>
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
>
> ClusterLabs home: https://www.clusterlabs.org/



More information about the Users mailing list