[ClusterLabs] pacemaker-fenced[11637]: warning: Can't create a sane reply

[Ken Gaillot]

On Wed, 2022-06-22 at 17:16 +0530, Priyanka Balotra wrote:
> Hi All, 
> 
> We are seeing an issue where we performed cluster shutdown followed
> by cluster boot operation. All the nodes joined the cluster excet one
> (the first node). Here are some pacemaker logs around that
> timestamp: 
> 
> 2022-06-19T07:02:08.690213+00:00 FILE-1 pacemaker-fenced[11637]: 
> notice: Operation 'off' targeting FILE-1 on FILE-2 for 
> pacemaker-controld.11523 at FILE-2.0b09e949: OK
> 2022-06-19T07:02:08.690604+00:00 FILE-1 pacemaker-fenced[11637]: 
> error: stonith_construct_reply: Triggered assert at
> fenced_commands.c:2363 : request != NULL
> 2022-06-19T07:02:08.690781+00:00 FILE-1 pacemaker-fenced[11637]: 
> warning: Can't create a sane reply

This assertion and message were dropped in the 2.0.5 release because
they do not really indicate an error. The new message in this situation
is:

  Missing request information for client notifications for operation
with result '...' (initiated before we came up?)

The most likely sequence in this particular case is that FILE-2 came up

and formed a cluster first, then fenced FILE-1 for being unseen. FILE-1 then finished coming up, joined the cluster, and got the notification
of its own fencing (which by the way means that either fencing is
misconfigured or perhaps you are using fabric fencing).

> 2022-06-19T07:02:08.691872+00:00 FILE-1 pacemaker-controld[11643]: 
> crit: We were allegedly just fenced by FILE-2 for FILE-2!
> 2022-06-19T07:02:08.693994+00:00 FILE-1 pacemakerd[11622]:  warning:
> Shutting cluster down because pacemaker-controld[11643] had fatal
> failure
> 2022-06-19T07:02:08.694209+00:00 FILE-1 pacemakerd[11622]:  notice:
> Shutting down Pacemaker
> 2022-06-19T07:02:08.694381+00:00 FILE-1 pacemakerd[11622]:  notice:
> Stopping pacemaker-schedulerd
> 
> 
> Let us know if you need any more logs to find an rca to this. 
> 
> Thanks
> Priyanka
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
> 
> ClusterLabs home: https://www.clusterlabs.org/
-- 
Ken Gaillot <kgaillot at redhat.com>