[ClusterLabs] [Problem] The output of `pcs status --full` is incorrect when a node rejoins.

[renayama19661014 at ybb.ne.jp]

Hi Chris,
Hi All,

We have further confirmed that the same issue occurs with the version of Pacemaker included in RHEL 9.8/RHEL 10.2.
The reproduction rate is quite low.

The root cause is, as suspected, that the `num_updates` value on a joining node has advanced, causing the replacement of the entire CIB (Cluster Information Base) to be skipped.


Next, we considered the following modification.

Proposed solution 1) Increment the epoch on the cluster side when a node leaves.

Proposed solution 2) Reset the epoch on the joining node when it joins the cluster. 
This should result in behavior similar to deleting `/var/lib/pacemaker/cib` on the joining node.

Proposed solution 3) Use both proposed solution 1 and proposed solution 2.

Proposed solution 4) Properly synchronize `num_updates` between the joining node and the cluster. 
This is likely to be extremely complicated and difficult.

Proposed solution 5) Force the application of the initial CIB synchronization from the DC (Designated Coordinator) node onto the joining node.

Proposed solution 6) Collect resource information from all nodes to rebuild the cluster configuration when a node joins. 
However, unless the entire CIB is synchronized, resource information—as well as values ​​like `num_updates` and the epoch—cannot be synchronized.

Manipulating the epoch has significant implications; given the need to account for nodes leaving and joining the cluster, it is a highly complex task to manage.

In principle, we believe that proposed change 5 is the best approach.
We intend to prepare a pull request (PR) for this change.

Please share any feedback or comments you may have.

Best Regards,
Hideo Yamauchi.