[ClusterLabs] [Problem] The output of `pcs status --full` is incorrect when a node rejoins.
renayama19661014 at ybb.ne.jp
renayama19661014 at ybb.ne.jp
Wed Sep 30 09:17:47 UTC 2026
Hi Chris,
Hi All,
It seems the message didn't go through properly, so I am resending it.
I am attaching the file as-is, without compressing it.
----
The issue has finally been reproduced.
In a 5-node configuration, if the fourth node is stopped and then rejoined after resources have been moved, the output of `pcs status --full` on that node becomes incorrect.
---
The issue can be reproduced with the following steps:
Step1) Build a cluster with 5 nodes.
Cluster name: test-rh98-cluster2
Cluster Summary:
* Stack: corosync (Pacemaker is running)
* Current DC: rh96-02 (2) (version 2.1.9-1.el9-49aab9983) - partition with quorum
(snip)
Node List:
* Online: [ rh96-01 rh96-02 rh96-03 rh96-04 rh96-05 ]
Full List of Resources:
* Resource Group: primary-group1:
* ipaddr_gwf_101 (ocf:heartbeat:Dummy): Started rh96-01
* ipaddr_dbf_101 (ocf:heartbeat:Dummy): Started rh96-01
* ipaddr_api_101 (ocf:heartbeat:Dummy): Started rh96-01
* ipaddr_kafka_101 (ocf:heartbeat:Dummy): Started rh96-01
* ipaddr_unyo_101 (ocf:heartbeat:Dummy): Started rh96-01
* fs_share_101 (ocf:heartbeat:Dummy): Started rh96-01
* mw_jp1base_101 (ocf:heartbeat:Dummy): Started rh96-01
* mw_jp1ajs3_101 (ocf:heartbeat:Dummy): Started rh96-01
* Resource Group: primary-group2:
* ipaddr_gwf_102 (ocf:heartbeat:Dummy): Started rh96-02
* ipaddr_dbf_102 (ocf:heartbeat:Dummy): Started rh96-02
* ipaddr_api_102 (ocf:heartbeat:Dummy): Started rh96-02
* ipaddr_kafka_102 (ocf:heartbeat:Dummy): Started rh96-02
* ipaddr_unyo_102 (ocf:heartbeat:Dummy): Started rh96-02
* fs_share_102 (ocf:heartbeat:Dummy): Started rh96-02
* mw_jp1base_102 (ocf:heartbeat:Dummy): Started rh96-02
* mw_jp1ajs3_102 (ocf:heartbeat:Dummy): Started rh96-02
* Resource Group: primary-group3:
* ipaddr_gwf_103 (ocf:heartbeat:Dummy): Started rh96-03
* ipaddr_dbf_103 (ocf:heartbeat:Dummy): Started rh96-03
* ipaddr_api_103 (ocf:heartbeat:Dummy): Started rh96-03
* ipaddr_kafka_103 (ocf:heartbeat:Dummy): Started rh96-03
* ipaddr_unyo_103 (ocf:heartbeat:Dummy): Started rh96-03
* fs_share_103 (ocf:heartbeat:Dummy): Started rh96-03
* mw_jp1base_103 (ocf:heartbeat:Dummy): Started rh96-03
* mw_jp1ajs3_103 (ocf:heartbeat:Dummy): Started rh96-03
* Clone Set: ping-clone [ping]:
* Started: [ rh96-01 rh96-02 rh96-03 rh96-04 rh96-05 ]
* fence_sbd_101 (stonith:fence_sbd): Started rh96-05
* fence_sbd_102 (stonith:fence_sbd): Started rh96-03
* fence_sbd_103 (stonith:fence_sbd): Started rh96-05
* fence_sbd_104 (stonith:fence_sbd): Started rh96-02
* fence_sbd_105 (stonith:fence_sbd): Started rh96-01
2) Move the resource to the fourth node (rh96-04).
# pcs resource move primary-group1 rh96-04
3) Once the move is complete, move the resource to the first node (rh96-01).
# pcs resource move primary-group1 rh96-01
4) Stop the fourth node.
5) When the fourth node is started again, the display of the resource's startup status on that node becomes incorrect. The status displays correctly on the other nodes.
[root at rh96-04 ~]#
(snip)
Node List:
* Online: [ rh96-01 rh96-02 rh96-03 rh96-04 rh96-05 ]
Full List of Resources:
* Resource Group: primary-group1:
* ipaddr_gwf_101 (ocf:heartbeat:Dummy): Stopped
* ipaddr_dbf_101 (ocf:heartbeat:Dummy): Stopped
* ipaddr_api_101 (ocf:heartbeat:Dummy): Stopped
* ipaddr_kafka_101 (ocf:heartbeat:Dummy): Stopped
* ipaddr_unyo_101 (ocf:heartbeat:Dummy): Stopped
* fs_share_101 (ocf:heartbeat:Dummy): Stopped
* mw_jp1base_101 (ocf:heartbeat:Dummy): Stopped
* mw_jp1ajs3_101 (ocf:heartbeat:Dummy): Stopped
* Resource Group: primary-group2:
* ipaddr_gwf_102 (ocf:heartbeat:Dummy): Started rh96-02
* ipaddr_dbf_102 (ocf:heartbeat:Dummy): Started rh96-02
* ipaddr_api_102 (ocf:heartbeat:Dummy): Started rh96-02
* ipaddr_kafka_102 (ocf:heartbeat:Dummy): Started rh96-02
* ipaddr_unyo_102 (ocf:heartbeat:Dummy): Started rh96-02
* fs_share_102 (ocf:heartbeat:Dummy): Started rh96-02
* mw_jp1base_102 (ocf:heartbeat:Dummy): Started rh96-02
* mw_jp1ajs3_102 (ocf:heartbeat:Dummy): Started rh96-02
* Resource Group: primary-group3:
* ipaddr_gwf_103 (ocf:heartbeat:Dummy): Stopped
* ipaddr_dbf_103 (ocf:heartbeat:Dummy): Stopped
* ipaddr_api_103 (ocf:heartbeat:Dummy): Stopped
* ipaddr_kafka_103 (ocf:heartbeat:Dummy): Stopped
* ipaddr_unyo_103 (ocf:heartbeat:Dummy): Stopped
* fs_share_103 (ocf:heartbeat:Dummy): Stopped
* mw_jp1base_103 (ocf:heartbeat:Dummy): Stopped
* mw_jp1ajs3_103 (ocf:heartbeat:Dummy): Stopped
* Clone Set: ping-clone [ping]:
* Started: [ rh96-02 rh96-04 ]
* Stopped: [ rh96-01 rh96-03 rh96-05 ]
* fence_sbd_101 (stonith:fence_sbd): Stopped
* fence_sbd_102 (stonith:fence_sbd): Stopped
* fence_sbd_103 (stonith:fence_sbd): Stopped
* fence_sbd_104 (stonith:fence_sbd): Started rh96-02
* fence_sbd_105 (stonith:fence_sbd): Stopped
[root at rh96-04 ~]#
[root at rh96-01 ~]#
(snip)
Node List:
* Online: [ rh96-01 rh96-02 rh96-03 rh96-04 rh96-05 ]
Full List of Resources:
* Resource Group: primary-group1:
* ipaddr_gwf_101 (ocf:heartbeat:Dummy): Started rh96-01
* ipaddr_dbf_101 (ocf:heartbeat:Dummy): Started rh96-01
* ipaddr_api_101 (ocf:heartbeat:Dummy): Started rh96-01
* ipaddr_kafka_101 (ocf:heartbeat:Dummy): Started rh96-01
* ipaddr_unyo_101 (ocf:heartbeat:Dummy): Started rh96-01
* fs_share_101 (ocf:heartbeat:Dummy): Started rh96-01
* mw_jp1base_101 (ocf:heartbeat:Dummy): Started rh96-01
* mw_jp1ajs3_101 (ocf:heartbeat:Dummy): Started rh96-01
* Resource Group: primary-group2:
* ipaddr_gwf_102 (ocf:heartbeat:Dummy): Started rh96-02
* ipaddr_dbf_102 (ocf:heartbeat:Dummy): Started rh96-02
* ipaddr_api_102 (ocf:heartbeat:Dummy): Started rh96-02
* ipaddr_kafka_102 (ocf:heartbeat:Dummy): Started rh96-02
* ipaddr_unyo_102 (ocf:heartbeat:Dummy): Started rh96-02
* fs_share_102 (ocf:heartbeat:Dummy): Started rh96-02
* mw_jp1base_102 (ocf:heartbeat:Dummy): Started rh96-02
* mw_jp1ajs3_102 (ocf:heartbeat:Dummy): Started rh96-02
* Resource Group: primary-group3:
* ipaddr_gwf_103 (ocf:heartbeat:Dummy): Started rh96-03
* ipaddr_dbf_103 (ocf:heartbeat:Dummy): Started rh96-03
* ipaddr_api_103 (ocf:heartbeat:Dummy): Started rh96-03
* ipaddr_kafka_103 (ocf:heartbeat:Dummy): Started rh96-03
* ipaddr_unyo_103 (ocf:heartbeat:Dummy): Started rh96-03
* fs_share_103 (ocf:heartbeat:Dummy): Started rh96-03
* mw_jp1base_103 (ocf:heartbeat:Dummy): Started rh96-03
* mw_jp1ajs3_103 (ocf:heartbeat:Dummy): Started rh96-03
* Clone Set: ping-clone [ping]:
* Started: [ rh96-01 rh96-02 rh96-03 rh96-04 rh96-05 ]
* fence_sbd_101 (stonith:fence_sbd): Started rh96-05
* fence_sbd_102 (stonith:fence_sbd): Started rh96-03
* fence_sbd_103 (stonith:fence_sbd): Started rh96-05
* fence_sbd_104 (stonith:fence_sbd): Started rh96-02
* fence_sbd_105 (stonith:fence_sbd): Started rh96-01
>> After successfully building and starting the 5-node cluster, we induced a failure in the fourth node (test4), triggered STONITH, and then rejoined test4 to the cluster.
>> When we subsequently ran `pcs status --full` from the fourth node (test4), we observed an issue where resources actually running on nodes test2 and test3 were displayed as "Stopped."
>Just to make sure this is definitely a pacemaker problem, can you try
>using the crm_mon command instead of pcs? So in this case, use
>`crm_mon -1r` to get the same output.
The display remains incorrect even when running `crm_mon -r1`.
>> Furthermore, the display did not correct itself even after leaving the system for some time.
>I imagine this is because there haven't been any CIB changes, so the
>fenced node hasn't received an update. You could test this by doing
>something that would trigger a CIB update.
I attempted to add an attribute using `attrd_updater` after the issue occurred, but the display remained incorrect.
----
This issue has a very low reproduction rate.
Our investigation suggests that the problem occurs because, after the fourth node (rh96-04) rejoins the cluster, it discards the CIB synchronization communication sent from the DC node (rh96-02).
When the issue is reproduced, the `num_update` value on the fourth node is higher than the `num_update` value in the synchronization communication.
Subsequently, the `status/lrm` information is updated on both the DC node and the fourth node; however, the `status/lrm` information on the other nodes remains empty because the initial CIB synchronization communication from the DC node failed.
We believe this issue stems from the processing logic in Pacemaker 2.1.9, though we have not yet verified whether it occurs in the latest version of Pacemaker.
Is this issue specific to Pacemaker 2.1.9?
If it also occurs in the latest version of Pacemaker, we believe it requires a fix.
As a workaround, we believe that deleting `/var/lib/pacemaker/cib` on the fourth node (rh96-04) before it rejoins the cluster will prevent the `num_update` synchronization from being discarded.
What is your opinion on this workaround?
I am attaching the logs from the DC node(rh92) and the fourth node(rh96-04).
I have also attached the output of `cibadmin -Q` from the rh96-04 node taken at the time the issue occurred.
This shows that while the rh96-02 and rh96-04 nodes are synchronized, the `status/lrm` for the other nodes is not.
---
(cib-occure.xml)
<cib crm_feature_set="3.19.6" validate-with="pacemaker-3.10" epoch="2081" num_updates="89" admin_epoch="0" cib-last-written="Tue Sep 29 20:34:12 2026" update-origin="rh96-04" update-client="root" update-user="root" have-quorum="1" dc-uuid="2">
(snip)
<status>
<node_state id="5" uname="rh96-05" in_ccm="1790547553" crmd="1790547553" expected="member" crm-debug-origin="do_state_transition" join="member">
<transient_attributes id="5">
<instance_attributes id="status-5">
<nvpair id="status-5-.feature-set" name="#feature-set" value="3.19.6"/>
<nvpair id="status-5-ping-status" name="ping-status" value="1"/>
</instance_attributes>
</transient_attributes>
</node_state>
<node_state id="1" uname="rh96-01" in_ccm="1790547553" crmd="1790547553" expected="member" crm-debug-origin="do_state_transition" join="member">
<transient_attributes id="1">
<instance_attributes id="status-1">
<nvpair id="status-1-ping-status" name="ping-status" value="1"/>
<nvpair id="status-1-.feature-set" name="#feature-set" value="3.19.6"/>
</instance_attributes>
</transient_attributes>
</node_state>
<node_state id="3" uname="rh96-03" in_ccm="1790547553" crmd="1790547553" expected="member" crm-debug-origin="do_state_transition" join="member">
<transient_attributes id="3">
<instance_attributes id="status-3">
<nvpair id="status-3-ping-status" name="ping-status" value="1"/>
<nvpair id="status-3-.feature-set" name="#feature-set" value="3.19.6"/>
</instance_attributes>
</transient_attributes>
</node_state>
<node_state id="4" uname="rh96-04" crmd="1790681720" crm-debug-origin="controld_update_resource_history" in_ccm="1790681720" expected="member" join="member">
<transient_attributes id="4">
<instance_attributes id="status-4">
<nvpair id="status-4-.feature-set" name="#feature-set" value="3.19.6"/>
<nvpair id="status-4-ping-status" name="ping-status" value="1"/>
</instance_attributes>
</transient_attributes>
<lrm id="4">
<lrm_resources>
<lrm_resource id="ipaddr_gwf_101" class="ocf" provider="heartbeat" type="Dummy">
<lrm_rsc_op id="ipaddr_gwf_101_last_0" operation_key="ipaddr_gwf_101_monitor_0" operation="monitor" crm-debug-origin="controld_update_resource_history" crm_feature_set="3.19.6" transition-key="39:58713:7:0c377504-f9af-489f-94d8-2b330a14d50c" transition-magic="0:7;39:58713:7:0c377504-f9af-489f-94d8-2b330a14d50c" exit-reason="" on_node="rh96-04" call-id="8" rc-code="7" op-status="0" interval="0" last-rc-change="1790681721" exec-time="20" queue-time="0" op-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8" op-force-restart=" state " op-restart-digest="f2317cad3d54cec5d7d7aa7d0bf35cf8"/>
</lrm_resource>
(sinp)
</lrm_resources>
</lrm>
</node_state>
<node_state id="2" uname="rh96-02" in_ccm="1790547548" crmd="1790547548" expected="member" crm-debug-origin="do_state_transition" join="member">
<transient_attributes id="2">
<instance_attributes id="status-2">
<nvpair id="status-2-ping-status" name="ping-status" value="1"/>
<nvpair id="status-2-.feature-set" name="#feature-set" value="3.19.6"/>
</instance_attributes>
</transient_attributes>
<lrm id="2">
<lrm_resources>
<lrm_resource id="ping" type="ping" class="ocf" provider="pacemaker">
<lrm_rsc_op id="ping_last_0" operation_key="ping_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.19.6" transition-key="172:2:0:0c377504-f9af-489f-94d8-2b330a14d50c" transition-magic="0:0;172:2:0:0c377504-f9af-489f-94d8-2b330a14d50c" exit-reason="" on_node="rh96-02" call-id="110" rc-code="0" op-status="0" interval="0" last-rc-change="1790547631" exec-time="1157" queue-time="0" op-digest="80b024b55e51728e0645fdb450992611" op-force-restart=" use_fping failure_score options host_list multiplier name pidfile " op-restart-digest="b3528bd016da4ba85360c697ba159e17"/>
<lrm_rsc_op id="ping_monitor_10000" operation_key="ping_monitor_10000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.19.6" transition-key="173:2:0:0c377504-f9af-489f-94d8-2b330a14d50c" transition-magic="0:0;173:2:0:0c377504-f9af-489f-94d8-2b330a14d50c" exit-reason="" on_node="rh96-02" call-id="126" rc-code="0" op-status="0" interval="10000" last-rc-change="1790547632" exec-time="1047" queue-time="0" op-digest="93845f3e2b87f3b3efaaf5eba7311412"/>
</lrm_resource>
(snip)
</lrm_resources>
</lrm>
</node_state>
</status>
</cib>
---
Best Regards,
Hideo Yamauchi.
-------------- next part --------------
A non-text attachment was scrubbed...
Name: rh96-02-pacemaker.log
Type: application/octet-stream
Size: 1169684 bytes
Desc: not available
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20260930/9ffc086f/attachment-0002.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: rh96-04-pacemaker.log
Type: application/octet-stream
Size: 274438 bytes
Desc: not available
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20260930/9ffc086f/attachment-0003.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cib-occure.xml
Type: text/xml
Size: 75097 bytes
Desc: not available
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20260930/9ffc086f/attachment-0001.xml>
More information about the Users
mailing list