<html xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Aptos;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
font-size:12.0pt;
font-family:"Aptos",sans-serif;
mso-ligatures:standardcontextual;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#467886;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Aptos",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 72.0pt 72.0pt 72.0pt;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang="EN-US" link="#467886" vlink="#96607D" style="word-wrap:break-word">
<div class="WordSection1">
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Hello<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">There is two-node (HPE DL345 Gen12 servers) shared-nothing DRBD-based sync (Protocol C) replication, distributed active/standby pacemaker storage metro-cluster. The distributed active/standby pacemaker storage metro-cluster configured with
qdevice, heuristics (parallel fping) and fencing - fence_ipmilan and diskless sbd (hpwdt, /dev/watchdog). All cluster resources are configured to always run together on the same node.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">The two storage cluster nodes and qdevice running on Rocky Linux 10.1<o:p></o:p></p>
<p class="MsoNormal">Pacemaker version 3.0.1<o:p></o:p></p>
<p class="MsoNormal">Corosync version 3.1.9<o:p></o:p></p>
<p class="MsoNormal">DRBD version 9.3.0<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">So, the question is – what is the most correct way of implementing STONITH/fencing with fence_iomilan + diskless sbd (hpwdt, /dev/watchdog) ?<o:p></o:p></p>
<p class="MsoNormal">I’m not sure about two-level fencing topology, because diskless sbd is not an external agent/resource…<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Currently it works without fencing topology, and both running in “parallel”. Really no matter who wins. I just want to make sure fenced node is powered off of rebooted.<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Here is log how it works now in “parallel”,<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">[root@memverge2 ~]# cat /var/log/messages|grep -i fence<o:p></o:p></p>
<p class="MsoNormal">Feb 2 12:46:07 memverge2 pacemaker-fenced[3902]: notice: Node memverge state is now lost<o:p></o:p></p>
<p class="MsoNormal">Feb 2 12:46:07 memverge2 pacemaker-fenced[3902]: notice: Removed 1 inactive node with cluster layer ID 27 from the membership cache<o:p></o:p></p>
<p class="MsoNormal">Feb 2 12:46:10 memverge2 pacemaker-schedulerd[3905]: warning: Cluster node memverge will be fenced: peer is no longer part of the cluster<o:p></o:p></p>
<p class="MsoNormal">Feb 2 12:46:10 memverge2 pacemaker-schedulerd[3905]: warning: ipmi-fence-memverge2_stop_0 on memverge is unrunnable (node is offline)<o:p></o:p></p>
<p class="MsoNormal">Feb 2 12:46:10 memverge2 pacemaker-schedulerd[3905]: warning: ipmi-fence-memverge2_stop_0 on memverge is unrunnable (node is offline)<o:p></o:p></p>
<p class="MsoNormal">Feb 2 12:46:10 memverge2 pacemaker-schedulerd[3905]: notice: Actions: Fence (reboot) memverge 'peer is no longer part of the cluster'<o:p></o:p></p>
<p class="MsoNormal">Feb 2 12:46:10 memverge2 pacemaker-schedulerd[3905]: notice: Actions: Stop ipmi-fence-memverge2 ( memverge ) due to node availability<o:p></o:p></p>
<p class="MsoNormal">Feb 2 12:46:10 memverge2 pacemaker-fenced[3902]: notice: Client pacemaker-controld.3906 wants to fence (reboot) memverge using any device<o:p></o:p></p>
<p class="MsoNormal">Feb 2 12:46:10 memverge2 pacemaker-fenced[3902]: notice: Requesting peer fencing (reboot) targeting memverge<o:p></o:p></p>
<p class="MsoNormal">Feb 2 12:46:10 memverge2 pacemaker-fenced[3902]: notice: Requesting that memverge2 perform 'reboot' action targeting memverge<o:p></o:p></p>
<p class="MsoNormal">Feb 2 12:46:10 memverge2 pacemaker-fenced[3902]: notice: Waiting 25s for memverge to self-fence (reboot) for client pacemaker-controld.3906<o:p></o:p></p>
<p class="MsoNormal">Feb 2 12:46:10 memverge2 pacemaker-fenced[3902]: notice: Delaying 'reboot' action targeting memverge using ipmi-fence-memverge for 5s<o:p></o:p></p>
<p class="MsoNormal">Feb 2 12:46:36 memverge2 pacemaker-fenced[3902]: notice: Self-fencing (reboot) by memverge for pacemaker-controld.3906 assumed complete<o:p></o:p></p>
<p class="MsoNormal">Feb 2 12:46:36 memverge2 pacemaker-fenced[3902]: notice: Operation 'reboot' targeting memverge by memverge2 for
<a href="mailto:pacemaker-controld.3906@memverge2">pacemaker-controld.3906@memverge2</a>: OK (Done)<o:p></o:p></p>
<p class="MsoNormal">Feb 2 12:46:36 memverge2 kernel: drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer<o:p></o:p></p>
<p class="MsoNormal">Feb 2 12:46:36 memverge2 kernel: drbd ha-iscsi memverge: helper command: /sbin/drbdadm fence-peer<o:p></o:p></p>
<p class="MsoNormal">Feb 2 12:46:36 memverge2 crm-fence-peer.9.sh[7332]: DRBD_BACKING_DEV_1=/dev/mapper/object_block_nfs_vg-ha_nfs_exports_lv_with_vdo_1x8 DRBD_BACKING_DEV_2=/dev/mapper/object_block_nfs_vg-ha_nfs_internal_lv_without_vdo DRBD_BACKING_DEV_5=/dev/mapper/object_block_nfs_vg-ha_samba_exports_lv_with_vdo_1x8
DRBD_CONF=/etc/drbd.conf DRBD_CSTATE=Connecting DRBD_LL_DISK=/dev/mapper/object_block_nfs_vg-ha_nfs_exports_lv_with_vdo_1x8\ /dev/mapper/object_block_nfs_vg-ha_nfs_internal_lv_without_vdo\ /dev/mapper/object_block_nfs_vg-ha_samba_exports_lv_with_vdo_1x8 DRBD_MINOR=1\
2\ 5 DRBD_MINOR_1=1 DRBD_MINOR_2=2 DRBD_MINOR_5=5 DRBD_MY_ADDRESS=192.168.0.8 DRBD_MY_AF=ipv4 DRBD_MY_NODE_ID=28 DRBD_NODE_ID_27=memverge DRBD_NODE_ID_28=memverge2 DRBD_PEER_ADDRESS=192.168.0.6 DRBD_PEER_AF=ipv4 DRBD_PEER_NODE_ID=27 DRBD_RESOURCE=ha-nfs DRBD_VOLUME=1\
2\ 5 UP_TO_DATE_NODES=0x10000000 /usr/lib/drbd/crm-fence-peer.9.sh<o:p></o:p></p>
<p class="MsoNormal">Feb 2 12:46:36 memverge2 crm-fence-peer.9.sh[7333]: DRBD_BACKING_DEV_3=/dev/mapper/object_block_nfs_vg-ha_block_exports_lv_with_vdo_1x8 DRBD_BACKING_DEV_4=/dev/mapper/object_block_nfs_vg-ha_block_exports_lv_without_vdo DRBD_CONF=/etc/drbd.conf
DRBD_CSTATE=Connecting DRBD_LL_DISK=/dev/mapper/object_block_nfs_vg-ha_block_exports_lv_with_vdo_1x8\ /dev/mapper/object_block_nfs_vg-ha_block_exports_lv_without_vdo DRBD_MINOR=3\ 4 DRBD_MINOR_3=3 DRBD_MINOR_4=4 DRBD_MY_ADDRESS=192.168.0.8 DRBD_MY_AF=ipv4
DRBD_MY_NODE_ID=28 DRBD_NODE_ID_27=memverge DRBD_NODE_ID_28=memverge2 DRBD_PEER_ADDRESS=192.168.0.6 DRBD_PEER_AF=ipv4 DRBD_PEER_NODE_ID=27 DRBD_RESOURCE=ha-iscsi DRBD_VOLUME=3\ 4 UP_TO_DATE_NODES=0x10000000 /usr/lib/drbd/crm-fence-peer.9.sh<o:p></o:p></p>
<p class="MsoNormal">Feb 2 12:46:36 memverge2 crm-fence-peer.9.sh[7333]: INFO Concurrency check: Peer is already marked clean/fenced by another resource. Returning success (Exit 4).<o:p></o:p></p>
<p class="MsoNormal">Feb 2 12:46:36 memverge2 crm-fence-peer.9.sh[7332]: INFO Concurrency check: Peer is already marked clean/fenced by another resource. Returning success (Exit 4).<o:p></o:p></p>
<p class="MsoNormal">Feb 2 12:46:36 memverge2 kernel: drbd ha-iscsi memverge: helper command: /sbin/drbdadm fence-peer exit code 4 (0x400)<o:p></o:p></p>
<p class="MsoNormal">Feb 2 12:46:36 memverge2 kernel: drbd ha-iscsi memverge: fence-peer helper returned 4 (peer was fenced)<o:p></o:p></p>
<p class="MsoNormal">Feb 2 12:46:36 memverge2 kernel: drbd ha-nfs memverge: helper command: /sbin/drbdadm fence-peer exit code 4 (0x400)<o:p></o:p></p>
<p class="MsoNormal">Feb 2 12:46:36 memverge2 kernel: drbd ha-nfs memverge: fence-peer helper returned 4 (peer was fenced)<o:p></o:p></p>
<p class="MsoNormal">Feb 2 12:46:37 memverge2 pacemaker-fenced[3902]: notice: Operation 'reboot' [7068] targeting memverge using ipmi-fence-memverge returned 0<o:p></o:p></p>
<p class="MsoNormal">Feb 2 12:46:37 memverge2 pacemaker-fenced[3902]: notice: Operation 'reboot' targeting memverge by memverge2 for
<a href="mailto:pacemaker-controld.3906@memverge2">pacemaker-controld.3906@memverge2</a>: Result arrived too late<o:p></o:p></p>
<p class="MsoNormal">[root@memverge2 ~]#<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
<p class="MsoNormal">Anton<o:p></o:p></p>
<p class="MsoNormal"><o:p> </o:p></p>
</div>
</body>
</html>