[ClusterLabs] Setting up an Active/Active Pacemaker cluster for a Postfix/Dovecot cluster, using a DRBD backend for the data storage
Raphael DUBOIS-LISKI
raphael.dubois-liski at soget.fr
Tue Dec 5 11:40:29 EST 2023
Hello,
I am seeking help for the setup of an Active/Active pacemaker cluster that relies on a DRBD cluster as the data storage backend, as the solution is mounted on 2 RHEL9 VMs, the file system used is GFS2.
Linked, is a PDF of the infrastructure that I am currently experimenting on.
For context, this is my Pacemaker cluster config:
Cluster Name: mycluster
Corosync Nodes:
Node1 Node2
Pacemaker Nodes:
Node1 Node2
Resources:
Clone: Data-clone
Meta Attributes: Data-clone-meta_attributes
clone-max=2
clone-node-max=1
notify=true
promotable=true
promoted-max=2
promoted-node-max=1
Resource: Data (class=ocf provider=linbit type=drbd)
Attributes: Data-instance_attributes
drbd_resource=drbd0
Operations:
demote: Data-demote-interval-0s
interval=0s timeout=90
monitor: Data-monitor-interval-60s
interval=60s
notify: Data-notify-interval-0s
interval=0s timeout=90
promote: Data-promote-interval-0s
interval=0s timeout=90
reload: Data-reload-interval-0s
interval=0s timeout=30
start: Data-start-interval-0s
interval=0s timeout=240
stop: Data-stop-interval-0s
interval=0s timeout=100
Clone: dlm-clone
Meta Attributes: dlm-clone-meta_attributes
clone-max=2
clone-node-max=1
Resource: dlm (class=ocf provider=pacemaker type=controld)
Operations:
monitor: dlm-monitor-interval-60s
interval=60s
start: dlm-start-interval-0s
interval=0s timeout=90s
stop: dlm-stop-interval-0s
interval=0s timeout=100s
Clone: FS-clone
Resource: FS (class=ocf provider=heartbeat type=Filesystem)
Attributes: FS-instance_attributes
device=/dev/drbd0
directory=/home/vusers
fstype=gfs2
Operations:
monitor: FS-monitor-interval-20s
interval=20s timeout=40s
start: FS-start-interval-0s
interval=0s timeout=60s
stop: FS-stop-interval-0s
interval=0s timeout=60s
Clone: smtp_postfix-clone
Meta Attributes: smtp_postfix-clone-meta_attributes
clone-max=2
clone-node-max=1
Resource: smtp_postfix (class=ocf provider=heartbeat type=postfix)
Operations:
monitor: smtp_postfix-monitor-interval-60s
interval=60s timeout=20s
reload: smtp_postfix-reload-interval-0s
interval=0s timeout=20s
start: smtp_postfix-start-interval-0s
interval=0s timeout=20s
stop: smtp_postfix-stop-interval-0s
interval=0s timeout=20s
Clone: WebSite-clone
Resource: WebSite (class=ocf provider=heartbeat type=apache)
Attributes: WebSite-instance_attributes
configfile=/etc/httpd/conf/httpd.conf
statusurl=http://localhost/server-status
Operations:
monitor: WebSite-monitor-interval-1min
interval=1min
start: WebSite-start-interval-0s
interval=0s timeout=40s
stop: WebSite-stop-interval-0s
interval=0s timeout=60s
Colocation Constraints:
resource 'FS-clone' with Promoted resource 'Data-clone' (id: colocation-FS-
Data-clone-INFINITY)
score=INFINITY
resource 'WebSite-clone' with resource 'FS-clone' (id: colocation-WebSite-FS-
INFINITY)
score=INFINITY
resource 'FS-clone' with resource 'dlm-clone' (id: colocation-FS-dlm-clone-
INFINITY)
score=INFINITY
resource 'FS-clone' with resource 'smtp_postfix-clone' (id: colocation-FS-
clone-smtp_postfix-clone-INFINITY)
score=INFINITY
Order Constraints:
promote resource 'Data-clone' then start resource 'FS-clone' (id: order-Data-
clone-FS-mandatory)
start resource 'FS-clone' then start resource 'WebSite-clone' (id: order-FS-
WebSite-mandatory)
start resource 'dlm-clone' then start resource 'FS-clone' (id: order-dlm-
clone-FS-mandatory)
start resource 'FS-clone' then start resource 'smtp_postfix-clone' (id: order-
FS-clone-smtp_postfix-clone-mandatory)
Resources Defaults:
Meta Attrs: build-resource-defaults
resource-stickiness=1 (id: build-resource-stickiness)
Operations Defaults:
Meta Attrs: op_defaults-meta_attributes
timeout=240s (id: op_defaults-meta_attributes-timeout)
Cluster Properties: cib-bootstrap-options
cluster-infrastructure=corosync
cluster-name=mycluster
dc-version=2.1.6-9.el9-6fdc9deea29
have-watchdog=true
last-lrm-refresh=1701787695
no-quorum-policy=ignore
stonith-enabled=true
stonith-watchdog-timeout=10
And this is my DRBD configuration :
global {
usage-count no;
}
common {
disk {
resync-rate 100M;
al-extents 257;
}
}
resource drbd0 {
protocol C;
handlers {
pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trig;
pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trig;
local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt;
fence-peer "/usr/lib/drbd/crm-fence-peer.9.sh";
after-resync-target "/usr/lib/drbd/crm-unfence-peer.9.sh";
split-brain "/usr/lib/drbd/notify-split-brain.sh root";
out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
}
startup {
wfc-timeout 1;
degr-wfc-timeout 1;
become-primary-on both;
}
net {
# The following lines are dedicated to handle
# split-brain situations (e.g., if one of the nodes fails)
after-sb-0pri discard-zero-changes; # If both nodes are secondary, just make one of them primary
after-sb-1pri discard-secondary; # If one is primary, one is not, trust the primary node
after-sb-2pri disconnect;
allow-two-primaries yes;
verify-alg sha1;
}
disk {
on-io-error detach;
}
options {
auto-promote yes;
}
on fradevtestmail1 {
device /dev/drbd0;
disk /dev/rootvg/drbdlv;
address X.X.X.X:7788;
flexible-meta-disk internal;
}
on fradevtestmail2 {
device /dev/drbd0;
disk /dev/rootvg/drbdlv;
address X.X.X.X:7788;
flexible-meta-disk internal;
}
}
Knowing all this,
The cluster works perfectly as expected when both nodes are up, but a problem arises when I put the cluster in a degraded state by killing one of the nodes improperly (to simulate an unexpected crash).
This causes the remaining node to reboot, restart the cluster, with all going well in the resource start process, until it's time to mount the File System, where it times out and fails.
Would you have any idea why this behaviour happens, and possibly how I would be able to fix this behaviour, so that the cluster is still usable even with one node down?
Until we can get the second node back and running in case of an unexpected crash ?
Many thanks for your help,
Have a nice day,
BR,
[SOGET]
Raphael DUBOIS-LISKI
Ingénieur Système et Réseau
+33 2 35 19 25 54
SOGET SA * 4, rue des Lamaneurs * 76600 Le Havre, FR
[web]<https://signature.soget.fr/l/dnA0alZ4cjRrTERoZlFWSDhaN0xYdz09-VzNGNG5oU1pWYjZRMTVQMUxBZ2xJZz09>
[linkedin]<https://signature.soget.fr/l/dnA0alZ4cjRrTERoZlFWSDhaN0xYdz09-eFZMaExNZWZGQjMvaVVJaDArTTl6Zz09>
[twitter]<https://signature.soget.fr/l/dnA0alZ4cjRrTERoZlFWSDhaN0xYdz09-VjFWNTBIYlNCUDdIbXlxKzJyRzFPUT09>
Disclaimer<http://soget.fr/disclaimer>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20231205/7b3a337b/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 6919 bytes
Desc: image001.png
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20231205/7b3a337b/attachment-0004.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 1215 bytes
Desc: image002.png
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20231205/7b3a337b/attachment-0005.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 828 bytes
Desc: image003.png
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20231205/7b3a337b/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.png
Type: image/png
Size: 1088 bytes
Desc: image004.png
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20231205/7b3a337b/attachment-0007.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: Infrastructure.pdf
Type: application/pdf
Size: 152300 bytes
Desc: Infrastructure.pdf
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20231205/7b3a337b/attachment-0001.pdf>
More information about the Users
mailing list