[ClusterLabs] Setting up an Active/Active Pacemaker cluster for a Postfix/Dovecot cluster, using a DRBD backend for the data storage
Damiano Giuliani
damianogiuliani87 at gmail.com
Tue Dec 5 12:29:52 EST 2023
It could be the watchdog? Are u using diskless watchdog?Two nodes are not
supported in diskless mode.
On Tue, Dec 5, 2023, 5:40 PM Raphael DUBOIS-LISKI <
raphael.dubois-liski at soget.fr> wrote:
> Hello,
>
>
>
> I am seeking help for the setup of an Active/Active pacemaker cluster that
> relies on a DRBD cluster as the data storage backend, as the solution is
> mounted on 2 RHEL9 VMs, the file system used is GFS2.
>
>
>
> Linked, is a PDF of the infrastructure that I am currently experimenting
> on.
>
>
>
> For context, this is my Pacemaker cluster config:
>
>
>
> Cluster Name: mycluster
>
> Corosync Nodes:
>
> Node1 Node2
>
> Pacemaker Nodes:
>
> Node1 Node2
>
>
>
> Resources:
>
> Clone: Data-clone
>
> Meta Attributes: Data-clone-meta_attributes
>
> clone-max=2
>
> clone-node-max=1
>
> notify=true
>
> promotable=true
>
> promoted-max=2
>
> promoted-node-max=1
>
> Resource: Data (class=ocf provider=linbit type=drbd)
>
> Attributes: Data-instance_attributes
>
> drbd_resource=drbd0
>
> Operations:
>
> demote: Data-demote-interval-0s
>
> interval=0s timeout=90
>
> monitor: Data-monitor-interval-60s
>
> interval=60s
>
> notify: Data-notify-interval-0s
>
> interval=0s timeout=90
>
> promote: Data-promote-interval-0s
>
> interval=0s timeout=90
>
> reload: Data-reload-interval-0s
>
> interval=0s timeout=30
>
> start: Data-start-interval-0s
>
> interval=0s timeout=240
>
> stop: Data-stop-interval-0s
>
> interval=0s timeout=100
>
> Clone: dlm-clone
>
> Meta Attributes: dlm-clone-meta_attributes
>
> clone-max=2
>
> clone-node-max=1
>
> Resource: dlm (class=ocf provider=pacemaker type=controld)
>
> Operations:
>
> monitor: dlm-monitor-interval-60s
>
> interval=60s
>
> start: dlm-start-interval-0s
>
> interval=0s timeout=90s
>
> stop: dlm-stop-interval-0s
>
> interval=0s timeout=100s
>
> Clone: FS-clone
>
> Resource: FS (class=ocf provider=heartbeat type=Filesystem)
>
> Attributes: FS-instance_attributes
>
> device=/dev/drbd0
>
> directory=/home/vusers
>
> fstype=gfs2
>
> Operations:
>
> monitor: FS-monitor-interval-20s
>
> interval=20s timeout=40s
>
> start: FS-start-interval-0s
>
> interval=0s timeout=60s
>
> stop: FS-stop-interval-0s
>
> interval=0s timeout=60s
>
> Clone: smtp_postfix-clone
>
> Meta Attributes: smtp_postfix-clone-meta_attributes
>
> clone-max=2
>
> clone-node-max=1
>
> Resource: smtp_postfix (class=ocf provider=heartbeat type=postfix)
>
> Operations:
>
> monitor: smtp_postfix-monitor-interval-60s
>
> interval=60s timeout=20s
>
> reload: smtp_postfix-reload-interval-0s
>
> interval=0s timeout=20s
>
> start: smtp_postfix-start-interval-0s
>
> interval=0s timeout=20s
>
> stop: smtp_postfix-stop-interval-0s
>
> interval=0s timeout=20s
>
> Clone: WebSite-clone
>
> Resource: WebSite (class=ocf provider=heartbeat type=apache)
>
> Attributes: WebSite-instance_attributes
>
> configfile=/etc/httpd/conf/httpd.conf
>
> statusurl=http://localhost/server-status
>
> Operations:
>
> monitor: WebSite-monitor-interval-1min
>
> interval=1min
>
> start: WebSite-start-interval-0s
>
> interval=0s timeout=40s
>
> stop: WebSite-stop-interval-0s
>
> interval=0s timeout=60s
>
>
>
> Colocation Constraints:
>
> resource 'FS-clone' with Promoted resource 'Data-clone' (id:
> colocation-FS-
>
> Data-clone-INFINITY)
>
> score=INFINITY
>
> resource 'WebSite-clone' with resource 'FS-clone' (id:
> colocation-WebSite-FS-
>
> INFINITY)
>
> score=INFINITY
>
> resource 'FS-clone' with resource 'dlm-clone' (id:
> colocation-FS-dlm-clone-
>
> INFINITY)
>
> score=INFINITY
>
> resource 'FS-clone' with resource 'smtp_postfix-clone' (id:
> colocation-FS-
>
> clone-smtp_postfix-clone-INFINITY)
>
> score=INFINITY
>
> Order Constraints:
>
> promote resource 'Data-clone' then start resource 'FS-clone' (id:
> order-Data-
>
> clone-FS-mandatory)
>
> start resource 'FS-clone' then start resource 'WebSite-clone' (id:
> order-FS-
>
> WebSite-mandatory)
>
> start resource 'dlm-clone' then start resource 'FS-clone' (id: order-dlm-
>
> clone-FS-mandatory)
>
> start resource 'FS-clone' then start resource 'smtp_postfix-clone' (id:
> order-
>
> FS-clone-smtp_postfix-clone-mandatory)
>
>
>
> Resources Defaults:
>
> Meta Attrs: build-resource-defaults
>
> resource-stickiness=1 (id: build-resource-stickiness)
>
>
>
> Operations Defaults:
>
> Meta Attrs: op_defaults-meta_attributes
>
> timeout=240s (id: op_defaults-meta_attributes-timeout)
>
>
>
> Cluster Properties: cib-bootstrap-options
>
> cluster-infrastructure=corosync
>
> cluster-name=mycluster
>
> dc-version=2.1.6-9.el9-6fdc9deea29
>
> have-watchdog=true
>
> last-lrm-refresh=1701787695
>
> no-quorum-policy=ignore
>
> stonith-enabled=true
>
> stonith-watchdog-timeout=10
>
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> And this is my DRBD configuration :
>
>
>
> global {
>
> usage-count no;
>
> }
>
> common {
>
> disk {
>
> resync-rate 100M;
>
> al-extents 257;
>
> }
>
> }
>
> resource drbd0 {
>
> protocol C;
>
> handlers {
>
> pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh;
> /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trig;
>
> pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh;
> /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trig;
>
> local-io-error "/usr/lib/drbd/notify-io-error.sh;
> /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ;
> halt;
>
> fence-peer "/usr/lib/drbd/crm-fence-peer.9.sh";
>
> after-resync-target "/usr/lib/drbd/crm-unfence-peer.9.sh";
>
> split-brain "/usr/lib/drbd/notify-split-brain.sh root";
>
> out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
>
> }
>
> startup {
>
> wfc-timeout 1;
>
> degr-wfc-timeout 1;
>
> become-primary-on both;
>
> }
>
> net {
>
> # The following lines are dedicated to handle
>
> # split-brain situations (e.g., if one of the nodes fails)
>
> after-sb-0pri discard-zero-changes; # If both nodes are secondary,
> just make one of them primary
>
> after-sb-1pri discard-secondary; # If one is primary, one is not,
> trust the primary node
>
> after-sb-2pri disconnect;
>
> allow-two-primaries yes;
>
> verify-alg sha1;
>
> }
>
> disk {
>
> on-io-error detach;
>
> }
>
> options {
>
> auto-promote yes;
>
> }
>
> on fradevtestmail1 {
>
> device /dev/drbd0;
>
> disk /dev/rootvg/drbdlv;
>
> address X.X.X.X:7788;
>
> flexible-meta-disk internal;
>
> }
>
> on fradevtestmail2 {
>
> device /dev/drbd0;
>
> disk /dev/rootvg/drbdlv;
>
> address X.X.X.X:7788;
>
> flexible-meta-disk internal;
>
> }
>
> }
>
>
>
>
>
> Knowing all this,
>
>
>
> The cluster works perfectly as expected when both nodes are up, but a
> problem arises when I put the cluster in a degraded state by killing one of
> the nodes improperly (to simulate an unexpected crash).
>
>
>
> This causes the remaining node to reboot, restart the cluster, with all
> going well in the resource start process, until it's time to mount the File
> System, where it times out and fails.
>
>
>
> Would you have any idea why this behaviour happens, and possibly how I
> would be able to fix this behaviour, so that the cluster is still usable
> even with one node down?
>
> Until we can get the second node back and running in case of an unexpected
> crash ?
>
>
>
> Many thanks for your help,
>
>
>
> Have a nice day,
>
>
>
> BR,
>
>
>
>
>
>
>
> [image: SOGET]
>
> *Raphael DUBOIS-LISKI*
> *Ingénieur Système et Réseau*
> +33 2 35 19 25 54
>
> SOGET SA • 4, rue des Lamaneurs • 76600 Le Havre, FR
>
> [image: web]
> <https://signature.soget.fr/l/dnA0alZ4cjRrTERoZlFWSDhaN0xYdz09-VzNGNG5oU1pWYjZRMTVQMUxBZ2xJZz09>
>
> [image: linkedin]
> <https://signature.soget.fr/l/dnA0alZ4cjRrTERoZlFWSDhaN0xYdz09-eFZMaExNZWZGQjMvaVVJaDArTTl6Zz09>
>
> [image: twitter]
> <https://signature.soget.fr/l/dnA0alZ4cjRrTERoZlFWSDhaN0xYdz09-VjFWNTBIYlNCUDdIbXlxKzJyRzFPUT09>
>
> Disclaimer <http://soget.fr/disclaimer>
>
>
> _______________________________________________
> Manage your subscription:
> https://lists.clusterlabs.org/mailman/listinfo/users
>
> ClusterLabs home: https://www.clusterlabs.org/
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20231205/8e4fb3e3/attachment-0001.htm>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image001.png
Type: image/png
Size: 6919 bytes
Desc: not available
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20231205/8e4fb3e3/attachment-0005.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 1215 bytes
Desc: not available
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20231205/8e4fb3e3/attachment-0006.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image003.png
Type: image/png
Size: 828 bytes
Desc: not available
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20231205/8e4fb3e3/attachment-0007.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image004.png
Type: image/png
Size: 1088 bytes
Desc: not available
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20231205/8e4fb3e3/attachment-0008.png>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: image002.png
Type: image/png
Size: 1215 bytes
Desc: not available
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20231205/8e4fb3e3/attachment-0009.png>
More information about the Users
mailing list