[ClusterLabs] DRBD split-brain investigations, automatic fixes and manual intervention...

[Andrei Borzenkov]

On Wed, Oct 20, 2021 at 11:54 AM Ian Diddams via Users
<users at clusterlabs.org> wrote:
>
> So - system logs recently show this
>
> ESTRELA
> Oct 18th
> Oct 18 04:04:28 wp-vldyn-estrela kernel: [584651.491139] drbd mysql01/0 drbd0: Split-Brain detected, 1 primaries, automatically solved. Sync from peer node
> Oct 18 04:04:28 wp-vldyn-estrela kernel: [584651.491139] drbd mysql01/0 drbd0: Split-Brain detected, 1 primaries, automatically solved. Sync from peer node
>
> Oct 19th
> Oct 19 03:45:43 wp-vldyn-estrela kernel: [47892.092191] drbd mysql01/0 drbd0: Split-Brain detected but unresolved, dropping connection!
> Oct 19 03:45:43 wp-vldyn-estrela kernel: [47892.092191] drbd mysql01/0 drbd0: Split-Brain detected but unresolved, dropping connection!
>
>
> RAFEIRO
> Oct 18
> Oct 18 04:04:28 wp-vldyn-rafeiro kernel: [584652.907126] drbd mysql01/0 drbd0: Split-Brain detected, 1 primaries, automatically solved. Sync from this node
> Oct 18 04:04:28 wp-vldyn-rafeiro kernel: [584652.907126] drbd mysql01/0 drbd0: Split-Brain detected, 1 primaries, automatically solved. Sync from this node
>
> Oct 19
> Oct 19 03:45:43 wp-vldyn-rafeiro kernel: [47864.401284] drbd mysql01/0 drbd0: Split-Brain detected but unresolved, dropping connection!
> Oct 19 03:45:43 wp-vldyn-rafeiro kernel: [47864.401284] drbd mysql01/0 drbd0: Split-Brain detected but unresolved, dropping connection!
>
>
>
> So on the 18th the split-brain issues was detected but (automatically?) fixed.
> But on the 19th it wasnt...
>

18th is probably initial synchronization. In this case DRBD knows who
is who and which side is primary.

You cannot resolve split brain without fencing. This is as simple as
that. Your pacemaker configuration (from another mail) shows

> pcs -f clust_cfg property set stonith-enabled=false
> pcs -f clust_cfg property set no-quorum-policy=ignore

This is a recipe for data corruption.