[ClusterLabs] drbd9 3 node "neverending" sync?

Mevo Govo govomevo at gmail.com
Mon Oct 8 11:01:32 EDT 2018


Hi,

When I assign an email sender handler to resynch events, it fill my email
account. (because of the lot of Outdated/Inconsistent/UpToDate cycles, I
think)

handlers {
  after-resync-target  "echo after-resync-target  | mail -r
root.$(hostname)@<my domain> -s after-resync-target  <my email>";
  before-resync-target "echo before-resync-target | mail -r
root.$(hostname)@<my domain> -s before-resync-target <my email>";
  ...
}

B.r: Laszlo.


Mevo Govo <govomevo at gmail.com> ezt írta (időpont: 2018. okt. 8., H, 14:12):

> Hi,
>
> I tried it with and without drbd quorum options, I resterted all the node,
> but same "neverending" refresh. Device is not mounted, so no "outer"
> modification. protocol is C (synchron). I recreated the resource on node 3
> (xetest3) with tis commands:
>
> drbdadm down r0
> dd if=/dev/zero of=/dev/sdb1 bs=100M count=1
> drbdadm create-md r0
>
> After this "full resync", the neverending sync stopped (UpToDate on every
> nodes). But after a resource down/up, the "Outdated/Inconsistent/UpToDate"
> cycle is restarted on xetest3.
> Maybe it is normal on 3rd node, even if protocol C. But I do not see it on
> our prod environment.
>
> B.r: Laszlo.
>
>
>
> Mevo Govo <govomevo at gmail.com> ezt írta (időpont: 2018. okt. 8., H,
> 11:20):
>
>> Hi,
>>
>> we are testing drbd9 with 3 node. After a test (network disable, then
>> enable on one host), the syncronization is not finishing.
>> As a recovery, I switched the pacemaker off, and I tried "primary
>> --force" on one node and "--discard-my-data" on the other nodes. It seem
>> solve the problem (everything is UpToDate), but after I down/up the
>> resource, the "neverending" synchronization is restarted (when the resource
>> is Secondary on all the 3 node).
>> Why sync is restarted after UpToDate state. Ho can I solve this problem?
>> Nodes are xetes1, xetest2, xetest3, here is a checking test ("drbdadm
>> status" on every node, then wait 10 secod. This repeated 10 times)
>>
>> Thanks for answers: Laci.
>>
>>
>> for i in {1..10}
>> do
>>   echo "################## check $i ##################"
>>   sleep 10
>>   date
>>   echo '################## on node xetest1'
>>   ssh xetest1 sudo drbdadm status
>>   echo '################## on node xetest2'
>>   ssh xetest2 sudo drbdadm status
>>   echo '################## on node xetest3'
>>   ssh xetest3 sudo drbdadm status
>> done
>>
>> ################## check 1 ##################
>>
>>
>>
>> Mon Oct  8 09:11:12 GMT 2018
>> ################## on node xetest1
>> r0 role:Secondary
>>   disk:UpToDate
>>   xetest2 role:Secondary
>>     peer-disk:UpToDate
>>   xetest3 role:Secondary
>>     peer-disk:UpToDate
>>
>> ################## on node xetest2
>> r0 role:Secondary
>>   disk:UpToDate
>>   xetest1 role:Secondary
>>     peer-disk:UpToDate
>>   xetest3 role:Secondary
>>     peer-disk:Inconsistent resync-suspended:peer
>>
>> ################## on node xetest3
>> r0 role:Secondary
>>   disk:Outdated blocked:upper
>>   xetest1 role:Secondary
>>     replication:WFBitMapT peer-disk:UpToDate
>>   xetest2 role:Secondary
>>     peer-disk:UpToDate
>>
>> ################## check 2 ##################
>> Mon Oct  8 09:11:23 GMT 2018
>> ################## on node xetest1
>> r0 role:Secondary
>>   disk:UpToDate
>>   xetest2 role:Secondary
>>     peer-disk:UpToDate
>>   xetest3 role:Secondary
>>     peer-disk:Inconsistent resync-suspended:peer
>>
>> ################## on node xetest2
>> r0 role:Secondary
>>   disk:UpToDate
>>   xetest1 role:Secondary
>>     peer-disk:UpToDate
>>   xetest3 role:Secondary
>>     replication:SyncSource peer-disk:Inconsistent
>>
>> ################## on node xetest3
>> r0 role:Secondary
>>   disk:Outdated blocked:upper
>>   xetest1 role:Secondary
>>     peer-disk:UpToDate
>>   xetest2 role:Secondary
>>     replication:WFBitMapT peer-disk:UpToDate
>>
>> ################## check 3 ##################
>> Mon Oct  8 09:11:35 GMT 2018
>> ################## on node xetest1
>> r0 role:Secondary
>>   disk:UpToDate
>>   xetest2 role:Secondary
>>     peer-disk:UpToDate
>>   xetest3 role:Secondary
>>     replication:SyncSource peer-disk:Inconsistent
>>
>> ################## on node xetest2
>> r0 role:Secondary
>>   disk:UpToDate
>>   xetest1 role:Secondary
>>     peer-disk:UpToDate
>>   xetest3 role:Secondary
>>     replication:SyncSource peer-disk:Inconsistent
>>
>> ################## on node xetest3
>> r0 role:Secondary
>>   disk:UpToDate
>>   xetest1 role:Secondary
>>     peer-disk:UpToDate
>>   xetest2 role:Secondary
>>     peer-disk:UpToDate
>>
>> ################## check 4 ##################
>> Mon Oct  8 09:11:46 GMT 2018
>> ################## on node xetest1
>> r0 role:Secondary
>>   disk:UpToDate
>>   xetest2 role:Secondary
>>     peer-disk:UpToDate
>>   xetest3 role:Secondary
>>     replication:WFBitMapS peer-disk:Outdated
>>
>> ################## on node xetest2
>> r0 role:Secondary
>>   disk:UpToDate
>>   xetest1 role:Secondary
>>     peer-disk:UpToDate
>>   xetest3 role:Secondary
>>     replication:SyncSource peer-disk:Inconsistent
>>
>> ################## on node xetest3
>> r0 role:Secondary
>>   disk:Inconsistent
>>   xetest1 role:Secondary
>>     replication:SyncTarget peer-disk:UpToDate
>>   xetest2 role:Secondary
>>     peer-disk:UpToDate
>>
>> ################## check 5 ##################
>> Mon Oct  8 09:11:58 GMT 2018
>> ################## on node xetest1
>> r0 role:Secondary
>>   disk:UpToDate
>>   xetest2 role:Secondary
>>     peer-disk:UpToDate
>>   xetest3 role:Secondary
>>     peer-disk:Outdated
>>
>> ################## on node xetest2
>> r0 role:Secondary
>>   disk:UpToDate
>>   xetest1 role:Secondary
>>     peer-disk:UpToDate
>>   xetest3 role:Secondary
>>     replication:SyncSource peer-disk:Inconsistent
>>
>> ################## on node xetest3
>> r0 role:Secondary
>>   disk:Outdated blocked:upper
>>   xetest1 role:Secondary
>>     replication:WFBitMapT peer-disk:UpToDate
>>   xetest2 role:Secondary
>>     peer-disk:UpToDate
>>
>> ################## check 6 ##################
>> Mon Oct  8 09:12:09 GMT 2018
>> ################## on node xetest1
>> r0 role:Secondary
>>   disk:UpToDate
>>   xetest2 role:Secondary
>>     peer-disk:UpToDate
>>   xetest3 role:Secondary
>>     peer-disk:UpToDate
>>
>> ################## on node xetest2
>> r0 role:Secondary
>>   disk:UpToDate
>>   xetest1 role:Secondary
>>     peer-disk:UpToDate
>>   xetest3 role:Secondary
>>     peer-disk:Inconsistent resync-suspended:peer
>>
>> ################## on node xetest3
>> r0 role:Secondary
>>   disk:Inconsistent
>>   xetest1 role:Secondary
>>     replication:SyncTarget peer-disk:UpToDate
>>   xetest2 role:Secondary
>>     peer-disk:UpToDate
>>
>> ################## check 7 ##################
>> Mon Oct  8 09:12:21 GMT 2018
>> ################## on node xetest1
>> r0 role:Secondary
>>   disk:UpToDate
>>   xetest2 role:Secondary
>>     peer-disk:UpToDate
>>   xetest3 role:Secondary
>>     peer-disk:UpToDate
>>
>> ################## on node xetest2
>> r0 role:Secondary
>>   disk:UpToDate
>>   xetest1 role:Secondary
>>     peer-disk:UpToDate
>>   xetest3 role:Secondary
>>     replication:SyncSource peer-disk:Inconsistent
>>
>> ################## on node xetest3
>> r0 role:Secondary
>>   disk:Outdated blocked:upper
>>   xetest1 role:Secondary
>>     peer-disk:UpToDate
>>   xetest2 role:Secondary
>>     replication:WFBitMapT peer-disk:UpToDate
>>
>> ################## check 8 ##################
>> Mon Oct  8 09:12:32 GMT 2018
>> ################## on node xetest1
>> r0 role:Secondary
>>   disk:UpToDate
>>   xetest2 role:Secondary
>>     peer-disk:UpToDate
>>   xetest3 role:Secondary
>>     peer-disk:Outdated
>>
>> ################## on node xetest2
>> r0 role:Secondary
>>   disk:UpToDate
>>   xetest1 role:Secondary
>>     peer-disk:UpToDate
>>   xetest3 role:Secondary
>>     replication:SyncSource peer-disk:Inconsistent
>>
>> ################## on node xetest3
>> r0 role:Secondary
>>   disk:Inconsistent
>>   xetest1 role:Secondary
>>     replication:SyncTarget peer-disk:UpToDate
>>   xetest2 role:Secondary
>>     peer-disk:UpToDate
>>
>> ################## check 9 ##################
>> Mon Oct  8 09:12:44 GMT 2018
>> ################## on node xetest1
>> r0 role:Secondary
>>   disk:UpToDate
>>   xetest2 role:Secondary
>>     peer-disk:UpToDate
>>   xetest3 role:Secondary
>>     peer-disk:Outdated
>>
>> ################## on node xetest2
>> r0 role:Secondary
>>   disk:UpToDate
>>   xetest1 role:Secondary
>>     peer-disk:UpToDate
>>   xetest3 role:Secondary
>>     replication:SyncSource peer-disk:Inconsistent
>>
>> ################## on node xetest3
>> r0 role:Secondary
>>   disk:Outdated blocked:upper
>>   xetest1 role:Secondary
>>     replication:WFBitMapT peer-disk:UpToDate
>>   xetest2 role:Secondary
>>     peer-disk:UpToDate
>>
>> ################## check 10 ##################
>> Mon Oct  8 09:12:55 GMT 2018
>> ################## on node xetest1
>> r0 role:Secondary
>>   disk:UpToDate
>>   xetest2 role:Secondary
>>     peer-disk:UpToDate
>>   xetest3 role:Secondary
>>     peer-disk:UpToDate
>>
>> ################## on node xetest2
>> r0 role:Secondary
>>   disk:UpToDate
>>   xetest1 role:Secondary
>>     peer-disk:UpToDate
>>   xetest3 role:Secondary
>>     replication:SyncSource peer-disk:Inconsistent
>>
>> ################## on node xetest3
>> r0 role:Secondary
>>   disk:Outdated blocked:upper
>>   xetest1 role:Secondary
>>     replication:WFBitMapT peer-disk:UpToDate
>>   xetest2 role:Secondary
>>     peer-disk:UpToDate
>>
>> [oracle at xetest1 ~]$
>>
>>
>>
>>
>>
>>
>> ####### our config file is:
>> [oracle at xetest1 ~]$ cat /etc/drbd.conf
>> include "drbd.d/global_common.conf";
>>
>> resource r0 {
>>  protocol C;
>>  meta-disk internal;
>>  device /dev/drbd0;
>>  options {
>>    auto-promote no;
>>    quorum majority;
>>    on-no-quorum io-error;
>> #   quorum-minimum-redundancy 2;
>>  }
>>  syncer {
>>   verify-alg sha1;
>>  }
>>  net {
>>   ping-int 5;
>>   after-sb-0pri discard-zero-changes;
>>   after-sb-1pri disconnect;
>>   after-sb-2pri disconnect;
>>  }
>>  on xetest1 {
>>   disk   /dev/sdb1;
>>   address  10.0.1.2:7790;
>>   node-id  1;
>>  }
>>  on xetest2 {
>>   disk   /dev/sdb1;
>>   address  10.0.2.2:7790;
>>   node-id  2;
>>  }
>>  on xetest3 {
>>   disk   /dev/sdb1;
>>   address  10.0.0.2:7790;
>>   node-id  3;
>>  }
>>  connection-mesh {
>>         hosts     xetest1 xetest2 xetest3;
>>  }
>> }
>> [oracle at xetest1 ~]$
>>
>>
>>
>>
>>
>>
>>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20181008/9cebe202/attachment-0002.html>


More information about the Users mailing list