[ClusterLabs] [corosync] Virtual Synchrony Property guarantees in case of network partition

[satish kumar]

Hello honza, thanks for the response !

With state sync, I simply mean that 'k-1' messages were delivered to N1, N2
and N3 and they have applied these messages to change their program state.
N1.state = apply(m(k-1);
N2.state = apply(m(k-1);
N3.state = apply(m(k-1);

The document you shared cleared many doubts. However I still need one
clarification.

According to the document:
"The configuration change messages warn the application that a membership
change has occurred, so that the application program can take appropriate
action based on the membership change. Extended virtual synchrony
guarantees a consistent order of messages delivery across a partition,
which is essential if the application program are to be able to reconcile
their states following repair of a failed processor or reemerging of the
partitioned network."

I just want to know that this property is not something related to
CPG_TYPE_SAFE, which is still not implemented.
Please consider this scenario:
0. N1, N2 and N3 has received the message m(k-1).
1. N1 mcast(CPG_TYPE_AGREED) m(k) message.
2. As it is not CPG_TYPE_SAFE, m(k) delievered to N1 but was not yet
delivered to N2 and N3.
3. Network partition separate N1 from N2 and N3. N2 and N3 can never see
m(k).
4. Configuration change message is now delivered to N1, N2 and N3.

Here, N1 will change its state to N1.state = apply(m(k), thinking all in
the current configuration has received the message.

According to your reply it looks like N1 will not receive m(k). So this is
what each node will see:
N1 will see: m(k-1) -> C1 (config change)
N2 will see: m(k-1) -> C1 (config change)
N3 will see: m(k-1) -> C1 (config change)

Message m(k) will be discarded, and will not be delivered to N1 even if it
was sent by N1 before the network partition.

This is the expected behavior with CPG_TYPE_AGREED?

Regards,
Satish


On Mon, Jun 6, 2016 at 4:15 PM, Jan Friesse <jfriesse at redhat.com> wrote:

> Hi,
>
> Hello,
>>
>> Virtual Synchrony Property - messages are delivered in agreed order and
>> configuration changes are delivered in agreed order relative to message.
>>
>> What happen to this property when network is partitioned the cluster into
>> two. Consider following scenario (which I took from one of the
>> previous query by Andrei Elkin):
>>
>> * N1, N2 and N3 are in state sync with m(k-1) messages are delivered.
>>
>
> What exactly you mean by "state sync"?
>
> * N1 sends m(k) and just now network partition N1 node from N2 and N3.
>>
>> Does CPG_TYPE_AGREED guarantee that virtual synchrony is held?
>>
>
> Yes it does (actually higher level of VS called EVS)
>
>
>> When property is held, configuration change message C1 is guaranteed to
>> delivered before m(k) to N1.
>> N1 will see: m(k-1) C1 m(k)
>> N2 and N3 will see: m(k-1) C1
>>
>> But if this property is violated:
>> N1 will see: m(k-1) m(k) C1
>> N2 and N3 will see: m(k-1) C1
>>
>> Violation will screw any user application running on the cluster.
>>
>> Could someone please explain what is the behavior of Corosync in this
>> scenario with CPG_TYPE_AGREED ordering.
>>
>
> For description how exactly totem synchronization works take a look to
> http://corosync.github.com/corosync/doc/DAAgarwal.thesis.ps.gz
>
> After totem is synchronized, there is another level of synchronization of
> services (not described in above doc). All services synchronize in very
> similar way, so you can take a look to CPG as example. Basically only state
> held by CPG is connected clients. So every node sends it's connected
> clients list to every other node. If sync is aborted (change of
> membership), it's restarted. These sync messages has priority over user
> messages (actually it's not possible to send messages during sync). User
> app can be sure that message was delivered only after it gets it's own
> message. Also app gets configuration change message so it knows, who got
> the message.
>
> Regards,
>   Honza
>
>
>> Regards,
>> Satish
>>
>>
>>
>> _______________________________________________
>> Users mailing list: Users at clusterlabs.org
>> http://clusterlabs.org/mailman/listinfo/users
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>>
>>
>
> _______________________________________________
> Users mailing list: Users at clusterlabs.org
> http://clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20160606/20737835/attachment-0002.html>