[ClusterLabs] [corosync] Virtual Synchrony Property guarantees in case of network partition

[Jan Friesse]

> But C1 is *guaranteed *to deliver *before *m(k)? No case where C1 is

Yes

> delivered after m(k)?

Nope.

>
>
> Regards,
> Satish
>
> On Mon, Jun 6, 2016 at 8:10 PM, Jan Friesse <jfriesse at redhat.com> wrote:
>
>> satish kumar napsal(a):
>>
>> Hello honza, thanks for the response !
>>>
>>> With state sync, I simply mean that 'k-1' messages were delivered to N1,
>>> N2
>>> and N3 and they have applied these messages to change their program state.
>>> N1.state = apply(m(k-1);
>>> N2.state = apply(m(k-1);
>>> N3.state = apply(m(k-1);
>>>
>>> The document you shared cleared many doubts. However I still need one
>>> clarification.
>>>
>>> According to the document:
>>> "The configuration change messages warn the application that a membership
>>> change has occurred, so that the application program can take appropriate
>>> action based on the membership change. Extended virtual synchrony
>>> guarantees a consistent order of messages delivery across a partition,
>>> which is essential if the application program are to be able to reconcile
>>> their states following repair of a failed processor or reemerging of the
>>> partitioned network."
>>>
>>> I just want to know that this property is not something related to
>>> CPG_TYPE_SAFE, which is still not implemented.
>>> Please consider this scenario:
>>> 0. N1, N2 and N3 has received the message m(k-1).
>>> 1. N1 mcast(CPG_TYPE_AGREED) m(k) message.
>>> 2. As it is not CPG_TYPE_SAFE, m(k) delievered to N1 but was not yet
>>> delivered to N2 and N3.
>>> 3. Network partition separate N1 from N2 and N3. N2 and N3 can never see
>>> m(k).
>>> 4. Configuration change message is now delivered to N1, N2 and N3.
>>>
>>> Here, N1 will change its state to N1.state = apply(m(k), thinking all in
>>> the current configuration has received the message.
>>>
>>> According to your reply it looks like N1 will not receive m(k). So this is
>>> what each node will see:
>>> N1 will see: m(k-1) -> C1 (config change)
>>> N2 will see: m(k-1) -> C1 (config change)
>>> N3 will see: m(k-1) -> C1 (config change)
>>>
>>
>> For N2 and N3, it's not same C1. So let's call it C2. Because C1 for N1 is
>> (N2 and N3 left) and C2 for N2 and N3 is (N1 left).
>>
>>
>>
>>> Message m(k) will be discarded, and will not be delivered to N1 even if it
>>> was sent by N1 before the network partition.
>>>
>>
>> No. m(k) will be delivered to app running on N1. So N1 will see m(k-1),
>> C1, m(k). So application exactly knows which node got message m(k).
>>
>> Regards,
>>    Honza
>>
>>
>>
>>> This is the expected behavior with CPG_TYPE_AGREED?
>>>
>>> Regards,
>>> Satish
>>>
>>>
>>> On Mon, Jun 6, 2016 at 4:15 PM, Jan Friesse <jfriesse at redhat.com> wrote:
>>>
>>> Hi,
>>>>
>>>> Hello,
>>>>
>>>>>
>>>>> Virtual Synchrony Property - messages are delivered in agreed order and
>>>>> configuration changes are delivered in agreed order relative to message.
>>>>>
>>>>> What happen to this property when network is partitioned the cluster
>>>>> into
>>>>> two. Consider following scenario (which I took from one of the
>>>>> previous query by Andrei Elkin):
>>>>>
>>>>> * N1, N2 and N3 are in state sync with m(k-1) messages are delivered.
>>>>>
>>>>>
>>>> What exactly you mean by "state sync"?
>>>>
>>>> * N1 sends m(k) and just now network partition N1 node from N2 and N3.
>>>>
>>>>>
>>>>> Does CPG_TYPE_AGREED guarantee that virtual synchrony is held?
>>>>>
>>>>>
>>>> Yes it does (actually higher level of VS called EVS)
>>>>
>>>>
>>>> When property is held, configuration change message C1 is guaranteed to
>>>>> delivered before m(k) to N1.
>>>>> N1 will see: m(k-1) C1 m(k)
>>>>> N2 and N3 will see: m(k-1) C1
>>>>>
>>>>> But if this property is violated:
>>>>> N1 will see: m(k-1) m(k) C1
>>>>> N2 and N3 will see: m(k-1) C1
>>>>>
>>>>> Violation will screw any user application running on the cluster.
>>>>>
>>>>> Could someone please explain what is the behavior of Corosync in this
>>>>> scenario with CPG_TYPE_AGREED ordering.
>>>>>
>>>>>
>>>> For description how exactly totem synchronization works take a look to
>>>> http://corosync.github.com/corosync/doc/DAAgarwal.thesis.ps.gz
>>>>
>>>> After totem is synchronized, there is another level of synchronization of
>>>> services (not described in above doc). All services synchronize in very
>>>> similar way, so you can take a look to CPG as example. Basically only
>>>> state
>>>> held by CPG is connected clients. So every node sends it's connected
>>>> clients list to every other node. If sync is aborted (change of
>>>> membership), it's restarted. These sync messages has priority over user
>>>> messages (actually it's not possible to send messages during sync). User
>>>> app can be sure that message was delivered only after it gets it's own
>>>> message. Also app gets configuration change message so it knows, who got
>>>> the message.
>>>>
>>>> Regards,
>>>>     Honza
>>>>
>>>>
>>>> Regards,
>>>>> Satish
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> Users mailing list: Users at clusterlabs.org
>>>>> http://clusterlabs.org/mailman/listinfo/users
>>>>>
>>>>> Project Home: http://www.clusterlabs.org
>>>>> Getting started:
>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>
>>>>>
>>>>>
>>>> _______________________________________________
>>>> Users mailing list: Users at clusterlabs.org
>>>> http://clusterlabs.org/mailman/listinfo/users
>>>>
>>>> Project Home: http://www.clusterlabs.org
>>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>> Bugs: http://bugs.clusterlabs.org
>>>>
>>>>
>>>
>>>
>>> _______________________________________________
>>> Users mailing list: Users at clusterlabs.org
>>> http://clusterlabs.org/mailman/listinfo/users
>>>
>>> Project Home: http://www.clusterlabs.org
>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>> Bugs: http://bugs.clusterlabs.org
>>>
>>>
>>
>> _______________________________________________
>> Users mailing list: Users at clusterlabs.org
>> http://clusterlabs.org/mailman/listinfo/users
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>>
>
>
>
> _______________________________________________
> Users mailing list: Users at clusterlabs.org
> http://clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>