[ClusterLabs] [corosync] Virtual Synchrony Property guarantees in case of network partition

Mon Jun 6 12:50:12 UTC 2016

Thanks, really appreciate your help.

On Mon, Jun 6, 2016 at 9:17 PM, Jan Friesse <jfriesse at redhat.com> wrote:

> But C1 is *guaranteed *to deliver *before *m(k)? No case where C1 is
>>
>
> Yes
>
> delivered after m(k)?
>>
>
> Nope.
>
>
>
>>
>> Regards,
>> Satish
>>
>> On Mon, Jun 6, 2016 at 8:10 PM, Jan Friesse <jfriesse at redhat.com> wrote:
>>
>> satish kumar napsal(a):
>>>
>>> Hello honza, thanks for the response !
>>>
>>>>
>>>> With state sync, I simply mean that 'k-1' messages were delivered to N1,
>>>> N2
>>>> and N3 and they have applied these messages to change their program
>>>> state.
>>>> N1.state = apply(m(k-1);
>>>> N2.state = apply(m(k-1);
>>>> N3.state = apply(m(k-1);
>>>>
>>>> The document you shared cleared many doubts. However I still need one
>>>> clarification.
>>>>
>>>> According to the document:
>>>> "The configuration change messages warn the application that a
>>>> membership
>>>> change has occurred, so that the application program can take
>>>> appropriate
>>>> action based on the membership change. Extended virtual synchrony
>>>> guarantees a consistent order of messages delivery across a partition,
>>>> which is essential if the application program are to be able to
>>>> reconcile
>>>> their states following repair of a failed processor or reemerging of the
>>>> partitioned network."
>>>>
>>>> I just want to know that this property is not something related to
>>>> CPG_TYPE_SAFE, which is still not implemented.
>>>> Please consider this scenario:
>>>> 0. N1, N2 and N3 has received the message m(k-1).
>>>> 1. N1 mcast(CPG_TYPE_AGREED) m(k) message.
>>>> 2. As it is not CPG_TYPE_SAFE, m(k) delievered to N1 but was not yet
>>>> delivered to N2 and N3.
>>>> 3. Network partition separate N1 from N2 and N3. N2 and N3 can never see
>>>> m(k).
>>>> 4. Configuration change message is now delivered to N1, N2 and N3.
>>>>
>>>> Here, N1 will change its state to N1.state = apply(m(k), thinking all in
>>>> the current configuration has received the message.
>>>>
>>>> According to your reply it looks like N1 will not receive m(k). So this
>>>> is
>>>> what each node will see:
>>>> N1 will see: m(k-1) -> C1 (config change)
>>>> N2 will see: m(k-1) -> C1 (config change)
>>>> N3 will see: m(k-1) -> C1 (config change)
>>>>
>>>>
>>> For N2 and N3, it's not same C1. So let's call it C2. Because C1 for N1
>>> is
>>> (N2 and N3 left) and C2 for N2 and N3 is (N1 left).
>>>
>>>
>>>
>>> Message m(k) will be discarded, and will not be delivered to N1 even if
>>>> it
>>>> was sent by N1 before the network partition.
>>>>
>>>>
>>> No. m(k) will be delivered to app running on N1. So N1 will see m(k-1),
>>> C1, m(k). So application exactly knows which node got message m(k).
>>>
>>> Regards,
>>>    Honza
>>>
>>>
>>>
>>> This is the expected behavior with CPG_TYPE_AGREED?
>>>>
>>>> Regards,
>>>> Satish
>>>>
>>>>
>>>> On Mon, Jun 6, 2016 at 4:15 PM, Jan Friesse <jfriesse at redhat.com>
>>>> wrote:
>>>>
>>>> Hi,
>>>>
>>>>>
>>>>> Hello,
>>>>>
>>>>>
>>>>>> Virtual Synchrony Property - messages are delivered in agreed order
>>>>>> and
>>>>>> configuration changes are delivered in agreed order relative to
>>>>>> message.
>>>>>>
>>>>>> What happen to this property when network is partitioned the cluster
>>>>>> into
>>>>>> two. Consider following scenario (which I took from one of the
>>>>>> previous query by Andrei Elkin):
>>>>>>
>>>>>> * N1, N2 and N3 are in state sync with m(k-1) messages are delivered.
>>>>>>
>>>>>>
>>>>>> What exactly you mean by "state sync"?
>>>>>
>>>>> * N1 sends m(k) and just now network partition N1 node from N2 and N3.
>>>>>
>>>>>
>>>>>> Does CPG_TYPE_AGREED guarantee that virtual synchrony is held?
>>>>>>
>>>>>>
>>>>>> Yes it does (actually higher level of VS called EVS)
>>>>>
>>>>>
>>>>> When property is held, configuration change message C1 is guaranteed to
>>>>>
>>>>>> delivered before m(k) to N1.
>>>>>> N1 will see: m(k-1) C1 m(k)
>>>>>> N2 and N3 will see: m(k-1) C1
>>>>>>
>>>>>> But if this property is violated:
>>>>>> N1 will see: m(k-1) m(k) C1
>>>>>> N2 and N3 will see: m(k-1) C1
>>>>>>
>>>>>> Violation will screw any user application running on the cluster.
>>>>>>
>>>>>> Could someone please explain what is the behavior of Corosync in this
>>>>>> scenario with CPG_TYPE_AGREED ordering.
>>>>>>
>>>>>>
>>>>>> For description how exactly totem synchronization works take a look to
>>>>> http://corosync.github.com/corosync/doc/DAAgarwal.thesis.ps.gz
>>>>>
>>>>> After totem is synchronized, there is another level of synchronization
>>>>> of
>>>>> services (not described in above doc). All services synchronize in very
>>>>> similar way, so you can take a look to CPG as example. Basically only
>>>>> state
>>>>> held by CPG is connected clients. So every node sends it's connected
>>>>> clients list to every other node. If sync is aborted (change of
>>>>> membership), it's restarted. These sync messages has priority over user
>>>>> messages (actually it's not possible to send messages during sync).
>>>>> User
>>>>> app can be sure that message was delivered only after it gets it's own
>>>>> message. Also app gets configuration change message so it knows, who
>>>>> got
>>>>> the message.
>>>>>
>>>>> Regards,
>>>>>     Honza
>>>>>
>>>>>
>>>>> Regards,
>>>>>
>>>>>> Satish
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>>> Users mailing list: Users at clusterlabs.org
>>>>>> http://clusterlabs.org/mailman/listinfo/users
>>>>>>
>>>>>> Project Home: http://www.clusterlabs.org
>>>>>> Getting started:
>>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>>
>>>>>>
>>>>>>
>>>>>> _______________________________________________
>>>>> Users mailing list: Users at clusterlabs.org
>>>>> http://clusterlabs.org/mailman/listinfo/users
>>>>>
>>>>> Project Home: http://www.clusterlabs.org
>>>>> Getting started:
>>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>> Bugs: http://bugs.clusterlabs.org
>>>>>
>>>>>
>>>>>
>>>>
>>>> _______________________________________________
>>>> Users mailing list: Users at clusterlabs.org
>>>> http://clusterlabs.org/mailman/listinfo/users
>>>>
>>>> Project Home: http://www.clusterlabs.org
>>>> Getting started:
>>>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>> Bugs: http://bugs.clusterlabs.org
>>>>
>>>>
>>>>
>>> _______________________________________________
>>> Users mailing list: Users at clusterlabs.org
>>> http://clusterlabs.org/mailman/listinfo/users
>>>
>>> Project Home: http://www.clusterlabs.org
>>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>> Bugs: http://bugs.clusterlabs.org
>>>
>>>
>>
>>
>> _______________________________________________
>> Users mailing list: Users at clusterlabs.org
>> http://clusterlabs.org/mailman/listinfo/users
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>>
>>
>
> _______________________________________________
> Users mailing list: Users at clusterlabs.org
> http://clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20160606/9ca641b8/attachment.htm>