<div dir="ltr"><font face="monospace, monospace">Hello Honza and others,</font><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">It seems, Corosync is not reliable in network partition. Here is the test I ran:</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">Process P1 on node N1</font></div><div><font face="monospace, monospace">Process P2 on node N2</font></div><div><font face="monospace, monospace">Process P3 on node N2 again</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">When all the processes has joined the cluster, this is what happens:</font></div><div><font face="monospace, monospace"><br></font></div><div><font face="monospace, monospace">1. P1 in one of its thread, continuously {while(1)} multi-casting messages.</font></div><div><font face="monospace, monospace">2. All the processes, i.e. P1,P2 and P3, </font></div><div><font face="monospace, monospace"> - has separate listening thread, calling {cpg_dispatch()}</font></div><div><font face="monospace, monospace"> - inside <span style="color:rgb(0,0,0)">cpg_deliver_fn_t, printf counter, which represent how many messages has received</span></font></div><div><font face="monospace, monospace"><span style="color:rgb(0,0,0)"> - inside </span><span style="color:rgb(0,0,0)">cpg_confchg_fn_t, put process to sleep as in when any process leaves the cluster</span></font></div><div><span style="color:rgb(0,0,0)"><font face="monospace, monospace"><br></font></span></div><div><font color="#000000" face="monospace, monospace">First test case: P2 was stopped forcefully.</font></div><div><font color="#000000" face="monospace, monospace"> - P1 and P3 received same number of messages before configuration change message</font></div><div><font color="#000000" face="monospace, monospace"> - PASS expected result, configuration message order is maintained</font></div><div><font color="#000000" face="monospace, monospace"><br></font></div><div><font color="#000000" face="monospace, monospace">Second test case: manually pulled the cable connecting node N1 and N2</font></div><div><font color="#000000" face="monospace, monospace"> - P1 received more number of messages in compare to P2 and P3 before configuration change message was delievered</font></div><div><font color="#000000" face="monospace, monospace"> - Configuration message was not delivered in order, FAIL</font></div><div><span style="color:rgb(0,0,0)"><font face="monospace, monospace"> </font></span></div><div><font face="monospace, monospace"><font color="#000000">In case of network partition, configuration messages are not ordered with regard to 'extended virtual synchrony' property. I believe </font><span style="color:rgb(0,0,0)">CPG_TYPE_SAFE implementation is required not only to guarantee that messages are received at all the process but also to guarantee the ordering of configuration messages in network partition.</span></font></div><div><span style="color:rgb(0,0,0)"><font face="monospace, monospace"><br></font></span></div><div><span style="color:rgb(0,0,0)"><font face="monospace, monospace">--</font></span></div><div><span style="color:rgb(0,0,0)"><font face="monospace, monospace">Satish</font></span></div><div><font color="#000000"><br></font></div></div><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jun 6, 2016 at 9:50 PM, satish kumar <span dir="ltr"><<a href="mailto:satish.kr2008@gmail.com" target="_blank">satish.kr2008@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir="ltr">Thanks, really appreciate your help.</div><div class="HOEnZb"><div class="h5"><div class="gmail_extra"><br><div class="gmail_quote">On Mon, Jun 6, 2016 at 9:17 PM, Jan Friesse <span dir="ltr"><<a href="mailto:jfriesse@redhat.com" target="_blank">jfriesse@redhat.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
But C1 is *guaranteed *to deliver *before *m(k)? No case where C1 is<br>
</blockquote>
<br>
Yes<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
delivered after m(k)?<br>
</blockquote>
<br>
Nope.<div><div><br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
<br>
Regards,<br>
Satish<br>
<br>
On Mon, Jun 6, 2016 at 8:10 PM, Jan Friesse <<a href="mailto:jfriesse@redhat.com" target="_blank">jfriesse@redhat.com</a>> wrote:<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
satish kumar napsal(a):<br>
<br>
Hello honza, thanks for the response !<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
With state sync, I simply mean that 'k-1' messages were delivered to N1,<br>
N2<br>
and N3 and they have applied these messages to change their program state.<br>
N1.state = apply(m(k-1);<br>
N2.state = apply(m(k-1);<br>
N3.state = apply(m(k-1);<br>
<br>
The document you shared cleared many doubts. However I still need one<br>
clarification.<br>
<br>
According to the document:<br>
"The configuration change messages warn the application that a membership<br>
change has occurred, so that the application program can take appropriate<br>
action based on the membership change. Extended virtual synchrony<br>
guarantees a consistent order of messages delivery across a partition,<br>
which is essential if the application program are to be able to reconcile<br>
their states following repair of a failed processor or reemerging of the<br>
partitioned network."<br>
<br>
I just want to know that this property is not something related to<br>
CPG_TYPE_SAFE, which is still not implemented.<br>
Please consider this scenario:<br>
0. N1, N2 and N3 has received the message m(k-1).<br>
1. N1 mcast(CPG_TYPE_AGREED) m(k) message.<br>
2. As it is not CPG_TYPE_SAFE, m(k) delievered to N1 but was not yet<br>
delivered to N2 and N3.<br>
3. Network partition separate N1 from N2 and N3. N2 and N3 can never see<br>
m(k).<br>
4. Configuration change message is now delivered to N1, N2 and N3.<br>
<br>
Here, N1 will change its state to N1.state = apply(m(k), thinking all in<br>
the current configuration has received the message.<br>
<br>
According to your reply it looks like N1 will not receive m(k). So this is<br>
what each node will see:<br>
N1 will see: m(k-1) -> C1 (config change)<br>
N2 will see: m(k-1) -> C1 (config change)<br>
N3 will see: m(k-1) -> C1 (config change)<br>
<br>
</blockquote>
<br>
For N2 and N3, it's not same C1. So let's call it C2. Because C1 for N1 is<br>
(N2 and N3 left) and C2 for N2 and N3 is (N1 left).<br>
<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Message m(k) will be discarded, and will not be delivered to N1 even if it<br>
was sent by N1 before the network partition.<br>
<br>
</blockquote>
<br>
No. m(k) will be delivered to app running on N1. So N1 will see m(k-1),<br>
C1, m(k). So application exactly knows which node got message m(k).<br>
<br>
Regards,<br>
Honza<br>
<br>
<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
This is the expected behavior with CPG_TYPE_AGREED?<br>
<br>
Regards,<br>
Satish<br>
<br>
<br>
On Mon, Jun 6, 2016 at 4:15 PM, Jan Friesse <<a href="mailto:jfriesse@redhat.com" target="_blank">jfriesse@redhat.com</a>> wrote:<br>
<br>
Hi,<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Hello,<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Virtual Synchrony Property - messages are delivered in agreed order and<br>
configuration changes are delivered in agreed order relative to message.<br>
<br>
What happen to this property when network is partitioned the cluster<br>
into<br>
two. Consider following scenario (which I took from one of the<br>
previous query by Andrei Elkin):<br>
<br>
* N1, N2 and N3 are in state sync with m(k-1) messages are delivered.<br>
<br>
<br>
</blockquote>
What exactly you mean by "state sync"?<br>
<br>
* N1 sends m(k) and just now network partition N1 node from N2 and N3.<br>
<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
Does CPG_TYPE_AGREED guarantee that virtual synchrony is held?<br>
<br>
<br>
</blockquote>
Yes it does (actually higher level of VS called EVS)<br>
<br>
<br>
When property is held, configuration change message C1 is guaranteed to<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
delivered before m(k) to N1.<br>
N1 will see: m(k-1) C1 m(k)<br>
N2 and N3 will see: m(k-1) C1<br>
<br>
But if this property is violated:<br>
N1 will see: m(k-1) m(k) C1<br>
N2 and N3 will see: m(k-1) C1<br>
<br>
Violation will screw any user application running on the cluster.<br>
<br>
Could someone please explain what is the behavior of Corosync in this<br>
scenario with CPG_TYPE_AGREED ordering.<br>
<br>
<br>
</blockquote>
For description how exactly totem synchronization works take a look to<br>
<a href="http://corosync.github.com/corosync/doc/DAAgarwal.thesis.ps.gz" rel="noreferrer" target="_blank">http://corosync.github.com/corosync/doc/DAAgarwal.thesis.ps.gz</a><br>
<br>
After totem is synchronized, there is another level of synchronization of<br>
services (not described in above doc). All services synchronize in very<br>
similar way, so you can take a look to CPG as example. Basically only<br>
state<br>
held by CPG is connected clients. So every node sends it's connected<br>
clients list to every other node. If sync is aborted (change of<br>
membership), it's restarted. These sync messages has priority over user<br>
messages (actually it's not possible to send messages during sync). User<br>
app can be sure that message was delivered only after it gets it's own<br>
message. Also app gets configuration change message so it knows, who got<br>
the message.<br>
<br>
Regards,<br>
Honza<br>
<br>
<br>
Regards,<br>
<blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
Satish<br>
<br>
<br>
<br>
_______________________________________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
<a href="http://clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://clusterlabs.org/mailman/listinfo/users</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started:<br>
<a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
<br>
<br>
<br>
</blockquote>
_______________________________________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
<a href="http://clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://clusterlabs.org/mailman/listinfo/users</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
<br>
<br>
</blockquote>
<br>
<br>
_______________________________________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
<a href="http://clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://clusterlabs.org/mailman/listinfo/users</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
<br>
<br>
</blockquote>
<br>
_______________________________________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
<a href="http://clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://clusterlabs.org/mailman/listinfo/users</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
<br>
</blockquote>
<br>
<br>
<br>
_______________________________________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
<a href="http://clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://clusterlabs.org/mailman/listinfo/users</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
<br>
</blockquote>
<br>
<br>
_______________________________________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org" target="_blank">Users@clusterlabs.org</a><br>
<a href="http://clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://clusterlabs.org/mailman/listinfo/users</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
</div></div></blockquote></div><br></div>
</div></div></blockquote></div><br></div>