<html><head><style type='text/css'>p { margin: 0; }</style></head><body><div style='font-family: Times New Roman; font-size: 12pt; color: #000000'><font size="3" style="color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 12pt; ">Hi Andreas,</font><div style="color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 12pt; "><br></div><div style="color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 12pt; ">Thanks, adding "respawn hacluster /usr/lib/heartbeat/ccm" to ha.cf worked. Since quorum-node is in standby, it shows up as "OFFLINE (standby)" in crm_mon. It seems that "cl_status nodestatus quorum-node" always returns "active", even if heartbeat is stopped on the quorum node. However, "cl_status hblinkstatus quorum-node br0" can correctly detect if heartbeat is down on quorum-node so I can use that to check its connectivity.</div><div style="color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 12pt; "><br></div><div style="color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 12pt; ">I was able to successfully test resources automatically stopping once quorum was lost. I did this by shutting down node2 so that only node1 and quorum-node remained. I then stopped heartbeat on quorum-node, which resulted in node1 losing quorum and the resources stopping (as expected). After starting heartbeat on quorum-node again, node1 reestablished quorum within about 1 minute. However, it took significantly longer (around 18 minutes) for the resources on node1 to start again. Looking through the logs, I discovered that this is because of the values of the cluster-recheck-interval (displayed as "PEngine Recheck Timer" in the logs) and crmd-integration-timeout (displayed as "Integration Timer" in the logs) properties. Here's the sequence of events as I understand it:</div><div style="color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 12pt; ">1. quorum is reestablished</div><div style="color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 12pt; ">2. the cluster-recheck-interval timer pops, sees that quorum has been reestablished, and schedules crmd-integration-timeout to run</div><div style="color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 12pt; ">3. after crmd-integration-timeout's timeout period, it pops and also sees that quorum has been restablished and thus starts the resources</div><div style="color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 12pt; "><br></div><div style="color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 12pt; ">Based on this, the maximum wait time for resources to start once quorum has been reestablished is the value of cluster-recheck-interval plus the value of crmd-integration-timeout, or (15m + 3m). I have confirmed this value through several runs of this test. This seems like a very long time to me, so I adjusted both of these values down to 1m. Running the test again I was able to confirm that the resources started 2m after quorum was reestablished:</div><div><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">## quorum reestablished</font></div><div><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">12:35:31 node1 ccm: [27015]: debug: quorum plugin: majority</font></div><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">12:35:31 node1 ccm: [27015]: debug: cluster:linux-ha, member_count=2, member_quorum_votes=200</font></div><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">12:35:31 node1 ccm: [27015]: debug: total_node_count=3, total_quorum_votes=300</font></div><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">12:35:31 node1 crmd: [27020]: info: crmd_ccm_msg_callback: Quorum (re)attained after event=NEW MEMBERSHIP (id=14)</font></div><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">12:35:31 node1 crmd: [27020]: info: crm_update_quorum: Updating quorum status to true (call=366)</font></div><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">## cluster-recheck-interval pops, schedules crmd-integration-timeout to run after its timout</font></div><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">12:36:18 node1 crmd: [27020]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped (60000ms)</font></div><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">12:36:18 node1 crmd: [27020]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]</font></div><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">12:36:18 node1 crmd: [27020]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED</font></div><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">## crmd-integration-timeout runs, starts the resources</font></div><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">12:37:18 node1 crmd: [27020]: ERROR: crm_timer_popped: Integration Timer (I_INTEGRATED) just popped in state S_INTEGRATION! (60000ms)</font></div><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">12:37:18 node1 crmd: [27020]: info: crm_timer_popped: Welcomed: 1, Integrated: 1</font></div><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">12:37:18 node1 crmd: [27020]: info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_TIMER_POPPED origin=crm_timer_popped ]</font></div><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">12:37:18 node1 crmd: [27020]: WARN: do_state_transition: Progressed to state S_FINALIZE_JOIN after C_TIMER_POPPED</font></div><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">12:37:20 node1 crmd: [27020]: info: do_dc_join_final: Ensuring DC, quorum and node attributes are up-to-date</font></div><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">12:37:20 node1 crmd: [27020]: info: crm_update_quorum: Updating quorum status to true (call=379)</font></div><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">12:37:20 node1 pengine: [29916]: notice: LogActions: Leave p_drbd_r0:1#011(Stopped)</font></div><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">12:37:20 node1 pengine: [29916]: notice: LogActions: Leave p_drbd_r1:1#011(Stopped)</font></div><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">12:37:20 node1 pengine: [29916]: notice: LogActions: Leave p_drbd_r2:1#011(Stopped)</font></div><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">12:37:20 node1 pengine: [29916]: notice: LogActions: Leave p_libvirt-bin:1#011(Stopped)</font></div><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">12:37:20 node1 pengine: [29916]: notice: LogActions: Leave p_libvirt-bin:2#011(Stopped)</font></div><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">12:37:21 node1 crmd: [27020]: notice: run_graph: Transition 77 (Complete=24, Pending=0, Fired=0, Skipped=3, Incomplete=0, Source=/var/lib/pengine/pe-input-526.bz2): Stopped</font></div><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">12:37:21 node1 pengine: [29916]: notice: LogActions: Leave p_drbd_r0:1#011(Stopped)</font></div><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">12:37:21 node1 pengine: [29916]: notice: LogActions: Leave p_drbd_r1:1#011(Stopped)</font></div><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">12:37:21 node1 pengine: [29916]: notice: LogActions: Leave p_drbd_r2:1#011(Stopped)</font></div><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">12:37:21 node1 pengine: [29916]: notice: LogActions: Leave p_libvirt-bin:1#011(Stopped)</font></div><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">12:37:21 node1 pengine: [29916]: notice: LogActions: Leave p_libvirt-bin:2#011(Stopped)</font></div><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">12:37:23 node1 lrmd: [27017]: info: RA output: (p_vm:start:stdout) Domain MyVM started</font></div><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">12:38:23 node1 crmd: [27020]: info: crm_timer_popped: PEngine Recheck Timer (I_PE_CALC) just popped (60000ms)</font></div><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">12:38:23 node1 crmd: [27020]: info: do_state_transition: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED origin=crm_timer_popped ]</font></div><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">12:38:23 node1 crmd: [27020]: info: do_state_transition: Progressed to state S_POLICY_ENGINE after C_TIMER_POPPED</font></div><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">12:38:23 node1 pengine: [29916]: notice: LogActions: Leave p_drbd_r0:1#011(Stopped)</font></div><div><font face="'courier new', courier, monaco, monospace, sans-serif" size="2">12:38:23 node1 pengine: [29916]: notice: LogActions: Leave p_drbd_r1:1#011(Stopped)</font></div></div><div><br></div><div>I've attached the full logs from this time period in addition to the excerpt above.</div><div><br></div><div style="color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 12pt; ">Is there a better way to trigger the starting of resources once quorum has been reestablished? Or is modifying these two properties a good way of doing it?</div></div><div style="color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 12pt; "><br></div><div style="color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 12pt; ">Thanks,</div><div style="color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 12pt; "><br></div><div style="color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 12pt; ">Andrew</div><div style="color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 12pt; "><br></div><div style="color: rgb(0, 0, 0); font-family: 'Times New Roman'; "><hr id="zwchr" style="color: rgb(0, 0, 0); font-family: 'Times New Roman'; font-size: 12pt; "><div style="color: rgb(0, 0, 0); font-weight: normal; font-style: normal; text-decoration: none; font-family: Helvetica, Arial, sans-serif; font-size: 12pt; "><b>From: </b><span>"Andreas Kurz" <<a class="smarterwiki-linkify" href="mailto:andreas@hastexo.com" title="[GMCP] Compose a new mail to andreas@hastexo.com" onclick="window.open('https://mail.google.com/mail/u/0/?view=cm&fs=1&tf=1&to=andreas@hastexo.com','Compose new message','width=640,height=480');return false" rel="noreferrer">andreas@hastexo.com</a>></span><br><b>To: </b><span><a class="smarterwiki-linkify" href="mailto:pacemaker@oss.clusterlabs.org" title="[GMCP] Compose a new mail to pacemaker@oss.clusterlabs.org" onclick="window.open('https://mail.google.com/mail/u/0/?view=cm&fs=1&tf=1&to=pacemaker@oss.clusterlabs.org','Compose new message','width=640,height=480');return false" rel="noreferrer">pacemaker@oss.clusterlabs.org</a></span><br><b>Sent: </b>Friday, February 24, 2012 7:26:59 AM<br><b>Subject: </b>Re: [Pacemaker] Configuring 3rd Node as Quorum Node in 2 Node Cluster<br><br>Hello,<br><br>On 02/23/2012 03:59 PM, Andrew Martin wrote:<br>> I set up the 3rd node ("quorum") yesterday by only installing heartbeat,<br>> not pacemaker. Is pacemaker necessary as well? I commented out the<br>> following lines in its ha.cf since it is always going to be running in<br>> standby:<br>> autojoin none<br>> mcast eth0 239.0.0.43 694 1 0<br>> bcast eth0<br>> warntime 5<br>> deadtime 15<br>> initdead 60<br>> keepalive 2<br>> node node1<br>> node node2<br>> node quorum<br>> #crm respawn<br>> #respawn hacluster /usr/lib/heartbeat/dopd<br>> #apiauth dopd gid=haclient uid=hacluster<br>><br><br>Hmm ... IIRC I had to enable ccm in ha.cf on the third node during my<br>last heartbeat tests to enable a quorum node:<br><br>respawn hacluster ccm<br><br>> Since this quorum node only has a single ethernet interface, can it be<br>> used for both the mcast and bcast parameters? How are both the multicast<br>> and broadcast pathways used for node communication? After saving these<br>> parameters and reloading heartbeat on all nodes, the "quorum" node is<br>> listed as offline in the cluster. Is there something missing in my<br>> configuration that is preventing it from communicating with the rest of<br>> the cluster?<br><br>cl_status and hbclient should give you some membership information, and<br>there should be some log entries on the nodes running Pacemaker. I don't<br>think the node will show up as ONLINE in crm_mon if no Pacemaker is<br>running there.<br><br>You only need the communication settings you also used for node1/2 on<br>the shared network ... so only the mcast directive is needed/possible on<br>node3.<br><br>> <br>> Also, another more general question about the failover - node1 and node2<br>> are each connected to the shared network over br0 and connected directly<br>> to each other with a crossover cable over br1:<br>> -------------------- ----------<br>> | Shared Network |------| quorum |<br>> -------------------- ----------<br>> |<br>> br0 / \ br0<br>> / \<br>> --------- ---------<br>> | node1 | --------- | node2 | <br>> --------- br1 ---------<br>> <br>> The corresponding configuration in ha.cf is<br>> autojoin none<br>> mcast br0 239.0.0.43 694 1 0<br>> bcast br1<br>> warntime 5<br>> ....<br>> <br>> If br0 to one of the nodes were to be cut when the "quorum" node was<br>> down, would they still be able to communicate over br1 (e.g. to maintain<br>> quorum between themselves and fail over to the other node that still has<br>> an active br0)? <br><br>as long as node1/2 can communicate the cluster has quorum, to fail over<br>resources to the node with best connectivity: configure ping resource<br>agent and constraints, there is a chapter in "Pacemake Explained" on<br><span>clusterlabs.org ... <a class="smarterwiki-linkify" href="http://goo.gl/x7dwK">http://goo.gl/x7dwK</a></span><br><br>Regards,<br>Andreas<br><br>-- <br>Need help with Pacemaker?<br><span><a class="smarterwiki-linkify" href="http://www.hastexo.com/now">http://www.hastexo.com/now</a></span><br><br>> <br>> Thanks,<br>> <br>> Andrew<br>> <br>> ------------------------------------------------------------------------<br><span>> *From: *"Andreas Kurz" <<a class="smarterwiki-linkify" href="mailto:andreas@hastexo.com" title="[GMCP] Compose a new mail to andreas@hastexo.com" onclick="window.open('https://mail.google.com/mail/u/0/?view=cm&fs=1&tf=1&to=andreas@hastexo.com','Compose new message','width=640,height=480');return false" rel="noreferrer">andreas@hastexo.com</a>></span><br><span>> *To: <a class="smarterwiki-linkify" href="mailto:*pacemaker@oss.clusterlabs.org" title="[GMCP] Compose a new mail to *pacemaker@oss.clusterlabs.org" onclick="window.open('https://mail.google.com/mail/u/0/?view=cm&fs=1&tf=1&to=*pacemaker@oss.clusterlabs.org','Compose new message','width=640,height=480');return false" rel="noreferrer">*pacemaker@oss.clusterlabs.org</a></span><br>> *Sent: *Monday, January 23, 2012 1:53:27 PM<br>> *Subject: *Re: [Pacemaker] Configuring 3rd Node as Quorum Node in<br>> 2 Node Cluster<br>> <br>> On 01/23/2012 03:36 PM, Andrew Martin wrote:<br>>> I think I will configure the 3rd (quorum) node in standby mode. In the<br>>> near future I am looking into setting up 2 additional clusters (each of<br>>> these are also 2-node clusters) and would like to use this same server<br>>> as the quorum node for those clusters as well. Is this possible? If so,<br>>> how do I have to configure heartbeat (or multiple instances of<br>>> heartbeat) to join multiple clusters at once and act as the quorum node<br>>> in each?<br>> <br>> No, multiple heartbeat instances per node are not supported ... but why<br>> not creating minimal VM instances ... though not to minimal, as you have<br>> a good chance that theses standby instances become DC role.<br>> <br>> Regards,<br>> Andreas<br>> <br>> -- <br>> Need help with Pacemaker?<br><span>> <a class="smarterwiki-linkify" href="http://www.hastexo.com/now">http://www.hastexo.com/now</a></span><br>> <br>>><br>>> Thanks,<br>>><br>>> Andrew<br>>><br>>> ------------------------------------------------------------------------<br><span>>> *From: *"Andreas Kurz" <<a class="smarterwiki-linkify" href="mailto:andreas@hastexo.com" title="[GMCP] Compose a new mail to andreas@hastexo.com" onclick="window.open('https://mail.google.com/mail/u/0/?view=cm&fs=1&tf=1&to=andreas@hastexo.com','Compose new message','width=640,height=480');return false" rel="noreferrer">andreas@hastexo.com</a>></span><br><span>>> *To: <a class="smarterwiki-linkify" href="mailto:*pacemaker@oss.clusterlabs.org" title="[GMCP] Compose a new mail to *pacemaker@oss.clusterlabs.org" onclick="window.open('https://mail.google.com/mail/u/0/?view=cm&fs=1&tf=1&to=*pacemaker@oss.clusterlabs.org','Compose new message','width=640,height=480');return false" rel="noreferrer">*pacemaker@oss.clusterlabs.org</a></span><br>>> *Sent: *Friday, January 13, 2012 6:35:48 AM<br>>> *Subject: *Re: [Pacemaker] Configuring 3rd Node as Quorum Node in 2<br>>> Node Cluster<br>>><br>>> On 01/13/2012 12:32 PM, Ivan Savčić | Epix wrote:<br>>>> On 1/11/2012 8:28 AM, Florian Haas wrote:<br>>>>> Another option would be to permanently run the 3rd node in standby mode.<br>>>><br>>>> Just wondering, wouldn't the standby mode prevent that node from<br>>>> performing the fencing actions? Also, can it act as DC then?<br>>><br>>> It would run no resources (including stonith resources) but can be the DC.<br>>><br>>> Another option for running a "pure" quorum node would be to only start<br>>> CCM but not pacemaker ... though that setup looks quite strange e.g. in<br>>> crm_mon output ....<br>>><br>>> Regards,<br>>> Andreas<br>>><br>>> --<br>>> Need help with Pacemaker?<br><span>>> <a class="smarterwiki-linkify" href="http://www.hastexo.com/now">http://www.hastexo.com/now</a></span><br>>><br>>>><br>>>> Thanks,<br>>>> Ivan<br>>>><br>>>> _______________________________________________<br><span>>>> Pacemaker mailing list: <a class="smarterwiki-linkify" href="mailto:Pacemaker@oss.clusterlabs.org" title="[GMCP] Compose a new mail to Pacemaker@oss.clusterlabs.org" onclick="window.open('https://mail.google.com/mail/u/0/?view=cm&fs=1&tf=1&to=Pacemaker@oss.clusterlabs.org','Compose new message','width=640,height=480');return false" rel="noreferrer">Pacemaker@oss.clusterlabs.org</a></span><br><span>>>> <a class="smarterwiki-linkify" href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a></span><br>>>><br><span>>>> Project Home: <a class="smarterwiki-linkify" href="http://www.clusterlabs.org">http://www.clusterlabs.org</a></span><br><span>>>> Getting started: <a class="smarterwiki-linkify" href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a></span><br><span>>>> Bugs: <a class="smarterwiki-linkify" href="http://bugs.clusterlabs.org">http://bugs.clusterlabs.org</a></span><br>>><br>>><br>>><br>>><br>>> _______________________________________________<br><span>>> Pacemaker mailing list: <a class="smarterwiki-linkify" href="mailto:Pacemaker@oss.clusterlabs.org" title="[GMCP] Compose a new mail to Pacemaker@oss.clusterlabs.org" onclick="window.open('https://mail.google.com/mail/u/0/?view=cm&fs=1&tf=1&to=Pacemaker@oss.clusterlabs.org','Compose new message','width=640,height=480');return false" rel="noreferrer">Pacemaker@oss.clusterlabs.org</a></span><br><span>>> <a class="smarterwiki-linkify" href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a></span><br>>><br><span>>> Project Home: <a class="smarterwiki-linkify" href="http://www.clusterlabs.org">http://www.clusterlabs.org</a></span><br><span>>> Getting started: <a class="smarterwiki-linkify" href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a></span><br><span>>> Bugs: <a class="smarterwiki-linkify" href="http://bugs.clusterlabs.org">http://bugs.clusterlabs.org</a></span><br>>><br>>><br>>><br>>> _______________________________________________<br><span>>> Pacemaker mailing list: <a class="smarterwiki-linkify" href="mailto:Pacemaker@oss.clusterlabs.org" title="[GMCP] Compose a new mail to Pacemaker@oss.clusterlabs.org" onclick="window.open('https://mail.google.com/mail/u/0/?view=cm&fs=1&tf=1&to=Pacemaker@oss.clusterlabs.org','Compose new message','width=640,height=480');return false" rel="noreferrer">Pacemaker@oss.clusterlabs.org</a></span><br><span>>> <a class="smarterwiki-linkify" href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a></span><br>>><br><span>>> Project Home: <a class="smarterwiki-linkify" href="http://www.clusterlabs.org">http://www.clusterlabs.org</a></span><br><span>>> Getting started: <a class="smarterwiki-linkify" href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a></span><br><span>>> Bugs: <a class="smarterwiki-linkify" href="http://bugs.clusterlabs.org">http://bugs.clusterlabs.org</a></span><br>> <br>> <br>> <br>> _______________________________________________<br><span>> Pacemaker mailing list: <a class="smarterwiki-linkify" href="mailto:Pacemaker@oss.clusterlabs.org" title="[GMCP] Compose a new mail to Pacemaker@oss.clusterlabs.org" onclick="window.open('https://mail.google.com/mail/u/0/?view=cm&fs=1&tf=1&to=Pacemaker@oss.clusterlabs.org','Compose new message','width=640,height=480');return false" rel="noreferrer">Pacemaker@oss.clusterlabs.org</a></span><br><span>> <a class="smarterwiki-linkify" href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a></span><br>> <br><span>> Project Home: <a class="smarterwiki-linkify" href="http://www.clusterlabs.org">http://www.clusterlabs.org</a></span><br><span>> Getting started: <a class="smarterwiki-linkify" href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a></span><br><span>> Bugs: <a class="smarterwiki-linkify" href="http://bugs.clusterlabs.org">http://bugs.clusterlabs.org</a></span><br>> <br>> <br>> <br>> _______________________________________________<br><span>> Pacemaker mailing list: <a class="smarterwiki-linkify" href="mailto:Pacemaker@oss.clusterlabs.org" title="[GMCP] Compose a new mail to Pacemaker@oss.clusterlabs.org" onclick="window.open('https://mail.google.com/mail/u/0/?view=cm&fs=1&tf=1&to=Pacemaker@oss.clusterlabs.org','Compose new message','width=640,height=480');return false" rel="noreferrer">Pacemaker@oss.clusterlabs.org</a></span><br><span>> <a class="smarterwiki-linkify" href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a></span><br>> <br><span>> Project Home: <a class="smarterwiki-linkify" href="http://www.clusterlabs.org">http://www.clusterlabs.org</a></span><br><span>> Getting started: <a class="smarterwiki-linkify" href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a></span><br><span>> Bugs: <a class="smarterwiki-linkify" href="http://bugs.clusterlabs.org">http://bugs.clusterlabs.org</a></span><br><br><br><br>_______________________________________________<br><span>Pacemaker mailing list: <a class="smarterwiki-linkify" href="mailto:Pacemaker@oss.clusterlabs.org" title="[GMCP] Compose a new mail to Pacemaker@oss.clusterlabs.org" onclick="window.open('https://mail.google.com/mail/u/0/?view=cm&fs=1&tf=1&to=Pacemaker@oss.clusterlabs.org','Compose new message','width=640,height=480');return false" rel="noreferrer">Pacemaker@oss.clusterlabs.org</a></span><br><span><a class="smarterwiki-linkify" href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a></span><br><br><span>Project Home: <a class="smarterwiki-linkify" href="http://www.clusterlabs.org">http://www.clusterlabs.org</a></span><br><span>Getting started: <a class="smarterwiki-linkify" href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a></span><br><span>Bugs: <a class="smarterwiki-linkify" href="http://bugs.clusterlabs.org">http://bugs.clusterlabs.org</a></span><br></div><br></div></div></body></html>