[Pacemaker] different behavior cibadmin -Ql with cman and corosync2

Andrey Groshev greenx at yandex.ru
Wed Sep 4 05:41:08 UTC 2013



04.09.2013, 01:06, "Andrew Beekhof" <andrew at beekhof.net>:
> On 03/09/2013, at 11:46 PM, Andrey Groshev <greenx at yandex.ru> wrote:
>
>>  03.09.2013, 08:27, "Andrew Beekhof" <andrew at beekhof.net>:
>>>  On 02/09/2013, at 5:27 PM, Andrey Groshev <greenx at yandex.ru> wrote:
>>>>   30.08.2013, 07:18, "Andrew Beekhof" <andrew at beekhof.net>:
>>>>>   On 29/08/2013, at 7:31 PM, Andrey Groshev <greenx at yandex.ru> wrote:
>>>>>>    29.08.2013, 12:25, "Andrey Groshev" <greenx at yandex.ru>:
>>>>>>>    29.08.2013, 02:55, "Andrew Beekhof" <andrew at beekhof.net>:
>>>>>>>>     On 28/08/2013, at 5:38 PM, Andrey Groshev <greenx at yandex.ru> wrote:
>>>>>>>>>      28.08.2013, 04:06, "Andrew Beekhof" <andrew at beekhof.net>:
>>>>>>>>>>      On 27/08/2013, at 1:13 PM, Andrey Groshev <greenx at yandex.ru> wrote:
>>>>>>>>>>>       27.08.2013, 05:39, "Andrew Beekhof" <andrew at beekhof.net>:
>>>>>>>>>>>>       On 26/08/2013, at 3:09 PM, Andrey Groshev <greenx at yandex.ru> wrote:
>>>>>>>>>>>>>        26.08.2013, 03:34, "Andrew Beekhof" <andrew at beekhof.net>:
>>>>>>>>>>>>>>        On 23/08/2013, at 9:39 PM, Andrey Groshev <greenx at yandex.ru> wrote:
>>>>>>>>>>>>>>>         Hello,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>         Today I try remake my test cluster from cman to corosync2.
>>>>>>>>>>>>>>>         I drew attention to the following:
>>>>>>>>>>>>>>>         If I reset cluster with cman through cibadmin --erase --force
>>>>>>>>>>>>>>>         In cib is still there exist names of nodes.
>>>>>>>>>>>>>>        Yes, the cluster puts back entries for all the nodes it know about automagically.
>>>>>>>>>>>>>>>         cibadmin -Ql
>>>>>>>>>>>>>>>         .....
>>>>>>>>>>>>>>>            <nodes>
>>>>>>>>>>>>>>>              <node id="dev-cluster2-node2.unix.tensor.ru" uname="dev-cluster2-node2"/>
>>>>>>>>>>>>>>>              <node id="dev-cluster2-node4.unix.tensor.ru" uname="dev-cluster2-node4"/>
>>>>>>>>>>>>>>>              <node id="dev-cluster2-node3.unix.tensor.ru" uname="dev-cluster2-node3"/>
>>>>>>>>>>>>>>>            </nodes>
>>>>>>>>>>>>>>>         ....
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>         Even if cman and pacemaker running only one node.
>>>>>>>>>>>>>>        I'm assuming all three are configured in cluster.conf?
>>>>>>>>>>>>>        Yes, there exist list nodes.
>>>>>>>>>>>>>>>         And if I do too on cluster with corosync2
>>>>>>>>>>>>>>>         I see only names of nodes which run corosync and pacemaker.
>>>>>>>>>>>>>>        Since you're not included your config, I can only guess that your corosync.conf does not have a nodelist.
>>>>>>>>>>>>>>        If it did, you should get the same behaviour.
>>>>>>>>>>>>>        I try and expected_node and nodelist.
>>>>>>>>>>>>       And it didn't work? What version of pacemaker?
>>>>>>>>>>>       It does not work as I expected.
>>>>>>>>>>      Thats because you've used IP addresses in the node list.
>>>>>>>>>>      ie.
>>>>>>>>>>
>>>>>>>>>>      node {
>>>>>>>>>>        ring0_addr: 10.76.157.17
>>>>>>>>>>      }
>>>>>>>>>>
>>>>>>>>>>      try including the node name as well, eg.
>>>>>>>>>>
>>>>>>>>>>      node {
>>>>>>>>>>        name: dev-cluster2-node2
>>>>>>>>>>        ring0_addr: 10.76.157.17
>>>>>>>>>>      }
>>>>>>>>>      The same thing.
>>>>>>>>     I don't know what to say.  I tested it here yesterday and it worked as expected.
>>>>>>>    I found that the reason that You and I have different results - I did not have reverse DNS zone for these nodes.
>>>>>>>    I know what it should be, but (PACEMAKER + CMAN) worked without a reverse area!
>>>>>>    Hasty. Deleted all. Reinstalled. Configured. Not working again. Damn!
>>>>>   It would have surprised me... pacemaker 1.1.11 doesn't do any dns lookups - reverse or otherwise.
>>>>>   Can you set
>>>>>
>>>>>    PCMK_trace_files=corosync.c
>>>>>
>>>>>   in your environment and retest?
>>>>>
>>>>>   On RHEL6 that means putting the following in /etc/sysconfig/pacemaker
>>>>>     export PCMK_trace_files=corosync.c
>>>>>
>>>>>   It should produce additional logging[1] that will help diagnose the issue.
>>>>>
>>>>>   [1] http://blog.clusterlabs.org/blog/2013/pacemaker-logging/
>>>>   Hello, Andrew.
>>>>
>>>>   You are a little misunderstood me.
>>>  No, I understood you fine.
>>>>   I wrote that I rushed to judgment.
>>>>   After I did the reverse DNS zone, the cluster behaved correctly.
>>>>   BUT after I took apart the cluster dropped configs and restarted on the new cluster,
>>>>   cluster again don't showed all the nodes in the nodes (only node with running pacemaker).
>>>>
>>>>   A small portion of the log. Full log
>>>>   In which (I thought) there is something interesting.
>>>>
>>>>   Aug 30 12:31:11 [9986] dev-cluster2-node4        cib: (  corosync.c:423   )   trace: check_message_sanity:      Verfied message 4: (dest=<all>:cib, from=dev-cluster2-node4:cib.9986, compressed=0, size=1551, total=2143)
>>>>   Aug 30 12:31:11 [9989] dev-cluster2-node4      attrd: (  corosync.c:96    )   trace: corosync_node_name:        Checking 172793107 vs 0 from nodelist.node.0.nodeid
>>>>   Aug 30 12:31:11 [9989] dev-cluster2-node4      attrd: (      ipcc.c:378   )   debug: qb_ipcc_disconnect:        qb_ipcc_disconnect()
>>>>   Aug 30 12:31:11 [9989] dev-cluster2-node4      attrd: (ringbuffer.c:294   )   debug: qb_rb_close:       Closing ringbuffer: /dev/shm/qb-cmap-request-9616-9989-27-header
>>>>   Aug 30 12:31:11 [9989] dev-cluster2-node4      attrd: (ringbuffer.c:294   )   debug: qb_rb_close:       Closing ringbuffer: /dev/shm/qb-cmap-response-9616-9989-27-header
>>>>   Aug 30 12:31:11 [9989] dev-cluster2-node4      attrd: (ringbuffer.c:294   )   debug: qb_rb_close:       Closing ringbuffer: /dev/shm/qb-cmap-event-9616-9989-27-header
>>>>   Aug 30 12:31:11 [9989] dev-cluster2-node4      attrd: (  corosync.c:134   )  notice: corosync_node_name:        Unable to get node name for nodeid 172793107
>>>  I wonder if you need to be including the nodeid too. ie.
>>>
>>>  node {
>>>   name: dev-cluster2-node2
>>>   ring0_addr: 10.76.157.17
>>>   nodeid: 2
>>>  }
>>>
>>>  I _thought_ that was implicit.
>>>  Chrissie: is "nodelist.node.%d.nodeid" always available for corosync2 or only if explicitly defined in the config?
>>  I begin from end.
>>  "nodelist.node.%d.nodeid" available if defined in /etc/corosync/corosync.conf.
>>  If I use only "expected_votes", corosync-cmapctl don't show nodelist.
>>
>>  And now... interesting garbage turned out:
>>  I try change config as you advised.
>>
>>  ....
>>  quorum {
>>         provider: corosync_votequorum
>>  }
>>
>>  nodelist {
>>  node {
>>   ring0_addr: 10.76.157.17
>>   name: dev-cluster2-node2
>>   nodeid: 0
>
> 0 is not allowed here. it has a special meaning

Usually, this tells the parser. :)

>
>>  }
>>  node {
>>   ring0_addr: 10.76.157.18
>>   name: dev-cluster2-node3
>>   nodeid: 1
>>  }
>>  node {
>>   ring0_addr: 10.76.157.19
>>   name: dev-cluster2-node4
>>   nodeid: 2
>>  }
>>  }
>>  ......
>>  And!
>>  # service pacemaker start
>>  Starting Pacemaker Cluster Manager:                        [FAILED]
>>
>>  I see logs. (full log: http://pastebin.com/dW7g6URj)
>>  .... snip ...
>>  Sep 03 17:28:16 [22140] dev-cluster2-node4       crmd: (  corosync.c:501   )   trace: corosync_initialize_nodelist:     Initializing corosync nodelist
>>  Sep 03 17:28:16 [22111] dev-cluster2-node4 corosync debug   [MAIN  ] connection created
>>  Sep 03 17:28:16 [22111] dev-cluster2-node4 corosync debug   [CMAP  ] lib_init_fn: conn=0x7f4de13c9870
>>  Sep 03 17:28:16 [22140] dev-cluster2-node4       crmd: (  corosync.c:96    )   trace: corosync_node_name:       Checking 2 vs 0 from nodelist.node.0.nodeid
>>  Sep 03 17:28:16 [22140] dev-cluster2-node4       crmd: (  corosync.c:96    )   trace: corosync_node_name:       Checking 2 vs 1 from nodelist.node.1.nodeid
>>  Sep 03 17:28:16 [22140] dev-cluster2-node4       crmd: (  corosync.c:96    )   trace: corosync_node_name:       Checking 2 vs 2 from nodelist.node.2.nodeid
>>  Sep 03 17:28:16 [22140] dev-cluster2-node4       crmd: (  corosync.c:104   )   trace: corosync_node_name:       Searching for node name for 2 in nodelist.node.2 (null)
>>  Sep 03 17:28:16 [22140] dev-cluster2-node4       crmd: (  corosync.c:108   )   trace: corosync_node_name:       nodelist.node.2.ring0_addr = 10.76.157.19
>>  Sep 03 17:28:16 [22140] dev-cluster2-node4       crmd: (  corosync.c:120   )   trace: corosync_node_name:       nodelist.node.2.name = dev-cluster2-node4 1
>>  Sep 03 17:28:16 [22140] dev-cluster2-node4       crmd: (  corosync.c:525   )    crit: corosync_initialize_nodelist:     Nodes 2 and 0 share the same name 'dev-cluster2-node4': shutting down
>>  Sep 03 17:28:16 [22140] dev-cluster2-node4       crmd: (       xml.c:2637  )    info: crm_xml_cleanup:  Cleaning up memory from libxml2
>>  Sep 03 17:28:16 [22135] dev-cluster2-node4        cib: (      ipcs.c:736   )   debug: qb_ipcs_dispatch_connection_request:      HUP conn (22135-22140-10)
>>  Sep 03 17:28:16 [22135] dev-cluster2-node4        cib: (      ipcs.c:578   )   debug: qb_ipcs_disconnect:       qb_ipcs_disconnect(22135-22140-10) state:2
>>  Sep 03 17:28:16 [22135] dev-cluster2-node4        cib: (       ipc.c:341   )    info: crm_client_destroy:       Destroying 0 events
>>  ..... snip ....
>>
>>  Ok, try renumbering and received the following: pacemaker starеed, but nodelist:
>>  .....
>>     <nodes>
>>       <node id="172793107" uname="dev-cluster2-node4.unix.tensor.ru"/>
>>       <node id="3" uname="dev-cluster2-node2.unix.tensor.ru"/>
>>       <node id="4" uname="dev-cluster2-node3.unix.tensor.ru"/>
>>       <node id="5" uname="dev-cluster2-node4.unix.tensor.ru"/>
>>     </nodes>
>>  .....
>>
>>  Ok, I send cibadmin --erase --force.
>>  .....
>>     <nodes>
>>       <node id="4" uname="dev-cluster2-node3.unix.tensor.ru"/>
>>       <node id="5" uname="dev-cluster2-node4.unix.tensor.ru"/>
>>       <node id="3" uname="dev-cluster2-node2.unix.tensor.ru"/>
>>     </nodes>
>>  .....
>>
>>  I think something, somewhere is not initialized :)
>>>>   Aug 30 12:31:11 [9989] dev-cluster2-node4      attrd: (   cluster.c:338   )  notice: get_node_name:     Defaulting to uname -n for the local corosync node name
>>>>   Aug 30 12:31:11 [9989] dev-cluster2-node4      attrd: (     attrd.c:651   )   debug: attrd_cib_callback:        Update 4 for probe_complete=true passed
>>>>   Aug 30 12:31:11 [9615] dev-cluster2-node4 corosync debug   [QB    ] HUP conn (9616-9989-27)
>>>>   Aug 30 12:31:11 [9615] dev-cluster2-node4 corosync debug   [QB    ] qb_ipcs_disconnect(9616-9989-27) state:2
>>>>   Aug 30 12:31:11 [9615] dev-cluster2-node4 corosync debug   [QB    ] epoll_ctl(del): Bad file descriptor (9)
>>>>   Aug 30 12:31:11 [9615] dev-cluster2-node4 corosync debug   [MAIN  ] cs_ipcs_connection_closed()
>>>>   Aug 30 12:31:11 [9615] dev-cluster2-node4 corosync debug   [CMAP  ] exit_fn for conn=0x7fa96bcb31b0
>>>>   Aug 30 12:31:11 [9615] dev-cluster2-node4 corosync debug   [MAIN  ] cs_ipcs_connection_destroyed()
>>>>   Aug 30 12:31:11 [9615] dev-cluster2-node4 corosync debug   [QB    ] Free'ing ringbuffer: /dev/shm/qb-cmap-response-9616-9989-27-header
>>>>   Aug 30 12:31:11 [9615] dev-cluster2-node4 corosync debug   [QB    ] Free'ing ringbuffer: /dev/shm/qb-cmap-event-9616-9989-27-header
>>>>   Aug 30 12:31:11 [9615] dev-cluster2-node4 corosync debug   [QB    ] Free'ing ringbuffer: /dev/shm/qb-cmap-request-9616-9989-27-header
>>>>   Aug 30 12:31:11 [9989] dev-cluster2-node4      attrd: (  corosync.c:423   )   trace: check_message_sanity:      Verfied message 1: (dest=<all>:attrd, from=dev-cluster2-node4:attrd.9989, compressed=0, size=181, total=773)
>>>>   Aug 30 12:31:42 [9984] dev-cluster2-node4 pacemakerd: (  mainloop.c:270   )    info: crm_signal_dispatch:       Invoking handler for signal 10: User defined signal 1
>>>>   Aug 30 12:31:59 [9986] dev-cluster2-node4        cib: (       ipc.c:307   )    info: crm_client_new:    Connecting 0x16c98e0 for uid=0 gid=0 pid=10007 id=f2f15044-8f76-4ea7-a714-984660619ae7
>>>>   Aug 30 12:31:59 [9986] dev-cluster2-node4        cib: ( ipc_setup.c:476   )   debug: handle_new_connection:     IPC credentials authenticated (9986-10007-13)
>>>>   Aug 30 12:31:59 [9986] dev-cluster2-node4        cib: (   ipc_shm.c:294   )   debug: qb_ipcs_shm_connect:       connecting to client [10007]
>>>>   Aug 30 12:31:59 [9986] dev-cluster2-node4        cib: (ringbuffer.c:227   )   debug: qb_rb_open_2:      shm size:524288; real_size:524288; rb->word_size:131072
>>>>   Aug 30 12:31:59 [9986] dev-cluster2-node4        cib: (ringbuffer.c:227   )   debug: qb_rb_open_2:      shm size:524288; real_size:524288; rb->word_size:131072
>>>>   Aug 30 12:31:59 [9986] dev-cluster2-node4        cib: (ringbuffer.c:227   )   debug: qb_rb_open_2:      shm size:524288; real_size:524288; rb->word_size:131072
>>>>   Aug 30 12:31:59 [9986] dev-cluster2-node4        cib: (        io.c:579   )   debug: activateCibXml:    Triggering CIB write for cib_erase op
>>>>   Aug 30 12:31:59 [9991] dev-cluster2-node4       crmd: (te_callbacks:122   )   debug: te_update_diff:    Processing diff (cib_erase): 0.9.3 -> 0.11.1 (S_IDLE)
>>>>   Aug 30 12:31:59 [9991] dev-cluster2-node4       crmd: (  te_utils.c:423   )    info: abort_transition_graph:    te_update_diff:126 - Triggered transition abort (complete=1, node=, tag=diff, id=(null), magic=NA, cib=0.11.1) : Non-status change
>>>>>>>>>      # corosync-cmapctl |grep nodelist
>>>>>>>>>      nodelist.local_node_pos (u32) = 2
>>>>>>>>>      nodelist.node.0.name (str) = dev-cluster2-node2
>>>>>>>>>      nodelist.node.0.ring0_addr (str) = 10.76.157.17
>>>>>>>>>      nodelist.node.1.name (str) = dev-cluster2-node3
>>>>>>>>>      nodelist.node.1.ring0_addr (str) = 10.76.157.18
>>>>>>>>>      nodelist.node.2.name (str) = dev-cluster2-node4
>>>>>>>>>      nodelist.node.2.ring0_addr (str) = 10.76.157.19
>>>>>>>>>
>>>>>>>>>      # corosync-quorumtool -s
>>>>>>>>>      Quorum information
>>>>>>>>>      ------------------
>>>>>>>>>      Date:             Wed Aug 28 11:29:49 2013
>>>>>>>>>      Quorum provider:  corosync_votequorum
>>>>>>>>>      Nodes:            1
>>>>>>>>>      Node ID:          172793107
>>>>>>>>>      Ring ID:          52
>>>>>>>>>      Quorate:          No
>>>>>>>>>
>>>>>>>>>      Votequorum information
>>>>>>>>>      ----------------------
>>>>>>>>>      Expected votes:   3
>>>>>>>>>      Highest expected: 3
>>>>>>>>>      Total votes:      1
>>>>>>>>>      Quorum:           2 Activity blocked
>>>>>>>>>      Flags:
>>>>>>>>>
>>>>>>>>>      Membership information
>>>>>>>>>      ----------------------
>>>>>>>>>         Nodeid      Votes Name
>>>>>>>>>      172793107          1 dev-cluster2-node4 (local)
>>>>>>>>>
>>>>>>>>>      # cibadmin -Q
>>>>>>>>>      <cib epoch="25" num_updates="3" admin_epoch="0" validate-with="pacemaker-1.2" crm_feature_set="3.0.7" cib-last-written="Wed Aug 28 11:24:06 2013" update-origin="dev-cluster2-node4" update-client="crmd" have-quorum="0" dc-uuid="172793107">
>>>>>>>>>       <configuration>
>>>>>>>>>         <crm_config>
>>>>>>>>>           <cluster_property_set id="cib-bootstrap-options">
>>>>>>>>>             <nvpair id="cib-bootstrap-options-dc-version" name="dc-version" value="1.1.11-1.el6-4f672bc"/>
>>>>>>>>>             <nvpair id="cib-bootstrap-options-cluster-infrastructure" name="cluster-infrastructure" value="corosync"/>
>>>>>>>>>           </cluster_property_set>
>>>>>>>>>         </crm_config>
>>>>>>>>>         <nodes>
>>>>>>>>>           <node id="172793107" uname="dev-cluster2-node4"/>
>>>>>>>>>         </nodes>
>>>>>>>>>         <resources/>
>>>>>>>>>         <constraints/>
>>>>>>>>>       </configuration>
>>>>>>>>>       <status>
>>>>>>>>>         <node_state id="172793107" uname="dev-cluster2-node4" in_ccm="true" crmd="online" crm-debug-origin="do_state_transition" join="member" expected="member">
>>>>>>>>>           <lrm id="172793107">
>>>>>>>>>             <lrm_resources/>
>>>>>>>>>           </lrm>
>>>>>>>>>           <transient_attributes id="172793107">
>>>>>>>>>             <instance_attributes id="status-172793107">
>>>>>>>>>               <nvpair id="status-172793107-probe_complete" name="probe_complete" value="true"/>
>>>>>>>>>             </instance_attributes>
>>>>>>>>>           </transient_attributes>
>>>>>>>>>         </node_state>
>>>>>>>>>       </status>
>>>>>>>>>      </cib>
>>>>>>>>>>>       I figured out a way get around this, but it would be easier to do if the CIB has worked as a with CMAN.
>>>>>>>>>>>       I just do not start the main resource if the attribute is not defined or it is not true.
>>>>>>>>>>>       This slightly changes the logic of the cluster.
>>>>>>>>>>>       But I'm not sure what the correct behavior.
>>>>>>>>>>>
>>>>>>>>>>>       libqb 0.14.4
>>>>>>>>>>>       corosync 2.3.1
>>>>>>>>>>>       pacemaker 1.1.11
>>>>>>>>>>>
>>>>>>>>>>>       All build from source in previews week.
>>>>>>>>>>>>>        Now in corosync.conf:
>>>>>>>>>>>>>
>>>>>>>>>>>>>        totem {
>>>>>>>>>>>>>               version: 2
>>>>>>>>>>>>>               crypto_cipher: none
>>>>>>>>>>>>>               crypto_hash: none
>>>>>>>>>>>>>               interface {
>>>>>>>>>>>>>                       ringnumber: 0
>>>>>>>>>>>>>        bindnetaddr: 10.76.157.18
>>>>>>>>>>>>>        mcastaddr: 239.94.1.56
>>>>>>>>>>>>>                       mcastport: 5405
>>>>>>>>>>>>>                       ttl: 1
>>>>>>>>>>>>>               }
>>>>>>>>>>>>>        }
>>>>>>>>>>>>>        logging {
>>>>>>>>>>>>>               fileline: off
>>>>>>>>>>>>>               to_stderr: no
>>>>>>>>>>>>>               to_logfile: yes
>>>>>>>>>>>>>               logfile: /var/log/cluster/corosync.log
>>>>>>>>>>>>>               to_syslog: yes
>>>>>>>>>>>>>               debug: on
>>>>>>>>>>>>>               timestamp: on
>>>>>>>>>>>>>               logger_subsys {
>>>>>>>>>>>>>                       subsys: QUORUM
>>>>>>>>>>>>>                       debug: on
>>>>>>>>>>>>>               }
>>>>>>>>>>>>>        }
>>>>>>>>>>>>>        quorum {
>>>>>>>>>>>>>               provider: corosync_votequorum
>>>>>>>>>>>>>        }
>>>>>>>>>>>>>        nodelist {
>>>>>>>>>>>>>        node {
>>>>>>>>>>>>>        ring0_addr: 10.76.157.17
>>>>>>>>>>>>>        }
>>>>>>>>>>>>>        node {
>>>>>>>>>>>>>        ring0_addr: 10.76.157.18
>>>>>>>>>>>>>        }
>>>>>>>>>>>>>        node {
>>>>>>>>>>>>>        ring0_addr: 10.76.157.19
>>>>>>>>>>>>>        }
>>>>>>>>>>>>>        }
>>>>>>>>>>>>>
>>>>>>>>>>>>>        _______________________________________________
>>>>>>>>>>>>>        Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>>>>>>>>>>>        http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>>>>>>>>>>
>>>>>>>>>>>>>        Project Home: http://www.clusterlabs.org
>>>>>>>>>>>>>        Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>>>>>>>>        Bugs: http://bugs.clusterlabs.org
>>>>>>>>>>>>       ,
>>>>>>>>>>>>       _______________________________________________
>>>>>>>>>>>>       Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>>>>>>>>>>       http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>>>>>>>>>
>>>>>>>>>>>>       Project Home: http://www.clusterlabs.org
>>>>>>>>>>>>       Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>>>>>>>       Bugs: http://bugs.clusterlabs.org
>>>>>>>>>>>       _______________________________________________
>>>>>>>>>>>       Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>>>>>>>>>       http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>>>>>>>>
>>>>>>>>>>>       Project Home: http://www.clusterlabs.org
>>>>>>>>>>>       Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>>>>>>       Bugs: http://bugs.clusterlabs.org
>>>>>>>>>>      ,
>>>>>>>>>>      _______________________________________________
>>>>>>>>>>      Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>>>>>>>>      http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>>>>>>>
>>>>>>>>>>      Project Home: http://www.clusterlabs.org
>>>>>>>>>>      Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>>>>>      Bugs: http://bugs.clusterlabs.org
>>>>>>>>>      _______________________________________________
>>>>>>>>>      Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>>>>>>>      http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>>>>>>
>>>>>>>>>      Project Home: http://www.clusterlabs.org
>>>>>>>>>      Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>>>>      Bugs: http://bugs.clusterlabs.org
>>>>>>>>     ,
>>>>>>>>     _______________________________________________
>>>>>>>>     Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>>>>>>     http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>>>>>
>>>>>>>>     Project Home: http://www.clusterlabs.org
>>>>>>>>     Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>>>     Bugs: http://bugs.clusterlabs.org
>>>>>>>    _______________________________________________
>>>>>>>    Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>>>>>    http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>>>>
>>>>>>>    Project Home: http://www.clusterlabs.org
>>>>>>>    Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>>    Bugs: http://bugs.clusterlabs.org
>>>>>>    _______________________________________________
>>>>>>    Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>>>>    http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>>>
>>>>>>    Project Home: http://www.clusterlabs.org
>>>>>>    Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>    Bugs: http://bugs.clusterlabs.org
>>>>>   ,
>>>>>   _______________________________________________
>>>>>   Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>>>   http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>>
>>>>>   Project Home: http://www.clusterlabs.org
>>>>>   Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>   Bugs: http://bugs.clusterlabs.org
>>>>   _______________________________________________
>>>>   Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>>   http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>
>>>>   Project Home: http://www.clusterlabs.org
>>>>   Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>   Bugs: http://bugs.clusterlabs.org
>>>  ,
>>>  _______________________________________________
>>>  Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>  http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>
>>>  Project Home: http://www.clusterlabs.org
>>>  Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>  Bugs: http://bugs.clusterlabs.org
>>  _______________________________________________
>>  Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>  http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>
>>  Project Home: http://www.clusterlabs.org
>>  Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>  Bugs: http://bugs.clusterlabs.org
>
> ,
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org




More information about the Pacemaker mailing list