[Pacemaker] different behavior cibadmin -Ql with cman and corosync2

Andrey Groshev greenx at yandex.ru
Tue Sep 3 13:46:31 UTC 2013



03.09.2013, 08:27, "Andrew Beekhof" <andrew at beekhof.net>:
> On 02/09/2013, at 5:27 PM, Andrey Groshev <greenx at yandex.ru> wrote:
>
>>  30.08.2013, 07:18, "Andrew Beekhof" <andrew at beekhof.net>:
>>>  On 29/08/2013, at 7:31 PM, Andrey Groshev <greenx at yandex.ru> wrote:
>>>>   29.08.2013, 12:25, "Andrey Groshev" <greenx at yandex.ru>:
>>>>>   29.08.2013, 02:55, "Andrew Beekhof" <andrew at beekhof.net>:
>>>>>>    On 28/08/2013, at 5:38 PM, Andrey Groshev <greenx at yandex.ru> wrote:
>>>>>>>     28.08.2013, 04:06, "Andrew Beekhof" <andrew at beekhof.net>:
>>>>>>>>     On 27/08/2013, at 1:13 PM, Andrey Groshev <greenx at yandex.ru> wrote:
>>>>>>>>>      27.08.2013, 05:39, "Andrew Beekhof" <andrew at beekhof.net>:
>>>>>>>>>>      On 26/08/2013, at 3:09 PM, Andrey Groshev <greenx at yandex.ru> wrote:
>>>>>>>>>>>       26.08.2013, 03:34, "Andrew Beekhof" <andrew at beekhof.net>:
>>>>>>>>>>>>       On 23/08/2013, at 9:39 PM, Andrey Groshev <greenx at yandex.ru> wrote:
>>>>>>>>>>>>>        Hello,
>>>>>>>>>>>>>
>>>>>>>>>>>>>        Today I try remake my test cluster from cman to corosync2.
>>>>>>>>>>>>>        I drew attention to the following:
>>>>>>>>>>>>>        If I reset cluster with cman through cibadmin --erase --force
>>>>>>>>>>>>>        In cib is still there exist names of nodes.
>>>>>>>>>>>>       Yes, the cluster puts back entries for all the nodes it know about automagically.
>>>>>>>>>>>>>        cibadmin -Ql
>>>>>>>>>>>>>        .....
>>>>>>>>>>>>>           <nodes>
>>>>>>>>>>>>>             <node id="dev-cluster2-node2.unix.tensor.ru" uname="dev-cluster2-node2"/>
>>>>>>>>>>>>>             <node id="dev-cluster2-node4.unix.tensor.ru" uname="dev-cluster2-node4"/>
>>>>>>>>>>>>>             <node id="dev-cluster2-node3.unix.tensor.ru" uname="dev-cluster2-node3"/>
>>>>>>>>>>>>>           </nodes>
>>>>>>>>>>>>>        ....
>>>>>>>>>>>>>
>>>>>>>>>>>>>        Even if cman and pacemaker running only one node.
>>>>>>>>>>>>       I'm assuming all three are configured in cluster.conf?
>>>>>>>>>>>       Yes, there exist list nodes.
>>>>>>>>>>>>>        And if I do too on cluster with corosync2
>>>>>>>>>>>>>        I see only names of nodes which run corosync and pacemaker.
>>>>>>>>>>>>       Since you're not included your config, I can only guess that your corosync.conf does not have a nodelist.
>>>>>>>>>>>>       If it did, you should get the same behaviour.
>>>>>>>>>>>       I try and expected_node and nodelist.
>>>>>>>>>>      And it didn't work? What version of pacemaker?
>>>>>>>>>      It does not work as I expected.
>>>>>>>>     Thats because you've used IP addresses in the node list.
>>>>>>>>     ie.
>>>>>>>>
>>>>>>>>     node {
>>>>>>>>       ring0_addr: 10.76.157.17
>>>>>>>>     }
>>>>>>>>
>>>>>>>>     try including the node name as well, eg.
>>>>>>>>
>>>>>>>>     node {
>>>>>>>>       name: dev-cluster2-node2
>>>>>>>>       ring0_addr: 10.76.157.17
>>>>>>>>     }
>>>>>>>     The same thing.
>>>>>>    I don't know what to say.  I tested it here yesterday and it worked as expected.
>>>>>   I found that the reason that You and I have different results - I did not have reverse DNS zone for these nodes.
>>>>>   I know what it should be, but (PACEMAKER + CMAN) worked without a reverse area!
>>>>   Hasty. Deleted all. Reinstalled. Configured. Not working again. Damn!
>>>  It would have surprised me... pacemaker 1.1.11 doesn't do any dns lookups - reverse or otherwise.
>>>  Can you set
>>>
>>>   PCMK_trace_files=corosync.c
>>>
>>>  in your environment and retest?
>>>
>>>  On RHEL6 that means putting the following in /etc/sysconfig/pacemaker
>>>    export PCMK_trace_files=corosync.c
>>>
>>>  It should produce additional logging[1] that will help diagnose the issue.
>>>
>>>  [1] http://blog.clusterlabs.org/blog/2013/pacemaker-logging/
>>  Hello, Andrew.
>>
>>  You are a little misunderstood me.
>
> No, I understood you fine.
>
>>  I wrote that I rushed to judgment.
>>  After I did the reverse DNS zone, the cluster behaved correctly.
>>  BUT after I took apart the cluster dropped configs and restarted on the new cluster,
>>  cluster again don't showed all the nodes in the nodes (only node with running pacemaker).
>>
>>  A small portion of the log. Full log
>>  In which (I thought) there is something interesting.
>>
>>  Aug 30 12:31:11 [9986] dev-cluster2-node4        cib: (  corosync.c:423   )   trace: check_message_sanity:      Verfied message 4: (dest=<all>:cib, from=dev-cluster2-node4:cib.9986, compressed=0, size=1551, total=2143)
>>  Aug 30 12:31:11 [9989] dev-cluster2-node4      attrd: (  corosync.c:96    )   trace: corosync_node_name:        Checking 172793107 vs 0 from nodelist.node.0.nodeid
>>  Aug 30 12:31:11 [9989] dev-cluster2-node4      attrd: (      ipcc.c:378   )   debug: qb_ipcc_disconnect:        qb_ipcc_disconnect()
>>  Aug 30 12:31:11 [9989] dev-cluster2-node4      attrd: (ringbuffer.c:294   )   debug: qb_rb_close:       Closing ringbuffer: /dev/shm/qb-cmap-request-9616-9989-27-header
>>  Aug 30 12:31:11 [9989] dev-cluster2-node4      attrd: (ringbuffer.c:294   )   debug: qb_rb_close:       Closing ringbuffer: /dev/shm/qb-cmap-response-9616-9989-27-header
>>  Aug 30 12:31:11 [9989] dev-cluster2-node4      attrd: (ringbuffer.c:294   )   debug: qb_rb_close:       Closing ringbuffer: /dev/shm/qb-cmap-event-9616-9989-27-header
>>  Aug 30 12:31:11 [9989] dev-cluster2-node4      attrd: (  corosync.c:134   )  notice: corosync_node_name:        Unable to get node name for nodeid 172793107
>
> I wonder if you need to be including the nodeid too. ie.
>
> node {
>  name: dev-cluster2-node2
>  ring0_addr: 10.76.157.17
>  nodeid: 2
> }
>
> I _thought_ that was implicit.
> Chrissie: is "nodelist.node.%d.nodeid" always available for corosync2 or only if explicitly defined in the config?
>

I begin from end.
"nodelist.node.%d.nodeid" available if defined in /etc/corosync/corosync.conf.
If I use only "expected_votes", corosync-cmapctl don't show nodelist.

And now... interesting garbage turned out:
I try change config as you advised.

....
quorum {
        provider: corosync_votequorum
}

nodelist {
 node {
  ring0_addr: 10.76.157.17
  name: dev-cluster2-node2
  nodeid: 0
 }
 node {
  ring0_addr: 10.76.157.18
  name: dev-cluster2-node3
  nodeid: 1
 }
 node {
  ring0_addr: 10.76.157.19
  name: dev-cluster2-node4
  nodeid: 2
 }
}
......
And!
# service pacemaker start
Starting Pacemaker Cluster Manager:                        [FAILED]

I see logs. (full log: http://pastebin.com/dW7g6URj)
.... snip ...
Sep 03 17:28:16 [22140] dev-cluster2-node4       crmd: (  corosync.c:501   )   trace: corosync_initialize_nodelist:     Initializing corosync nodelist
Sep 03 17:28:16 [22111] dev-cluster2-node4 corosync debug   [MAIN  ] connection created
Sep 03 17:28:16 [22111] dev-cluster2-node4 corosync debug   [CMAP  ] lib_init_fn: conn=0x7f4de13c9870
Sep 03 17:28:16 [22140] dev-cluster2-node4       crmd: (  corosync.c:96    )   trace: corosync_node_name:       Checking 2 vs 0 from nodelist.node.0.nodeid
Sep 03 17:28:16 [22140] dev-cluster2-node4       crmd: (  corosync.c:96    )   trace: corosync_node_name:       Checking 2 vs 1 from nodelist.node.1.nodeid
Sep 03 17:28:16 [22140] dev-cluster2-node4       crmd: (  corosync.c:96    )   trace: corosync_node_name:       Checking 2 vs 2 from nodelist.node.2.nodeid
Sep 03 17:28:16 [22140] dev-cluster2-node4       crmd: (  corosync.c:104   )   trace: corosync_node_name:       Searching for node name for 2 in nodelist.node.2 (null)
Sep 03 17:28:16 [22140] dev-cluster2-node4       crmd: (  corosync.c:108   )   trace: corosync_node_name:       nodelist.node.2.ring0_addr = 10.76.157.19
Sep 03 17:28:16 [22140] dev-cluster2-node4       crmd: (  corosync.c:120   )   trace: corosync_node_name:       nodelist.node.2.name = dev-cluster2-node4 1
Sep 03 17:28:16 [22140] dev-cluster2-node4       crmd: (  corosync.c:525   )    crit: corosync_initialize_nodelist:     Nodes 2 and 0 share the same name 'dev-cluster2-node4': shutting down
Sep 03 17:28:16 [22140] dev-cluster2-node4       crmd: (       xml.c:2637  )    info: crm_xml_cleanup:  Cleaning up memory from libxml2
Sep 03 17:28:16 [22135] dev-cluster2-node4        cib: (      ipcs.c:736   )   debug: qb_ipcs_dispatch_connection_request:      HUP conn (22135-22140-10)
Sep 03 17:28:16 [22135] dev-cluster2-node4        cib: (      ipcs.c:578   )   debug: qb_ipcs_disconnect:       qb_ipcs_disconnect(22135-22140-10) state:2
Sep 03 17:28:16 [22135] dev-cluster2-node4        cib: (       ipc.c:341   )    info: crm_client_destroy:       Destroying 0 events
..... snip ....


Ok, try renumbering and received the following: pacemaker starеed, but nodelist:
.....
    <nodes>
      <node id="172793107" uname="dev-cluster2-node4.unix.tensor.ru"/>
      <node id="3" uname="dev-cluster2-node2.unix.tensor.ru"/>
      <node id="4" uname="dev-cluster2-node3.unix.tensor.ru"/>
      <node id="5" uname="dev-cluster2-node4.unix.tensor.ru"/>
    </nodes>
.....


Ok, I send cibadmin --erase --force. 
.....
    <nodes>
      <node id="4" uname="dev-cluster2-node3.unix.tensor.ru"/>
      <node id="5" uname="dev-cluster2-node4.unix.tensor.ru"/>
      <node id="3" uname="dev-cluster2-node2.unix.tensor.ru"/>
    </nodes>
.....

I think something, somewhere is not initialized :)


>>  Aug 30 12:31:11 [9989] dev-cluster2-node4      attrd: (   cluster.c:338   )  notice: get_node_name:     Defaulting to uname -n for the local corosync node name
>>  Aug 30 12:31:11 [9989] dev-cluster2-node4      attrd: (     attrd.c:651   )   debug: attrd_cib_callback:        Update 4 for probe_complete=true passed
>>  Aug 30 12:31:11 [9615] dev-cluster2-node4 corosync debug   [QB    ] HUP conn (9616-9989-27)
>>  Aug 30 12:31:11 [9615] dev-cluster2-node4 corosync debug   [QB    ] qb_ipcs_disconnect(9616-9989-27) state:2
>>  Aug 30 12:31:11 [9615] dev-cluster2-node4 corosync debug   [QB    ] epoll_ctl(del): Bad file descriptor (9)
>>  Aug 30 12:31:11 [9615] dev-cluster2-node4 corosync debug   [MAIN  ] cs_ipcs_connection_closed()
>>  Aug 30 12:31:11 [9615] dev-cluster2-node4 corosync debug   [CMAP  ] exit_fn for conn=0x7fa96bcb31b0
>>  Aug 30 12:31:11 [9615] dev-cluster2-node4 corosync debug   [MAIN  ] cs_ipcs_connection_destroyed()
>>  Aug 30 12:31:11 [9615] dev-cluster2-node4 corosync debug   [QB    ] Free'ing ringbuffer: /dev/shm/qb-cmap-response-9616-9989-27-header
>>  Aug 30 12:31:11 [9615] dev-cluster2-node4 corosync debug   [QB    ] Free'ing ringbuffer: /dev/shm/qb-cmap-event-9616-9989-27-header
>>  Aug 30 12:31:11 [9615] dev-cluster2-node4 corosync debug   [QB    ] Free'ing ringbuffer: /dev/shm/qb-cmap-request-9616-9989-27-header
>>  Aug 30 12:31:11 [9989] dev-cluster2-node4      attrd: (  corosync.c:423   )   trace: check_message_sanity:      Verfied message 1: (dest=<all>:attrd, from=dev-cluster2-node4:attrd.9989, compressed=0, size=181, total=773)
>>  Aug 30 12:31:42 [9984] dev-cluster2-node4 pacemakerd: (  mainloop.c:270   )    info: crm_signal_dispatch:       Invoking handler for signal 10: User defined signal 1
>>  Aug 30 12:31:59 [9986] dev-cluster2-node4        cib: (       ipc.c:307   )    info: crm_client_new:    Connecting 0x16c98e0 for uid=0 gid=0 pid=10007 id=f2f15044-8f76-4ea7-a714-984660619ae7
>>  Aug 30 12:31:59 [9986] dev-cluster2-node4        cib: ( ipc_setup.c:476   )   debug: handle_new_connection:     IPC credentials authenticated (9986-10007-13)
>>  Aug 30 12:31:59 [9986] dev-cluster2-node4        cib: (   ipc_shm.c:294   )   debug: qb_ipcs_shm_connect:       connecting to client [10007]
>>  Aug 30 12:31:59 [9986] dev-cluster2-node4        cib: (ringbuffer.c:227   )   debug: qb_rb_open_2:      shm size:524288; real_size:524288; rb->word_size:131072
>>  Aug 30 12:31:59 [9986] dev-cluster2-node4        cib: (ringbuffer.c:227   )   debug: qb_rb_open_2:      shm size:524288; real_size:524288; rb->word_size:131072
>>  Aug 30 12:31:59 [9986] dev-cluster2-node4        cib: (ringbuffer.c:227   )   debug: qb_rb_open_2:      shm size:524288; real_size:524288; rb->word_size:131072
>>  Aug 30 12:31:59 [9986] dev-cluster2-node4        cib: (        io.c:579   )   debug: activateCibXml:    Triggering CIB write for cib_erase op
>>  Aug 30 12:31:59 [9991] dev-cluster2-node4       crmd: (te_callbacks:122   )   debug: te_update_diff:    Processing diff (cib_erase): 0.9.3 -> 0.11.1 (S_IDLE)
>>  Aug 30 12:31:59 [9991] dev-cluster2-node4       crmd: (  te_utils.c:423   )    info: abort_transition_graph:    te_update_diff:126 - Triggered transition abort (complete=1, node=, tag=diff, id=(null), magic=NA, cib=0.11.1) : Non-status change
>>>>>>>     # corosync-cmapctl |grep nodelist
>>>>>>>     nodelist.local_node_pos (u32) = 2
>>>>>>>     nodelist.node.0.name (str) = dev-cluster2-node2
>>>>>>>     nodelist.node.0.ring0_addr (str) = 10.76.157.17
>>>>>>>     nodelist.node.1.name (str) = dev-cluster2-node3
>>>>>>>     nodelist.node.1.ring0_addr (str) = 10.76.157.18
>>>>>>>     nodelist.node.2.name (str) = dev-cluster2-node4
>>>>>>>     nodelist.node.2.ring0_addr (str) = 10.76.157.19
>>>>>>>
>>>>>>>     # corosync-quorumtool -s
>>>>>>>     Quorum information
>>>>>>>     ------------------
>>>>>>>     Date:             Wed Aug 28 11:29:49 2013
>>>>>>>     Quorum provider:  corosync_votequorum
>>>>>>>     Nodes:            1
>>>>>>>     Node ID:          172793107
>>>>>>>     Ring ID:          52
>>>>>>>     Quorate:          No
>>>>>>>
>>>>>>>     Votequorum information
>>>>>>>     ----------------------
>>>>>>>     Expected votes:   3
>>>>>>>     Highest expected: 3
>>>>>>>     Total votes:      1
>>>>>>>     Quorum:           2 Activity blocked
>>>>>>>     Flags:
>>>>>>>
>>>>>>>     Membership information
>>>>>>>     ----------------------
>>>>>>>        Nodeid      Votes Name
>>>>>>>     172793107          1 dev-cluster2-node4 (local)
>>>>>>>
>>>>>>>     # cibadmin -Q
>>>>>>>     <cib epoch="25" num_updates="3" admin_epoch="0" validate-with="pacemaker-1.2" crm_feature_set="3.0.7" cib-last-written="Wed Aug 28 11:24:06 2013" update-origin="dev-cluster2-node4" update-client="crmd" have-quorum="0" dc-uuid="172793107">
>>>>>>>      <configuration>
>>>>>>>        <crm_config>
>>>>>>>          <cluster_property_set id="cib-bootstrap-options">
>>>>>>>            <nvpair id="cib-bootstrap-options-dc-version" name="dc-version" value="1.1.11-1.el6-4f672bc"/>
>>>>>>>            <nvpair id="cib-bootstrap-options-cluster-infrastructure" name="cluster-infrastructure" value="corosync"/>
>>>>>>>          </cluster_property_set>
>>>>>>>        </crm_config>
>>>>>>>        <nodes>
>>>>>>>          <node id="172793107" uname="dev-cluster2-node4"/>
>>>>>>>        </nodes>
>>>>>>>        <resources/>
>>>>>>>        <constraints/>
>>>>>>>      </configuration>
>>>>>>>      <status>
>>>>>>>        <node_state id="172793107" uname="dev-cluster2-node4" in_ccm="true" crmd="online" crm-debug-origin="do_state_transition" join="member" expected="member">
>>>>>>>          <lrm id="172793107">
>>>>>>>            <lrm_resources/>
>>>>>>>          </lrm>
>>>>>>>          <transient_attributes id="172793107">
>>>>>>>            <instance_attributes id="status-172793107">
>>>>>>>              <nvpair id="status-172793107-probe_complete" name="probe_complete" value="true"/>
>>>>>>>            </instance_attributes>
>>>>>>>          </transient_attributes>
>>>>>>>        </node_state>
>>>>>>>      </status>
>>>>>>>     </cib>
>>>>>>>>>      I figured out a way get around this, but it would be easier to do if the CIB has worked as a with CMAN.
>>>>>>>>>      I just do not start the main resource if the attribute is not defined or it is not true.
>>>>>>>>>      This slightly changes the logic of the cluster.
>>>>>>>>>      But I'm not sure what the correct behavior.
>>>>>>>>>
>>>>>>>>>      libqb 0.14.4
>>>>>>>>>      corosync 2.3.1
>>>>>>>>>      pacemaker 1.1.11
>>>>>>>>>
>>>>>>>>>      All build from source in previews week.
>>>>>>>>>>>       Now in corosync.conf:
>>>>>>>>>>>
>>>>>>>>>>>       totem {
>>>>>>>>>>>              version: 2
>>>>>>>>>>>              crypto_cipher: none
>>>>>>>>>>>              crypto_hash: none
>>>>>>>>>>>              interface {
>>>>>>>>>>>                      ringnumber: 0
>>>>>>>>>>>       bindnetaddr: 10.76.157.18
>>>>>>>>>>>       mcastaddr: 239.94.1.56
>>>>>>>>>>>                      mcastport: 5405
>>>>>>>>>>>                      ttl: 1
>>>>>>>>>>>              }
>>>>>>>>>>>       }
>>>>>>>>>>>       logging {
>>>>>>>>>>>              fileline: off
>>>>>>>>>>>              to_stderr: no
>>>>>>>>>>>              to_logfile: yes
>>>>>>>>>>>              logfile: /var/log/cluster/corosync.log
>>>>>>>>>>>              to_syslog: yes
>>>>>>>>>>>              debug: on
>>>>>>>>>>>              timestamp: on
>>>>>>>>>>>              logger_subsys {
>>>>>>>>>>>                      subsys: QUORUM
>>>>>>>>>>>                      debug: on
>>>>>>>>>>>              }
>>>>>>>>>>>       }
>>>>>>>>>>>       quorum {
>>>>>>>>>>>              provider: corosync_votequorum
>>>>>>>>>>>       }
>>>>>>>>>>>       nodelist {
>>>>>>>>>>>       node {
>>>>>>>>>>>       ring0_addr: 10.76.157.17
>>>>>>>>>>>       }
>>>>>>>>>>>       node {
>>>>>>>>>>>       ring0_addr: 10.76.157.18
>>>>>>>>>>>       }
>>>>>>>>>>>       node {
>>>>>>>>>>>       ring0_addr: 10.76.157.19
>>>>>>>>>>>       }
>>>>>>>>>>>       }
>>>>>>>>>>>
>>>>>>>>>>>       _______________________________________________
>>>>>>>>>>>       Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>>>>>>>>>       http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>>>>>>>>
>>>>>>>>>>>       Project Home: http://www.clusterlabs.org
>>>>>>>>>>>       Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>>>>>>       Bugs: http://bugs.clusterlabs.org
>>>>>>>>>>      ,
>>>>>>>>>>      _______________________________________________
>>>>>>>>>>      Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>>>>>>>>      http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>>>>>>>
>>>>>>>>>>      Project Home: http://www.clusterlabs.org
>>>>>>>>>>      Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>>>>>      Bugs: http://bugs.clusterlabs.org
>>>>>>>>>      _______________________________________________
>>>>>>>>>      Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>>>>>>>      http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>>>>>>
>>>>>>>>>      Project Home: http://www.clusterlabs.org
>>>>>>>>>      Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>>>>      Bugs: http://bugs.clusterlabs.org
>>>>>>>>     ,
>>>>>>>>     _______________________________________________
>>>>>>>>     Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>>>>>>     http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>>>>>
>>>>>>>>     Project Home: http://www.clusterlabs.org
>>>>>>>>     Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>>>     Bugs: http://bugs.clusterlabs.org
>>>>>>>     _______________________________________________
>>>>>>>     Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>>>>>     http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>>>>
>>>>>>>     Project Home: http://www.clusterlabs.org
>>>>>>>     Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>>     Bugs: http://bugs.clusterlabs.org
>>>>>>    ,
>>>>>>    _______________________________________________
>>>>>>    Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>>>>    http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>>>
>>>>>>    Project Home: http://www.clusterlabs.org
>>>>>>    Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>    Bugs: http://bugs.clusterlabs.org
>>>>>   _______________________________________________
>>>>>   Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>>>   http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>>
>>>>>   Project Home: http://www.clusterlabs.org
>>>>>   Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>   Bugs: http://bugs.clusterlabs.org
>>>>   _______________________________________________
>>>>   Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>>   http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>>
>>>>   Project Home: http://www.clusterlabs.org
>>>>   Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>   Bugs: http://bugs.clusterlabs.org
>>>  ,
>>>  _______________________________________________
>>>  Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>>  http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>>
>>>  Project Home: http://www.clusterlabs.org
>>>  Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>  Bugs: http://bugs.clusterlabs.org
>>  _______________________________________________
>>  Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>>  http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>
>>  Project Home: http://www.clusterlabs.org
>>  Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>  Bugs: http://bugs.clusterlabs.org
>
> ,
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org




More information about the Pacemaker mailing list