[ClusterLabs] Coming in 1.1.15: Event-driven alerts

Klaus Wenninger kwenning at redhat.com
Fri May 13 15:51:48 UTC 2016


On 05/13/2016 04:59 PM, renayama19661014 at ybb.ne.jp wrote:
> i Klaus,
>
> After all we want transmission order.
> I think that I am going to use meta_attribute which you suggested and am enough.
>
> (snip)
> <alert id="alert-1"
> path="/xxx/xxxx/xxxxxxx.sh">
> <meta_attributes id="meta-alert-1-queue">
> <nvpair id="alert-1-queue" name="queue" value="alert1-queue" />
Maybe we find a name that somehow implies what the purpose of the
queue is. Something like "serialization-queue" coming to my mind
although it is a little bit of an alliteration of course.
> </meta_attributes>
> </alert>
> <alert id="alert-2"
> path="/xxx/xxxx/xxxxxxx.sh">
>
> (snip)
>
> I intend to write the correction that included a cue in this meta_attribute, what do you think?
> I think that processing when queue was changed is troublesome, but think that I can write the patch somehow.
The current implementation for the alerts-feature doesn't directly use
data from cib-diffs coming
in, but just triggers a query for the whole section, purges all local
data and feeds it in again from
what the query returns.
Thus I would just drain the queues - actively or just wait - before
deleting them together with
all the other local data. The alerts section is not expected to be
altered frequently during operation.
>
> We intend to make the correction that pcmk_snmp_helper.shkeeps versatility.
I tried to ask around a little bit to find out which tools were widely
used to collect and process
traps and how these are coping with traps arriving out of order because of
whatever reason (local reordering done by the scheduler - as we have it
here,
different delays on the network from different trap-sources, loss of
order on the
network, ...).
Unfortunately I didn't get real answers.
But maybe somebody here on the list can give a hint on that topic.
Anyway I found OID hrSystemDate (1.3.6.1.2.1.25.1.2) - part of MIB-2.
We could at least add this one to the pacemaker-mib-OIDs in
pcmk_snmp_helper.sh
and feed it with a correct timestamp (created by crmd).

>
> We want to use this function in Pacemaker1.1.15.
>
> Best Regards,
> Hideo Yamauch.
>
>
>
> ----- Original Message -----
>> From: "renayama19661014 at ybb.ne.jp" <renayama19661014 at ybb.ne.jp>
>> To: "kwenning at redhat.com" <kwenning at redhat.com>; "users at clusterlabs.org" <users at clusterlabs.org>; Cluster Labs - All topics related to open-source clustering welcomed <users at clusterlabs.org>
>> Cc: 
>> Date: 2016/5/12, Thu 06:28
>> Subject: Re: [ClusterLabs] Coming in 1.1.15: Event-driven alerts
>>
>> Hi Klaus,
>>
>> Thank you for comment.
>>
>> I confirm your comment.
>> I think that I ask you a question again.
>>
>>
>> Many thanks!
>> Hideo Yamauchi.
>>
>>
>> ----- Original Message -----
>>>  From: Klaus Wenninger <kwenning at redhat.com>
>>>  To: users at clusterlabs.org
>>>  Cc: 
>>>  Date: 2016/5/11, Wed 14:13
>>>  Subject: Re: [ClusterLabs] Coming in 1.1.15: Event-driven alerts
>>>
>>>  On 05/10/2016 11:19 PM, renayama19661014 at ybb.ne.jp wrote:
>>>>   Hi All,
>>>>
>>>>   After all our member needs the control of the turn of the transmission 
>> of 
>>>  the SNMP trap.
>>>>   We make a patch of the control of the turn of the transmission and 
>> intend 
>>>  to send it.
>>>>   Probably, with the patch, we add the "ordered" attribute 
>> that we 
>>>  sent by an email before.
>>>  Actually I still don't think that simple serialization of the calling 
>> of
>>>  the snmptrap-tool
>>>  is a good solution to tackle the problem of loosing the order of traps
>>>  arriving at
>>>  some management station:
>>>
>>>  - makes things worse in case of traps coming from multiple nodes
>>>  - doesn't help when the order is lost on the network.
>>>
>>>  Anyway I see 2 other scenarios where a certain degree of serialization 
>> might
>>>  be helpful:
>>>
>>>  - alert agent-scripts that can't handle being called concurrently
>>>  - performance issues that might arise on some systems that lack the
>>>    performance-headroom needed and/or the agent-scripts in place
>>>    require significant effort and/or there are a lot of resources/events
>>>    that trigger a vast amount of alerts being handled in parallel
>>>
>>>  So I could imagine the introduction of a meta-atribute that specifies a
>>>  queue
>>>  to be used for serialization.
>>>
>>>  - 'none' is default and leads to the behavior we have at the 
>> moment.
>>>  - any other queue-name leads to the instantiation of an additional queue
>>>
>>>  This approach should allow merely any kind of serialization you can think 
>> of
>>>  with as little impact as needed.
>>>  e.g. if the agent doesn't cope with concurrent calls you use a queue 
>> per
>>>  agent leading to all recipients being handled in a serialized way (and of
>>>  course the different alerts as well). And all the other agents are running
>>>  in parallel.
>>>  e.g. you can have a separate queue for a single recipient leading to
>>>  the alerts being sent there being serialized.
>>>  e.g. if the performance impact should be kept at a minimal level you
>>>  would use a single queue for all agents and all recipients 
>>>
>>>>
>>>>   Best Regards,
>>>>   Hideo Yamauchi.
>>>>
>>>>
>>>>   ----- Original Message -----
>>>>>   From: "renayama19661014 at ybb.ne.jp" 
>>>  <renayama19661014 at ybb.ne.jp>
>>>>>   To: "kwenning at redhat.com" <kwenning at redhat.com>; 
>>>  "users at clusterlabs.org" <users at clusterlabs.org>; Cluster 
>> Labs - 
>>>  All topics related to open-source clustering welcomed 
>>>  <users at clusterlabs.org>
>>>>>   Cc: 
>>>>>   Date: 2016/4/28, Thu 22:43
>>>>>   Subject: Re: [ClusterLabs] Coming in 1.1.15: Event-driven alerts
>>>>>
>>>>>   Hi Klaus,
>>>>>
>>>>>   Because the script is performed the effectiveness of in async, I 
>> think 
>>>  that it 
>>>>>   is difficult to set "uptime" by the method of the 
>> sample.
>>>>>   After all we may request the transmission of the order.
>>>>>   #The patch before mine only controls a practice turn of the async 
>> and 
>>>  is not a 
>>>>>   thing giving load of crmd.
>>>>>
>>>>>   Japan begins a rest for one week from tomorrow.
>>>>>   I discuss after vacation with a member.
>>>>>
>>>>>   Best Regards,
>>>>>   Hideo Yamauchi.
>>>>>
>>>>>
>>>>>
>>>>>   ----- Original Message -----
>>>>>>    From: Klaus Wenninger <kwenning at redhat.com>
>>>>>>    To: users at clusterlabs.org
>>>>>>    Cc: 
>>>>>>    Date: 2016/4/28, Thu 03:14
>>>>>>    Subject: Re: [ClusterLabs] Coming in 1.1.15: Event-driven 
>> alerts
>>>>>>    On 04/27/2016 04:19 PM, renayama19661014 at ybb.ne.jp wrote:
>>>>>>>     Hi All,
>>>>>>>
>>>>>>>     We have a request for a new SNMP function.
>>>>>>>
>>>>>>>
>>>>>>>     The order of traps is not right.
>>>>>>>
>>>>>>>     The turn of the trap is not sometimes followed.
>>>>>>>     This is because the handling of notice carries out 
>>>  "path" in 
>>>>>>    async.
>>>>>>>     I think that it is necessary to wait for completion of 
>> the 
>>>  practice at 
>>>>>>    "path" unit of "alerts".
>>>>>>>      
>>>>>>>     The turn of the trap is different from the real stop 
>> order of 
>>>  the 
>>>>>   resource.
>>>>>>    Writing the alerts in a local list and having the 
>> alert-scripts 
>>>  called
>>>>>>    in a serialized manner
>>>>>>    would lead to the snmptrap-tool creating timestamps in the 
>> order 
>>>  of the
>>>>>>    occurrence 
>>>>>>    of the alerts.
>>>>>>    Having the snmp-manager order the traps by timestamp this 
>> would 
>>>  indeed
>>>>>>    lead to
>>>>>>    seeing them in the order they had occured.
>>>>>>
>>>>>>    But this approach has a number of drawbacks:
>>>>>>
>>>>>>    - it works just when the traps are coming from one node as 
>> there 
>>>  is no
>>>>>>    way to serialize
>>>>>>      over nodes - at least none that would work under all 
>>>  circumstances we
>>>>>>    want alerts
>>>>>>      to be delivered
>>>>>>
>>>>>>    - it distorts the timestamps created even more from the 
>> points in 
>>>  time
>>>>>>    when the
>>>>>>      alert had been triggered - making the result in a 
>>>  multi-node-scenario
>>>>>>    even worse and
>>>>>>      making it hard to correlate with other sources of 
>> information 
>>>  like
>>>>>>    logfiles
>>>>>>
>>>>>>    - if you imagine a scenario with multiple mechanisms of 
>> delivering 
>>>  an
>>>>>>    alert + multiple
>>>>>>      recipients we couldn't use a single list but we would 
>> need 
>>>  something 
>>>>>   more
>>>>>>      complicated to prevent unneeded delays, delays coming from 
>> one 
>>>  of the
>>>>>>    delivery
>>>>>>      methods not working properly due to e.g. a recipient that 
>> is not
>>>>>>    reachable, ...
>>>>>>      (all solvable of course but if it doesn't solve your 
>> problem 
>>>  in the
>>>>>>    first place why the effort)
>>>>>>
>>>>>>    The alternative approach taken doesn't create the 
>> timestamps 
>>>  in the
>>>>>>    scripts but
>>>>>>    provides timestamps to the scripts already.
>>>>>>    This way it doesn't matter if the execution of the script 
>> is 
>>>  delayed.
>>>>>>
>>>>>>    A short example how this approach could be used with 
>> snmp-traps:
>>>>>>    edit pcmk_snmp_helper.sh:
>>>>>>
>>>>>>    ...
>>>>>>    starttickfile="/var/run/starttick"
>>>>>>
>>>>>>    # hack to have a reference
>>>>>>    # can have it e.g. in an attribute to be visible throughout 
>> the 
>>>  cluster
>>>>>>    if [ ! -f ${starttickfile} ] ; then
>>>>>>            echo ${CRM_alert_timestamp} > ${starttickfile}
>>>>>>    fi
>>>>>>
>>>>>>    starttick=`cat ${starttickfile}`
>>>>>>    ticks=`eval ${CRM_alert_timestamp} - ${starttick}`
>>>>>>
>>>>>>    if [[ ${CRM_alert_rc} != 0 && ${CRM_alert_task} == 
>>>>>   "monitor" 
>>>>>>    ]] || [[
>>>>>>    ${CRM_alert_task} != "monitor" ]] ; then
>>>>>>        # This trap is compliant with PACEMAKER MIB
>>>>>>        # 
>>>>>>    
>>>  https://github.com/ClusterLabs/pacemaker/blob/master/extra/PCMK-MIB.txt
>>>>>>        /usr/bin/snmptrap -v 2c -c public ${CRM_alert_recipient} 
>>>  ${ticks}
>>>>>>    PACEMAKER-MIB::pacemakerNotificationTrap \
>>>>>>            PACEMAKER-MIB::pacemakerNotificationNode s 
>>>>>   "${CRM_alert_node}" 
>>>>>>    \
>>>>>>            PACEMAKER-MIB::pacemakerNotificationResource s 
>>>>>>    "${CRM_alert_rsc}" \
>>>>>>            PACEMAKER-MIB::pacemakerNotificationOperation s
>>>>>>    "${CRM_alert_task}" \
>>>>>>            PACEMAKER-MIB::pacemakerNotificationDescription s
>>>>>>    "${CRM_alert_desc}" \
>>>>>>            PACEMAKER-MIB::pacemakerNotificationStatus i 
>>>>>>    "${CRM_alert_status}" \
>>>>>>            PACEMAKER-MIB::pacemakerNotificationReturnCode i 
>>>  ${CRM_alert_rc} 
>>>>>   \
>>>>>>            PACEMAKER-MIB::pacemakerNotificationTargetReturnCode 
>> i
>>>>>>    ${CRM_alert_target_rc} && exit 0 || exit 1
>>>>>>    fi
>>>>>>
>>>>>>    exit 0
>>>>>>    ...
>>>>>>
>>>>>>    add a section to the cib:
>>>>>>
>>>>>>    cibadmin --create --xml-text '<configuration> 
>>>  <alerts> 
>>>>>   <alert
>>>>>>    id="snmp_traps" 
>>>>>>    
>>>  path="/usr/share/pacemaker/tests/pcmk_snmp_helper.sh">
>>>>>>    <meta_attributes id="meta_snmp_traps"> 
>> <nvpair 
>>>>>>    id="snmp_timestamp"
>>>>>>    name="tstamp_format" value="%s%02N"/> 
>>>>>>    </meta_attributes> <recipient
>>>>>>    id="trap_destination" 
>>>  value="192.168.123.3"/> 
>>>>>>    </alert> </alerts>
>>>>>>    </configuration>'
>>>>>>
>>>>>>
>>>>>>    This should solve the issue of correct order after being 
>> sorted by
>>>>>>    timestamps
>>>>>>    without having the ugly side-effects as described above.
>>>>>>
>>>>>>    I hope I understood your scenario correctly and this small 
>> example
>>>>>>    points out how I roughly would suggest to cope with the 
>> issue.
>>>>>>    Regards,
>>>>>>    Klaus  
>>>>>>>     ----
>>>>>>>     [root at rh72-01 ~]# grep Operation  /var/log/ha-log | grep 
>> stop
>>>>>>>     Apr 25 18:48:48 rh72-01 crmd[28897]:  notice: Operation 
>>>>>   prmDummy1_stop_0: 
>>>>>>    ok (node=rh72-01, call=33, rc=0, cib-update=56, 
>> confirmed=true)
>>>>>>>     Apr 25 18:48:48 rh72-01 crmd[28897]:  notice: Operation 
>>>>>   prmDummy3_stop_0: 
>>>>>>    ok (node=rh72-01, call=37, rc=0, cib-update=57, 
>> confirmed=true)
>>>>>>>     Apr 25 18:48:48 rh72-01 crmd[28897]:  notice: Operation 
>>>>>   prmDummy4_stop_0: 
>>>>>>    ok (node=rh72-01, call=39, rc=0, cib-update=58, 
>> confirmed=true)
>>>>>>>     Apr 25 18:48:48 rh72-01 crmd[28897]:  notice: Operation 
>>>>>   prmDummy2_stop_0: 
>>>>>>    ok (node=rh72-01, call=35, rc=0, cib-update=59, 
>> confirmed=true)
>>>>>>>     Apr 25 18:48:48 rh72-01 crmd[28897]:  notice: Operation 
>>>>>   prmDummy5_stop_0: 
>>>>>>    ok (node=rh72-01, call=41, rc=0, cib-update=60, 
>> confirmed=true)
>>>>>>>     Apr 25 18:48:50 snmp-manager snmptrapd[6865]: 2016-04-25 
>>>  18:48:50 
>>>>>>    <UNKNOWN> [UDP: 
>>>>>>
>> [192.168.28.170]:40613->[192.168.28.189]:162]:#012DISMAN-EVENT-MIB::sysUpTimeInstance 
>>
>>>>>>    = Timeticks: (25512486) 2 days, 
>>>  22:52:04.86#011SNMPv2-MIB::snmpTrapOID.0 = 
>>>>>   OID: 
>>>>>
>> PACEMAKER-MIB::pacemakerNotificationTrap#011PACEMAKER-MIB::pacemakerNotificationNode 
>>
>>>>>>    = STRING: 
>>>>>   
>> "rh72-01"#011PACEMAKER-MIB::pacemakerNotificationResource = 
>>>>>>    STRING: 
>>>>>   
>> "prmDummy3"#011PACEMAKER-MIB::pacemakerNotificationOperation 
>>>  = 
>>>>>>    STRING: 
>>>  "stop"#011PACEMAKER-MIB::pacemakerNotificationDescription 
>>>>>   = 
>>>>>>    STRING: 
>>>  "ok"#011PACEMAKER-MIB::pacemakerNotificationStatus = 
>>>>>   INTEGER: 
>>>>>>    0#011PACEMAKER-MIB::pacemakerNotificationReturnCode = 
>> INTEGER: 
>>>>>>    0#011PACEMAKER-MIB::pacemakerNotificationTargetReturnCode = 
>>>  INTEGER: 0
>>>>>>>     Apr 25 18:48:50 snmp-manager snmptrapd[6865]: 2016-04-25 
>>>  18:48:50 
>>>>>>    <UNKNOWN> [UDP: 
>>>>>>
>> [192.168.28.170]:39581->[192.168.28.189]:162]:#012DISMAN-EVENT-MIB::sysUpTimeInstance 
>>
>>>>>>    = Timeticks: (25512489) 2 days, 
>>>  22:52:04.89#011SNMPv2-MIB::snmpTrapOID.0 = 
>>>>>   OID: 
>>>>>
>> PACEMAKER-MIB::pacemakerNotificationTrap#011PACEMAKER-MIB::pacemakerNotificationNode 
>>
>>>>>>    = STRING: 
>>>>>   
>> "rh72-01"#011PACEMAKER-MIB::pacemakerNotificationResource = 
>>>>>>    STRING: 
>>>>>   
>> "prmDummy4"#011PACEMAKER-MIB::pacemakerNotificationOperation 
>>>  = 
>>>>>>    STRING: 
>>>  "stop"#011PACEMAKER-MIB::pacemakerNotificationDescription 
>>>>>   = 
>>>>>>    STRING: 
>>>  "ok"#011PACEMAKER-MIB::pacemakerNotificationStatus = 
>>>>>   INTEGER: 
>>>>>>    0#011PACEMAKER-MIB::pacemakerNotificationReturnCode = 
>> INTEGER: 
>>>>>>    0#011PACEMAKER-MIB::pacemakerNotificationTargetReturnCode = 
>>>  INTEGER: 0
>>>>>>>     Apr 25 18:48:50 snmp-manager snmptrapd[6865]: 2016-04-25 
>>>  18:48:50 
>>>>>>    <UNKNOWN> [UDP: 
>>>>>>
>> [192.168.28.170]:37166->[192.168.28.189]:162]:#012DISMAN-EVENT-MIB::sysUpTimeInstance 
>>
>>>>>>    = Timeticks: (25512490) 2 days, 
>>>  22:52:04.90#011SNMPv2-MIB::snmpTrapOID.0 = 
>>>>>   OID: 
>>>>>
>> PACEMAKER-MIB::pacemakerNotificationTrap#011PACEMAKER-MIB::pacemakerNotificationNode 
>>
>>>>>>    = STRING: 
>>>>>   
>> "rh72-01"#011PACEMAKER-MIB::pacemakerNotificationResource = 
>>>>>>    STRING: 
>>>>>   
>> "prmDummy1"#011PACEMAKER-MIB::pacemakerNotificationOperation 
>>>  = 
>>>>>>    STRING: 
>>>  "stop"#011PACEMAKER-MIB::pacemakerNotificationDescription 
>>>>>   = 
>>>>>>    STRING: 
>>>  "ok"#011PACEMAKER-MIB::pacemakerNotificationStatus = 
>>>>>   INTEGER: 
>>>>>>    0#011PACEMAKER-MIB::pacemakerNotificationReturnCode = 
>> INTEGER: 
>>>>>>    0#011PACEMAKER-MIB::pacemakerNotificationTargetReturnCode = 
>>>  INTEGER: 0
>>>>>>>     Apr 25 18:48:50 snmp-manager snmptrapd[6865]: 2016-04-25 
>>>  18:48:50 
>>>>>>    <UNKNOWN> [UDP: 
>>>>>>
>> [192.168.28.170]:53502->[192.168.28.189]:162]:#012DISMAN-EVENT-MIB::sysUpTimeInstance 
>>
>>>>>>    = Timeticks: (25512494) 2 days, 
>>>  22:52:04.94#011SNMPv2-MIB::snmpTrapOID.0 = 
>>>>>   OID: 
>>>>>
>> PACEMAKER-MIB::pacemakerNotificationTrap#011PACEMAKER-MIB::pacemakerNotificationNode 
>>
>>>>>>    = STRING: 
>>>>>   
>> "rh72-01"#011PACEMAKER-MIB::pacemakerNotificationResource = 
>>>>>>    STRING: 
>>>>>   
>> "prmDummy2"#011PACEMAKER-MIB::pacemakerNotificationOperation 
>>>  = 
>>>>>>    STRING: 
>>>  "stop"#011PACEMAKER-MIB::pacemakerNotificationDescription 
>>>>>   = 
>>>>>>    STRING: 
>>>  "ok"#011PACEMAKER-MIB::pacemakerNotificationStatus = 
>>>>>   INTEGER: 
>>>>>>    0#011PACEMAKER-MIB::pacemakerNotificationReturnCode = 
>> INTEGER: 
>>>>>>    0#011PACEMAKER-MIB::pacemakerNotificationTargetReturnCode = 
>>>  INTEGER: 0
>>>>>>>     Apr 25 18:48:50 snmp-manager snmptrapd[6865]: 2016-04-25 
>>>  18:48:50 
>>>>>>    <UNKNOWN> [UDP: 
>>>>>>
>> [192.168.28.170]:45956->[192.168.28.189]:162]:#012DISMAN-EVENT-MIB::sysUpTimeInstance 
>>
>>>>>>    = Timeticks: (25512497) 2 days, 
>>>  22:52:04.97#011SNMPv2-MIB::snmpTrapOID.0 = 
>>>>>   OID: 
>>>>>
>> PACEMAKER-MIB::pacemakerNotificationTrap#011PACEMAKER-MIB::pacemakerNotificationNode 
>>
>>>>>>    = STRING: 
>>>>>   
>> "rh72-01"#011PACEMAKER-MIB::pacemakerNotificationResource = 
>>>>>>    STRING: 
>>>>>   
>> "prmDummy5"#011PACEMAKER-MIB::pacemakerNotificationOperation 
>>>  = 
>>>>>>    STRING: 
>>>  "stop"#011PACEMAKER-MIB::pacemakerNotificationDescription 
>>>>>   = 
>>>>>>    STRING: 
>>>  "ok"#011PACEMAKER-MIB::pacemakerNotificationStatus = 
>>>>>   INTEGER: 
>>>>>>    0#011PACEMAKER-MIB::pacemakerNotificationReturnCode = 
>> INTEGER: 
>>>>>>    0#011PACEMAKER-MIB::pacemakerNotificationTargetReturnCode = 
>>>  INTEGER: 0
>>>>>>>     ----
>>>>>>>
>>>>>>>     I think that there is "timestamp" attribute 
>> for 
>>>  async by 
>>>>>   this 
>>>>>>    change.
>>>>>>>     The order of traps may be important to a user.
>>>>>>>     I suggest addition to "alert" element with 
>>>>>   "orderd" 
>>>>>>    attribute.
>>>>>>>      * orderd 
>>>>>>>         false : The present processing.
>>>>>>>         true  : Control the transmission order of the trap.
>>>>>>>
>>>>>>>     ----
>>>>>>>     <configuration>
>>>>>>>       <alerts>
>>>>>>>         <alert id="notify_9"
>>>>>>>    
>>>  path="/usr/share/pacemaker/tests/pcmk_alert_sample1.sh" 
>>>>>>    ordered="true">
>>>>>>>     (snip)
>>>>>>>         </alert>
>>>>>>>         <alert id="notify_9"
>>>>>>>    
>>>  path="/usr/share/pacemaker/tests/pcmk_alert_sample2.sh" 
>>>>>>    ordered="false">
>>>>>>>     (snip)
>>>>>>>         </alert>
>>>>>>>       </alerts>
>>>>>>>     </configuration>
>>>>>>>
>>>>>>>     ----
>>>>>>>
>>>>>>>     I send a patch to cope with this problem before.
>>>>>>>     The former patch may be useful for the correction.
>>>>>>>      * https://github.com/ClusterLabs/pacemaker/pull/847
>>>>>>>
>>>>>>>     I intend to write the patch if everybody agrees to 
>>>  "ordered" 
>>>>>>    attribute.
>>>>>>>     Best Regards,
>>>>>>>     Hideo Yamauchi.
>>>>>>>
>>>>>>>     _______________________________________________
>>>>>>>     Users mailing list: Users at clusterlabs.org
>>>>>>>    http://clusterlabs.org/mailman/listinfo/users
>>>>>>>
>>>>>>>     Project Home: http://www.clusterlabs.org
>>>>>>>     Getting started: 
>>>>>   http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>>     Bugs: http://bugs.clusterlabs.org
>>>>>>    _______________________________________________
>>>>>>    Users mailing list: Users at clusterlabs.org
>>>>>>    http://clusterlabs.org/mailman/listinfo/users
>>>>>>
>>>>>>    Project Home: http://www.clusterlabs.org
>>>>>>    Getting started: 
>>>  http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>>    Bugs: http://bugs.clusterlabs.org
>>>>>>
>>>>>   _______________________________________________
>>>>>   Users mailing list: Users at clusterlabs.org
>>>>>   http://clusterlabs.org/mailman/listinfo/users
>>>>>
>>>>>   Project Home: http://www.clusterlabs.org
>>>>>   Getting started: 
>>>  http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>>   Bugs: http://bugs.clusterlabs.org
>>>>>
>>>>   _______________________________________________
>>>>   Users mailing list: Users at clusterlabs.org
>>>>   http://clusterlabs.org/mailman/listinfo/users
>>>>
>>>>   Project Home: http://www.clusterlabs.org
>>>>   Getting started: 
>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>>   Bugs: http://bugs.clusterlabs.org
>>>
>>>  _______________________________________________
>>>  Users mailing list: Users at clusterlabs.org
>>>  http://clusterlabs.org/mailman/listinfo/users
>>>
>>>  Project Home: http://www.clusterlabs.org
>>>  Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>>>  Bugs: http://bugs.clusterlabs.org
>>>
>> _______________________________________________
>> Users mailing list: Users at clusterlabs.org
>> http://clusterlabs.org/mailman/listinfo/users
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>>





More information about the Users mailing list