[ClusterLabs] Interface confusion
Ken Gaillot
kgaillot at redhat.com
Tue Mar 19 17:59:32 EDT 2019
On Tue, 2019-03-19 at 21:49 +0100, Adam Budziński wrote:
> Not sure I got “or (due to two_node / wait_for_all in corosync.conf)
> waits until it can see the other node before doing anything” right. I
> mean according tohttps://access.redhat.com/solutions/1295713 “Th
> e 'two_node' parameter sets the quorum to '1' and allows one node to
> remain quorate and continue cluster operations after the second one
> is fenced.”
You've got a great setup now. There's nothing wrong with qdevice not
being enabled, it just means you have to start it manually along with
corosync and pacemaker.
With qdevice, I'm not sure two_node or wait_for_all is relevant; a
corosync dev would have to jump in to say for sure.
At least without qdevice, two_node implies wait_for_all (in addition to
the quorum effect mentioned by that article). With wait_for_all, the
cluster can go down from two nodes to one node without a problem -- but
it can't start with just one node. At start-up, each node will wait
until it has seen the other before assuming everything is OK.
>
> Exactly the same parameters are set for my cluster:
>
> [root at srv1 ~]# corosync-quorumtool -s
> Quorum information
> ------------------
> Date: Tue Mar 19 16:15:50 2019
> Quorum provider: corosync_votequorum
> Nodes: 2
> Node ID: 1
> Ring ID: 1/464
> Quorate: Yes
>
> Votequorum information
> ----------------------
> Expected votes: 2
> Highest expected: 2
> Total votes: 2
> Quorum: 1
> Flags: 2Node Quorate WaitForAll
>
> Membership information
> ----------------------
> Nodeid Votes Name
> 1 1 srv1cr1 (local)
> 2 1 srv2cr1
>
> I was testing fencing (I’m using fence_vmware_soap) and followed
> https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/high_availability_add-on_reference/s1-stonithtest-haarand
> could in each case see that when:
>
> a) Fencing the passive node the active remained active *
> b) Fencing the active node caused the passive node to take over
> the active role*
> *in all cases pacemaker and corosync was not configured to start on
> boot
>
>
>
> Yes you are absolutely right regarding the fence condition when both
> shoot at the same time and I even tried option 3 from your list that
> is corosync qdevice. Here I followed
> https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/high_availability_add-on_reference/s1-quorumdev-haar#s2-managequorum-HAAR
>
> So here’s quorum runtime status before adding the qdevice:
>
> [root at srv1 ~]# pcs quorum status
> Quorum information
> ------------------
> Date: Tue Mar 19 16:34:12 2019
> Quorum provider: corosync_votequorum
> Nodes: 2
> Node ID: 1
> Ring ID: 1/464
> Quorate: Yes
>
> Votequorum information
> ----------------------
> Expected votes: 2
> Highest expected: 2
> Total votes: 2
> Quorum: 1
> Flags: 2Node Quorate WaitForAll
>
> Membership information
> ----------------------
> Nodeid Votes Qdevice Name
> 1 1 NR srv1cr1 (local)
> 2 1 NR srv2cr1
>
>
> And here’s the one after adding it:
>
> [root at srv1 ~]# pcs quorum status
> Quorum information
> ------------------
> Date: Tue Mar 19 16:35:06 2019
> Quorum provider: corosync_votequorum
> Nodes: 2
> Node ID: 1
> Ring ID: 1/464
> Quorate: Yes
>
> Votequorum information
> ----------------------
> Expected votes: 3
> Highest expected: 3
> Total votes: 3
> Quorum: 2
> Flags: Quorate WaitForAll Qdevice
>
> Membership information
> ----------------------
> Nodeid Votes Qdevice Name
> 1 1 A,V,NMW srv1cr1 (local)
> 2 1 A,V,NMW srv2cr1
> 0 1 Qdevice
>
>
>
> I got some while adding the quorum device to the cluster because as
> mentioned I have on both nodes pacemaker and corosync set to not
> start at boot:
>
> [root at srv1 ~]# pcs quorum device add model net host=otrs_db
> algorithm=ffsplit
> Setting up qdevice certificates on nodes...
> srv2cr1: Succeeded
> srv1cr1: Succeeded
> Enabling corosync-qdevice...
> srv2cr1: not enabling corosync-qdevice: corosync is not enabled
> srv1cr1: not enabling corosync-qdevice: corosync is not enabled
> Sending updated corosync.conf to nodes...
> srv2cr1: Succeeded
> srv1cr1: Succeeded
> Corosync configuration reloaded
> Starting corosync-qdevice...
> srv2cr1: corosync-qdevice started
> srv1cr1: corosync-qdevice started
>
>
>
> Is this a problem ? what can I do now, how can I test it ?
>
> Thank you !
>
> wt., 19.03.2019, 19:01 użytkownik Ken Gaillot <kgaillot at redhat.com>
> napisał:
> > On Tue, 2019-03-19 at 15:55 +0100, Adam Budziński wrote:
> > > Hello Ken,
> > >
> > > Thank you.
> > >
> > > But if I have a two node cluster and a working fencing mechanism
> > > wouldn't it be enough to disable the corosync and pacemaker
> > service
> > > on both nodes so when it fence they won't come up?
> >
> > There's actually no problem when a fenced node comes back. Either
> > it
> > joins the remaining cluster normally, or (due to two_node /
> > wait_for_all in corosync.conf) waits until it can see the other
> > node
> > before doing anything.
> >
> > Disabling or enabling the services is a personal preference based
> > on
> > whether you'd rather investigate why a node was shot before letting
> > it
> > back in the cluster, or get the cluster back to full strength as
> > quickly as possible.
> >
> > The fence delay is for when both nodes are running and
> > communicating
> > correctly, then suddenly they lose communication with each other.
> > From
> > each node's point of view, the other node is lost. So, each will
> > attempt to fence the other. A delay on one node in this situation
> > makes
> > it less likely that they both pull the trigger at the same time,
> > ending
> > up with both nodes dead.
> >
> > > Thank you
> > >
> > > pon., 18.03.2019, 16:19 użytkownik Ken Gaillot <
> > kgaillot at redhat.com>
> > > napisał:
> > > > On Sat, 2019-03-16 at 11:10 +0100, Adam Budziński wrote:
> > > > > Hello Andrei,
> > > > >
> > > > > Ok I see your point. So per my understanding if the resource
> > is
> > > > > started successfully in that case fence vmware it will be
> > > > monitored
> > > > > indefinitely but as you sad it will monitor the current
> > active
> > > > node.
> > > > > So how does the fence agent gets aware of problems with the
> > > > slave? I
> > > >
> > > > The fence agent doesn't monitor the active node, or any node --
> > it
> > > > monitors the fence device.
> > > >
> > > > The cluster layer (i.e. corosync) monitors all nodes, and
> > reports
> > > > any
> > > > issues to pacemaker, which will initiate fencing if necessary.
> > > >
> > > > Pacemaker also monitors each resource and fence device, via any
> > > > recurring monitors that have been configured.
> > > >
> > > > > mean if in a two node cluster the cluster splits in to two
> > > > partitions
> > > > > each of them will fence the other or does that happen because
> > > > both
> > > > > will assume they are the only survivors and thus need to
> > fence
> > > > the
> > > > > other node which is in a unknow state so to say?
> > > >
> > > > If both nodes are functional but can't see each other, they
> > will
> > > > each
> > > > want to initiate fencing. If one of them is quicker than the
> > other
> > > > to
> > > > determine this, the other one will get shot before it has a
> > chance
> > > > to
> > > > do anything itself.
> > > >
> > > > However there is the possibility that both nodes will shoot at
> > > > about
> > > > the same time, resulting in both nodes getting shot (a "stonith
> > > > death
> > > > match"). This is only a problem in 2-node clusters. There are a
> > few
> > > > ways around this:
> > > >
> > > > 1. Configure two separate fence devices, each targeting one of
> > the
> > > > nodes, and put a delay on one of them (or a random delay on
> > both).
> > > > This
> > > > makes it highly unlikely that they will shoot at the same time.
> > > >
> > > > 2. Configure a fencing topology with a fence heuristics device
> > plus
> > > > your real device. A fence heuristics device runs some test, and
> > > > refuses
> > > > to shoot the other node if the test fails. For example,
> > > > fence_heuristics_ping tries to ping an IP address you give it;
> > the
> > > > idea
> > > > is that if a node can't ping that IP, you don't want it to
> > survive.
> > > > This ensures that only a node that passes the test can shoot
> > (which
> > > > means there still might be some cases where the nodes can both
> > > > shoot
> > > > each other, and cases where the cluster will freeze because
> > neither
> > > > node can see the IP).
> > > >
> > > > 3. Configure corosync with qdevice to provide true quorum via a
> > > > third
> > > > host (which doesn't participate in the cluster otherwise).
> > > >
> > > > 4. Use sbd with a hardware watchdog and a shared storage device
> > as
> > > > the
> > > > fencing device. This is not a reliable option with VMWare, but
> > I'm
> > > > listing it for the general case.
> > > >
> > > >
> > > > >
> > > > > Thank you and Best Regards,
> > > > > Adam
> > > > >
> > > > > sob., 16.03.2019, 07:17 użytkownik Andrei Borzenkov <
> > > > > arvidjaar at gmail.com> napisał:
> > > > > > 16.03.2019 9:01, Adam Budziński пишет:
> > > > > > > Thank you Andrei. The problem is that I can see with 'pcs
> > > > status'
> > > > > > that
> > > > > > > resources are runnin on srv2cr1 but its at the same time
> > its
> > > > > > telling that
> > > > > > > the fence_vmware_soap is running on srv1cr1. That's
> > somewhat
> > > > > > confusing.
> > > > > > > Could you possibly explain this?
> > > > > > >
> > > > > >
> > > > > > Two points.
> > > > > >
> > > > > > It is actually logical to have stonith agent running on
> > > > different
> > > > > > node
> > > > > > than node with active resources - because it is the *other*
> > > > node
> > > > > > that
> > > > > > will initiate fencing when node with active resources
> > fails.
> > > > > >
> > > > > > But even considering the above, active (running) state of
> > fence
> > > > (or
> > > > > > stonith) agent just determines on which node recurring
> > monitor
> > > > > > operation
> > > > > > will be started. The actual result of this monitor
> > operation
> > > > has no
> > > > > > impact on subsequent stonith attempt and serves just as
> > warning
> > > > to
> > > > > > administrator. When stonith request comes, agent may be
> > used by
> > > > any
> > > > > > node
> > > > > > where stonith agent is not prohibited to run by (co-
> > )location
> > > > > > rules. My
> > > > > > understanding is that this node is selected by DC in
> > partition.
> > > > > >
> > > > > > > Thank you!
> > > > > > >
> > > > > > > sob., 16.03.2019, 05:37 użytkownik Andrei Borzenkov <
> > > > > > arvidjaar at gmail.com>
> > > > > > > napisał:
> > > > > > >
> > > > > > >> 16.03.2019 1:16, Adam Budziński пишет:
> > > > > > >>> Hi Tomas,
> > > > > > >>>
> > > > > > >>> Ok but how then pacemaker or the fence agent knows
> > which
> > > > route
> > > > > > to take to
> > > > > > >>> reach the vCenter?
> > > > > > >>
> > > > > > >> They do not know or care at all. It is up to your
> > underlying
> > > > > > operating
> > > > > > >> system and its routing tables.
> > > > > > >>
> > > > > > >>> Btw. Do I have to add the stonith resource on each of
> > the
> > > > nodes
> > > > > > or is it
> > > > > > >>> just enough to add it on one as for other resources?
> > > > > > >>
> > > > > > >> If your fencing agent can (should) be able to run on any
> > > > node,
> > > > > > it should
> > > > > > >> be enough to define it just once as long as it can
> > properly
> > > > > > determine
> > > > > > >> "port" to use on fencing "device" for a given node.
> > There
> > > > are
> > > > > > cases when
> > > > > > >> you may want to restrict fencing agent to only subset on
> > > > nodes
> > > > > > or when
> > > > > > >> you are forced to set unique parameter for each node
> > > > (consider
> > > > > > IPMI IP
> > > > > > >> address), in this case you would need separate instance
> > of
> > > > agent
> > > > > > in each
> > > > > > >> case.
> > > > > > >>
> > > > > > >>> Thank you!
> > > > > > >>>
> > > > > > >>> pt., 15.03.2019, 15:48 użytkownik Tomas Jelinek <
> > > > > > tojeline at redhat.com>
> > > > > > >>> napisał:
> > > > > > >>>
> > > > > > >>>> Dne 15. 03. 19 v 15:09 Adam Budziński napsal(a):
> > > > > > >>>>> Hello Tomas,
> > > > > > >>>>>
> > > > > > >>>>> Thank you! So far I need to say how great this
> > community
> > > > is,
> > > > > > would
> > > > > > >>>>> never expect so much positive vibes! A big thank you
> > your
> > > > > > doing a great
> > > > > > >>>>> job!
> > > > > > >>>>>
> > > > > > >>>>> Now let's talk business :)
> > > > > > >>>>>
> > > > > > >>>>> So if pcsd is using ring0 and it fails will ring1 not
> > be
> > > > used
> > > > > > at all?
> > > > > > >>>>
> > > > > > >>>> Pcs and pcsd never use ring1, but they are just tools
> > for
> > > > > > managing
> > > > > > >>>> clusters. You can have a perfectly functioning cluster
> > > > without
> > > > > > pcs and
> > > > > > >>>> pcsd running or even installed, it would be just more
> > > > > > complicated to set
> > > > > > >>>> it up and manage it.
> > > > > > >>>>
> > > > > > >>>> Even if ring0 fails, you will be able to use pcs (in
> > > > somehow
> > > > > > limited
> > > > > > >>>> manner) as most of its commands don't go through
> > network
> > > > > > anyway.
> > > > > > >>>>
> > > > > > >>>> Corosync, which is the actual cluster messaging layer,
> > > > will of
> > > > > > course
> > > > > > >>>> use ring1 in case of ring0 failure.
> > > > > > >>>>
> > > > > > >>>>>
> > > > > > >>>>> So in regards to VMware that would mean that the
> > > > interface
> > > > > > should be
> > > > > > >>>>> configured with a network that can access the
> > vCenter to
> > > > > > fence right?
> > > > > > >>>>> But wouldn't it then use only ring0 so if that fails
> > it
> > > > > > wouldn't switch
> > > > > > >>>>> to ring1?
> > > > > > >>>>
> > > > > > >>>> If you are talking about pcmk_host_map, that does not
> > > > really
> > > > > > have
> > > > > > >>>> anything to do with network interfaces of cluster
> > nodes.
> > > > It
> > > > > > maps node
> > > > > > >>>> names (parts before :) to "ports" of a fence device
> > (parts
> > > > > > after :).
> > > > > > >>>> Pcs-0.9.x does not support defining custom node names,
> > > > > > therefore node
> > > > > > >>>> names are the same as ring0 addresses.
> > > > > > >>>>
> > > > > > >>>> I am not an expert on fence agents / devices, but I'm
> > sure
> > > > > > someone else
> > > > > > >>>> on this list will be able to help you with configuring
> > > > fencing
> > > > > > for your
> > > > > > >>>> cluster.
> > > > > > >>>>
> > > > > > >>>>
> > > > > > >>>> Tomas
> > > > > > >>>>
> > > > > > >>>>>
> > > > > > >>>>> Thank you!
> > > > > > >>>>>
> > > > > > >>>>> pt., 15.03.2019, 13:14 użytkownik Tomas Jelinek <
> > > > > > tojeline at redhat.com
> > > > > > >>>>> <mailto:tojeline at redhat.com>> napisał:
> > > > > > >>>>>
> > > > > > >>>>> Dne 15. 03. 19 v 12:32 Adam Budziński napsal(a):
> > > > > > >>>>> > Hello Folks,____
> > > > > > >>>>> >
> > > > > > >>>>> > __ __
> > > > > > >>>>> >
> > > > > > >>>>> > Tow node active/passive VMware VM cluster.____
> > > > > > >>>>> >
> > > > > > >>>>> > __ __
> > > > > > >>>>> >
> > > > > > >>>>> > /etc/hosts____
> > > > > > >>>>> >
> > > > > > >>>>> > __ __
> > > > > > >>>>> >
> > > > > > >>>>> > 10.116.63.83 srv1____
> > > > > > >>>>> >
> > > > > > >>>>> > 10.116.63.84 srv2____
> > > > > > >>>>> >
> > > > > > >>>>> > 172.16.21.12 srv2cr1____
> > > > > > >>>>> >
> > > > > > >>>>> > 172.16.22.12 srv2cr2____
> > > > > > >>>>> >
> > > > > > >>>>> > 172.16.21.11 srv1cr1____
> > > > > > >>>>> >
> > > > > > >>>>> > 172.16.22.11 srv1cr2____
> > > > > > >>>>> >
> > > > > > >>>>> > __ __
> > > > > > >>>>> >
> > > > > > >>>>> > __ __
> > > > > > >>>>> >
> > > > > > >>>>> > I have 3 NIC’s on each VM:____
> > > > > > >>>>> >
> > > > > > >>>>> > __ __
> > > > > > >>>>> >
> > > > > > >>>>> > 10.116.63.83 srv1 and 10.116.63.84 srv2
> > are
> > > > > > networks used
> > > > > > >>>> to
> > > > > > >>>>> > access the VM’s via SSH or any resource
> > directly
> > > > if
> > > > > > not via a
> > > > > > >>>>> VIP.____
> > > > > > >>>>> >
> > > > > > >>>>> > __ __
> > > > > > >>>>> >
> > > > > > >>>>> > Everything with cr in its name is used for
> > > > corosync
> > > > > > >>>>> communication, so
> > > > > > >>>>> > basically I have two rings (this are two no
> > > > routable
> > > > > > networks
> > > > > > >>>>> just for
> > > > > > >>>>> > that).____
> > > > > > >>>>> >
> > > > > > >>>>> > __ __
> > > > > > >>>>> >
> > > > > > >>>>> > My questions are:____
> > > > > > >>>>> >
> > > > > > >>>>> > __ __
> > > > > > >>>>> >
> > > > > > >>>>> > __1.__With ‘pcs cluster auth’ which interface
> > /
> > > > > > interfaces
> > > > > > >> should
> > > > > > >>>>> I use
> > > > > > >>>>> > ?____
> > > > > > >>>>>
> > > > > > >>>>> Hi Adam,
> > > > > > >>>>>
> > > > > > >>>>> I can see you are using pcs-0.9.x. In that case
> > you
> > > > > > should do:
> > > > > > >>>>> pcs cluster auth srv1cr1 srv2cr1
> > > > > > >>>>>
> > > > > > >>>>> In other words, use the first address of each
> > node.
> > > > > > >>>>> Authenticating all the other addresses should not
> > > > cause
> > > > > > any issues.
> > > > > > >>>> It
> > > > > > >>>>> is pointless, though, as pcs only communicates
> > via
> > > > ring0
> > > > > > addresses.
> > > > > > >>>>>
> > > > > > >>>>> >
> > > > > > >>>>> > __2.__With ‘pcs cluster setup –name’ I would
> > use
> > > > the
> > > > > > corosync
> > > > > > >>>>> interfaces
> > > > > > >>>>> > e.g. ‘pcs cluster setup –name MyCluster
> > > > > > srv1cr1,srv1cr2
> > > > > > >>>>> srv2cr1,srv2cr2’
> > > > > > >>>>> > right ?____
> > > > > > >>>>>
> > > > > > >>>>> Yes, that is correct.
> > > > > > >>>>>
> > > > > > >>>>> >
> > > > > > >>>>> > __3.__With fence_vmware_soap
> > > > > > >> inpcmk_host_map="X:VM_C;X:VM:OTRS_D"
> > > > > > >>>>> which
> > > > > > >>>>> > interface should replace X ?____
> > > > > > >>>>>
> > > > > > >>>>> X should be replaced by node names as seen by
> > > > pacemaker.
> > > > > > Once you
> > > > > > >>>>> set up
> > > > > > >>>>> and start your cluster, run 'pcs status' to get
> > > > (amongs
> > > > > > other info)
> > > > > > >>>> the
> > > > > > >>>>> node names. In your configuration, they should be
> > > > srv1cr1
> > > > > > and
> > > > > > >>>> srv2cr1.
> > > > > > >>>>>
> > > > > > >>>>>
> > > > > > >>>>> Regards,
> > > > > > >>>>> Tomas
> > > > > > >>>>>
> > > > > > >>>>> > __ __
> > > > > > >>>>> >
> > > > > > >>>>> > Thank you!
> > > > > > >>>>> >
> > > > > > >>>>> >
> > > > > > >>>>> >
> > _______________________________________________
> > > > > > >>>>> > Users mailing list: Users at clusterlabs.org
> > > > > > >>>>> <mailto:Users at clusterlabs.org>
> > > > > > >>>>> >
> > > > https://lists.clusterlabs.org/mailman/listinfo/users
> > > > > > >>>>> >
> > > > > > >>>>> > Project Home: http://www.clusterlabs.org
> > > > > > >>>>> > Getting started:
> > > > > > >>>>>
> > > > http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > > > > > >>>>> > Bugs: http://bugs.clusterlabs.org
> > > > > > >>>>> >
> > > > > > >>>>> _______________________________________________
> > > > > > >>>>> Users mailing list: Users at clusterlabs.org
> > <mailto:
> > > > > > >>>> Users at clusterlabs.org>
> > > > > > >>>>>
> > https://lists.clusterlabs.org/mailman/listinfo/users
> > > > > > >>>>>
> > > > > > >>>>> Project Home: http://www.clusterlabs.org
> > > > > > >>>>> Getting started:
> > > > > > >>>>
> > http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > > > > > >>>>> Bugs: http://bugs.clusterlabs.org
> > > > > > >>>>>
> > > > > > >>>>>
> > > > > > >>>>> _______________________________________________
> > > > > > >>>>> Users mailing list: Users at clusterlabs.org
> > > > > > >>>>> https://lists.clusterlabs.org/mailman/listinfo/users
> > > > > > >>>>>
> > > > > > >>>>> Project Home: http://www.clusterlabs.org
> > > > > > >>>>> Getting started:
> > > > > > >> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > > > > > >>>>> Bugs: http://bugs.clusterlabs.org
> > > > > > >>>>>
> > > > > > >>>> _______________________________________________
> > > > > > >>>> Users mailing list: Users at clusterlabs.org
> > > > > > >>>> https://lists.clusterlabs.org/mailman/listinfo/users
> > > > > > >>>>
> > > > > > >>>> Project Home: http://www.clusterlabs.org
> > > > > > >>>> Getting started:
> > > > > > >> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > > > > > >>>> Bugs: http://bugs.clusterlabs.org
> > > > > > >>>>
> > > > > > >>>
> > > > > > >>>
> > > > > > >>> _______________________________________________
> > > > > > >>> Users mailing list: Users at clusterlabs.org
> > > > > > >>> https://lists.clusterlabs.org/mailman/listinfo/users
> > > > > > >>>
> > > > > > >>> Project Home: http://www.clusterlabs.org
> > > > > > >>> Getting started:
> > > > > > http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > > > > > >>> Bugs: http://bugs.clusterlabs.org
> > > > > > >>>
> > > > > > >>
> > > > > > >> _______________________________________________
> > > > > > >> Users mailing list: Users at clusterlabs.org
> > > > > > >> https://lists.clusterlabs.org/mailman/listinfo/users
> > > > > > >>
> > > > > > >> Project Home: http://www.clusterlabs.org
> > > > > > >> Getting started:
> > > > > > http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > > > > > >> Bugs: http://bugs.clusterlabs.org
> > > > > > >>
> > > > > > >
> > > > > > >
> > > > > > > _______________________________________________
> > > > > > > Users mailing list: Users at clusterlabs.org
> > > > > > > https://lists.clusterlabs.org/mailman/listinfo/users
> > > > > > >
> > > > > > > Project Home: http://www.clusterlabs.org
> > > > > > > Getting started:
> > > > > > http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > > > > > > Bugs: http://bugs.clusterlabs.org
> > > > > > >
> > > > > >
> > > > > > _______________________________________________
> > > > > > Users mailing list: Users at clusterlabs.org
> > > > > > https://lists.clusterlabs.org/mailman/listinfo/users
> > > > > >
> > > > > > Project Home: http://www.clusterlabs.org
> > > > > > Getting started:
> > > > > > http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > > > > > Bugs: http://bugs.clusterlabs.org
> > > > >
> > > > > _______________________________________________
> > > > > Users mailing list: Users at clusterlabs.org
> > > > > https://lists.clusterlabs.org/mailman/listinfo/users
> > > > >
> > > > > Project Home: http://www.clusterlabs.org
> > > > > Getting started:
> > > > > http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > > > > Bugs: http://bugs.clusterlabs.org
> > > > _______________________________________________
> > > > Manage your subscription:
> > > > https://lists.clusterlabs.org/mailman/listinfo/users
> > > >
> > > > ClusterLabs home: https://www.clusterlabs.org/
> > _______________________________________________
> > Manage your subscription:
> > https://lists.clusterlabs.org/mailman/listinfo/users
> >
> > ClusterLabs home: https://www.clusterlabs.org/
--
Ken Gaillot <kgaillot at redhat.com>
More information about the Users
mailing list