[ClusterLabs] Problems with corosync and pacemaker with error scenarios
Ken Gaillot
kgaillot at redhat.com
Thu Feb 9 18:39:18 EST 2017
On 01/16/2017 11:18 AM, Gerhard Wiesinger wrote:
> Hello Ken,
>
> thank you for the answers.
>
> On 16.01.2017 16:43, Ken Gaillot wrote:
>> On 01/16/2017 08:56 AM, Gerhard Wiesinger wrote:
>>> Hello,
>>>
>>> I'm new to corosync and pacemaker and I want to setup a nginx cluster
>>> with quorum.
>>>
>>> Requirements:
>>> - 3 Linux maschines
>>> - On 2 maschines floating IP should be handled and nginx as a load
>>> balancing proxy
>>> - 3rd maschine is for quorum only, no services must run there
>>>
>>> Installed on all 3 nodes corosync/pacemaker, firewall ports openend are:
>>> 5404, 5405, 5406 for udp in both directions
>> If you're using firewalld, the easiest configuration is:
>>
>> firewall-cmd --permanent --add-service=high-availability
>>
>> If not, depending on what you're running, you may also want to open TCP
>> ports 2224 (pcsd), 3121 (Pacemaker Remote), and 21064 (DLM).
>
> I'm using shorewall on the lb01/lb02 nodes and firewalld on kvm01.
>
> pcs status
> Cluster name: lbcluster
> Stack: corosync
> Current DC: lb01 (version 1.1.16-1.fc25-94ff4df) - partition with quorum
> Last updated: Mon Jan 16 16:46:52 2017
> Last change: Mon Jan 16 15:07:59 2017 by root via cibadmin on lb01
>
> 3 nodes configured
> 40 resources configured
>
> Online: [ kvm01 lb01 lb02 ]
>
> Full list of resources:
> ...
>
> Daemon Status:
> corosync: active/enabled
> pacemaker: active/enabled
> pcsd: inactive/disabled
>
> BTW: I'm not running pcsd, as far as I know it is for UI configuration
> only So ports ports 2224 (pcsd), 3121 (Pacemaker Remote), and 21064
> (DLM) are closed. Shouldn't be a problem, right?
pcs uses pcsd for most of its commands, so if you want to use pcs, it
should be enabled and allowed between nodes.
You don't have Pacemaker Remote nodes, so you can leave that port
closed. DLM is only necessary for certain resource types (such as clvmd).
>>> OS: Fedora 25
>>>
>>> Configuration of corosync (only the bindnetaddr is different on every
>>> maschine) and pacemaker below.
>> FYI you don't need a different bindnetaddr. You can (and generally
>> should) use the *network* address, which is the same on all hosts.
>
> Only lb01 and lb02 are on the same network, kvm01 is on a different
> location and network therefore.
I'm not familiar with corosync nodes on the same ring using different
networks, but I suppose it's OK since you're using udpu, with ring0_addr
specified for each node.
>>> Configuration works so far but error test scenarios don't work like
>>> expected:
>>> 1.) I had cases in testing without qourum and quorum again where the
>>> cluster kept in Stopped state
>>> I had to restart the whole stack to get it online again (killall -9
>>> corosync;systemctl restart corosync;systemctl restart pacemaker)
>>> Any ideas?
>> It will be next to impossible to say without logs. It's definitely not
>> expected behavior. Stopping is the correct response to losing quorum;
>> perhaps quorum is not being properly restored for some reason. What is
>> your test methodology?
>
> I had it when I rebooted just one node.
>
> Testing scenarios are:
> *) Rebooting
> *) Starting/stopping corosync
> *) network down simulation on lb01/lb02
> *) putting an interface down with ifconfig eth1:1 down (simulation of
> loosing an IP address)
> *) see also below
>
> Tested now again with all nodes up (I've configured 13 ip adresses for
> the sake of getting a faster overview I posted only the config for 2 ip
> adresses):
> No automatic recovery happens.
> e.g. ifconfig eth1:1 down
> Resource Group: ClusterNetworking
> ClusterIP_01 (ocf::heartbeat:IPaddr2): FAILED lb02
> ClusterIPRoute_01 (ocf::heartbeat:Route): FAILED lb02
> ClusterIPRule_01 (ocf::heartbeat:Iprule): Started lb02
> ClusterIP_02 (ocf::heartbeat:IPaddr2): FAILED lb02
> ClusterIPRoute_02 (ocf::heartbeat:Route): FAILED lb02 (blocked)
> ClusterIPRule_02 (ocf::heartbeat:Iprule): Stopped
> ClusterIP_03 (ocf::heartbeat:IPaddr2): Stopped
> ClusterIPRoute_03 (ocf::heartbeat:Route): Stopped
> ClusterIPRule_03 (ocf::heartbeat:Iprule): Stopped
> ...
> ClusterIP_13 (ocf::heartbeat:IPaddr2): Stopped
> ClusterIPRoute_13 (ocf::heartbeat:Route): Stopped
> ClusterIPRule_13 (ocf::heartbeat:Iprule): Stopped
> webserver (ocf::heartbeat:nginx): Stopped
>
>
> Failed Actions:
> * ClusterIP_01_monitor_10000 on lb02 'not running' (7): call=176,
> status=complete, exitreason='none',
> last-rc-change='Mon Jan 16 16:53:49 2017', queued=0ms, exec=0ms
> * ClusterIP_02_monitor_10000 on lb02 'not running' (7): call=182,
> status=complete, exitreason='none',
> last-rc-change='Mon Jan 16 16:54:01 2017', queued=0ms, exec=0ms
>
> Only this helps:
> killall -9 corosync;systemctl restart corosync;systemctl restart pacemaker
The question is why is ClusterIPRoute_02 blocked. As long as it's
blocked, the cluster can't recover.
Did it have a stop failure? If so, only fencing can recover from that,
and that would explain why it's blocked.
>
> ================================================================================================================================================================
>
> also had now on a test:
> Failed Actions:
> * webserver_start_0 on lb02 'not configured' (6): call=499,
> status=complete, exitreason='none',
> last-rc-change='Mon Jan 16 17:04:13 2017', queued=0ms, exec=4120ms
>
> Why is it not configured now?
That's an error code returned by the resource agent. You'd have to look
at the agent to see what situations cause it to return OCF_ERR_CONFIGURED.
Pacemaker considers OCF_ERR_CONFIGURED a fatal error, meaning the
resource will not be recovered anywhere. Pacemaker expects that agents
will only return this if the resource's *cluster configuration* is
invalid, but some agents return it if the resource's underlying service
configuration is invalid, which is unfortunate since it prevents
recovery on another node.
That seems to be the case here; a quick look at the nginx agent shows
OCF_ERR_CONFIGURED being returned for conditions local to the node.
> killall -9 corosync;systemctl restart corosync;systemctl restart pacemaker
> got to the next situation below where some of them are FAILED and
> blocked on lb02 and cluster didn't start up (why is it started on
> different clusters anyway???)
That looks like http://bugs.clusterlabs.org/show_bug.cgi?id=5301
It likely only occurs when stonith is disabled.
> ClusterIP_05 (ocf::heartbeat:IPaddr2): Started lb01
> ClusterIPRoute_05 (ocf::heartbeat:Route): Started lb01
> ClusterIPRule_05 (ocf::heartbeat:Iprule): Started lb01
> ClusterIP_06 (ocf::heartbeat:IPaddr2): Started lb01
> ClusterIPRoute_06 (ocf::heartbeat:Route): FAILED lb02 (blocked)
> ClusterIPRule_06 (ocf::heartbeat:Iprule): Stopped
> ClusterIP_07 (ocf::heartbeat:IPaddr2): Stopped
> ClusterIPRoute_07 (ocf::heartbeat:Route): Stopped
> ClusterIPRule_07 (ocf::heartbeat:Iprule): Stopped
>
>
>>
>>> 2.) Restarting pacemaker on inactive node also restarts resources on the
>>> other active node:
>>> a.) Everything up & ok
>>> b.) lb01 handles all resources
>>> c.) on lb02 which handles no resrouces: systemctl restart pacemaker:
>>> All resources will also be restart with a short outage on lb01 (state
>>> is Stopped, Started[ lb01 lb02 ] and then Started lb02)
>>> How can this be avoided?
>> This is not expected behavior, except with clones, which I don't see you
>> using.
>
> Is my config OK? I'm not using any clones ...
I can't explain this. Are you saying that for a short time, the
resources are started on both nodes?
The config looks OK, although I would use some number rather than
INFINITY to enable the resources on each node. INFINITY should be
reserved for mandatory placement (e.g. if you want to explicitly move
the resources to one node).
>>> 3.) Stopping and starting corosync doesn't awake the node up again:
>>> systemctl stop corosync;sleep 10;systemctl restart corosync
>>> Online: [ kvm01 lb01 ]
>>> OFFLINE: [ lb02 ]
>>> Stays in that state until pacemaker is restarted: systemctl restart
>>> pacemaker
>>> Bug?
>> No, pacemaker should always restart if corosync restarts. That is
>> specified in the systemd units, so I'm not sure why pacemaker didn't
>> automatically restart in your case.
>
> OK, can you reproduce it?
I see what happened: when you stop corosync, systemd properly stops
pacemaker, too. But when you start corosync, that doesn't require
pacemaker to be started. You have to explicitly start pacemaker.
It's generally better to use pcs cluster start/stop, which will handle
both corosync and pacemaker.
>>> 4.) "systemctl restart corosync" hangs sometimes (waiting 2 min)
>>> needs a
>>> killall -9 corosync;systemctl restart corosync;systemctl restart
>>> pacemaker
>>> sequence to get it up gain
>>>
>>> 5.) Simulation of split brain: Disabling/reenabling local firewall
>>> (ports 5404, 5405, 5406) on node lb01 and lb02 for the following ports
>> FYI for an accurate simulation, be sure to block both incoming and
>> outgoing traffic on the corosync ports.
>
> Yes I'm blocking both incoming and outgoing:
> ?IF 1
> CORO1(ACCEPT) net $FW
> CORO1(ACCEPT) $FW net
> CORO2(ACCEPT) net $FW
> CORO2(ACCEPT) $FW net
> CORO3(ACCEPT) net $FW
> CORO3(ACCEPT) $FW net
> ?ELSE
> CORO1(DROP) net $FW
> CORO1(DROP) $FW net
> CORO2(DROP) net $FW
> CORO2(DROP) $FW net
> CORO3(DROP) net $FW
> CORO3(DROP) $FW net
> ?ENDIF
>
> Where CORO1-CORO3 are macros for ports 5404, 5405, 5406.
>
>>
>>> doesn't bring corosync up again after reenabling lb02 firewall
>>> partition WITHOUT quorum
>>> Online: [ kvm01 ]
>>> OFFLINE: [ lb01 lb02 ]
>>> NOK: restart on lb02: systemctl restart corosync;systemctl restart
>>> pacemaker
>>> OK: restart on lb02 and kvm01 (quorum host): systemctl restart
>>> corosync;systemctl restart pacemaker
>>> I also see that non enabled hosts (quorum hosts) are also tried to be
>>> started on kvm01
>>> Started[ kvm01 lb02 ]
>>> Started lb02
>>> Any ideas?
>>>
>>> I've also written a new ocf:heartbeat:Iprule to modify "ip rule"
>>> accordingly.
>>>
>>> Versions are:
>>> corosync: 2.4.2
>>> pacemaker: 1.1.16
>>> Kernel: 4.9.3-200.fc25.x86_64
>>>
>>> Thnx.
>>>
>>> Ciao,
>>> Gerhard
>>>
>>> Corosync config:
>>> ================================================================================================================================================================
>>>
>>>
>>> totem {
>>> version: 2
>>> cluster_name: lbcluster
>>> crypto_cipher: aes256
>>> crypto_hash: sha512
>>> interface {
>>> ringnumber: 0
>>> bindnetaddr: 1.2.3.35
>>> mcastport: 5405
>>> }
>>> transport: udpu
>>> }
>>> logging {
>>> fileline: off
>>> to_logfile: yes
>>> to_syslog: yes
>>> logfile: /var/log/cluster/corosync.log
>>> debug: off
>>> timestamp: on
>>> logger_subsys {
>>> subsys: QUORUM
>>> debug: off
>>> }
>>> }
>>> nodelist {
>>> node {
>>> ring0_addr: lb01
>>> nodeid: 1
>>> }
>>> node {
>>> ring0_addr: lb02
>>> nodeid: 2
>>> }
>>> node {
>>> ring0_addr: kvm01
>>> nodeid: 3
>>> }
>>> }
>>> quorum {
>>> # Enable and configure quorum subsystem (default: off)
>>> # see also corosync.conf.5 and votequorum.5
>>> #provider: corosync_votequorum
>>> provider: corosync_votequorum
>>> # Only for 2 node setup!
>>> # two_node: 1
>>> }
>>> ================================================================================================================================================================
>>>
>>>
>>> ################################################################################################################################################################
>>>
>>>
>>> # Default properties
>>> ################################################################################################################################################################
>>>
>>>
>>> pcs property set stonith-enabled=false
>> FYI fencing is the only way to recover from certain failure scenarios,
>> so be aware you'll have problems if those happen.
>>
>> E.g. if one of the lb's experiences crippling CPU or I/O load, it will
>> be unable to function as a member of the cluster (including stopping
>> resources), but the cluster will be unable to recover resources
>> elsewhere because it can't be sure they are not still active.
>
> I'm aware of that, I can add this later on.
>
>>
>>> pcs property set no-quorum-policy=stop
>>> pcs property set default-resource-stickiness=100
>>> pcs property set symmetric-cluster=false
>>> ################################################################################################################################################################
>>>
>>>
>>> # Delete & cleanup resources
>>> ################################################################################################################################################################
>>>
>>>
>>> pcs resource delete webserver
>>> pcs resource cleanup webserver
>>> pcs resource delete ClusterIP_01
>>> pcs resource cleanup ClusterIP_01
>>> pcs resource delete ClusterIPRoute_01
>>> pcs resource cleanup ClusterIPRoute_01
>>> pcs resource delete ClusterIPRule_01
>>> pcs resource cleanup ClusterIPRule_01
>>> pcs resource delete ClusterIP_02
>>> pcs resource cleanup ClusterIP_02
>>> pcs resource delete ClusterIPRoute_02
>>> pcs resource cleanup ClusterIPRoute_02
>>> pcs resource delete ClusterIPRule_02
>>> pcs resource cleanup ClusterIPRule_02
>>> ################################################################################################################################################################
>>>
>>>
>>> # Create resources
>>> ################################################################################################################################################################
>>>
>>>
>>> pcs resource create ClusterIP_01 ocf:heartbeat:IPaddr2 ip=1.2.3.81
>>> nic=eth1 cidr_netmask=28 broadcast=1.2.3.95 iflabel=1 meta
>>> migration-threshold=2 op monitor timeout=20s interval=10s
>>> on-fail=restart --group ClusterNetworking
>>> pcs resource create ClusterIPRoute_01 ocf:heartbeat:Route params
>>> device=eth1 source=1.2.3.81 destination=default gateway=1.2.3.94
>>> table=125 meta migration-threshold=2 op monitor timeout=20s interval=10s
>>> on-fail=restart --group ClusterNetworking --after ClusterIP_01
>>> pcs resource create ClusterIPRule_01 ocf:heartbeat:Iprule params
>>> from=1.2.3.81 table=125 meta migration-threshold=2 op monitor
>>> timeout=20s interval=10s on-fail=restart --group ClusterNetworking
>>> --after ClusterIPRoute_01
>>> pcs constraint location ClusterIP_01 prefers lb01=INFINITY
>>> pcs constraint location ClusterIP_01 prefers lb02=INFINITY
>>> pcs constraint location ClusterIPRoute_01 prefers lb01=INFINITY
>>> pcs constraint location ClusterIPRoute_01 prefers lb02=INFINITY
>>> pcs constraint location ClusterIPRule_01 prefers lb01=INFINITY
>>> pcs constraint location ClusterIPRule_01 prefers lb02=INFINITY
>>> pcs resource create ClusterIP_02 ocf:heartbeat:IPaddr2 ip=1.2.3.82
>>> nic=eth1 cidr_netmask=28 broadcast=1.2.3.95 iflabel=2 meta
>>> migration-threshold=2 op monitor timeout=20s interval=10s
>>> on-fail=restart --group ClusterNetworking
>>> pcs resource create ClusterIPRoute_02 ocf:heartbeat:Route params
>>> device=eth1 source=1.2.3.82 destination=default gateway=1.2.3.94
>>> table=126 meta migration-threshold=2 op monitor timeout=20s interval=10s
>>> on-fail=restart --group ClusterNetworking --after ClusterIP_02
>>> pcs resource create ClusterIPRule_02 ocf:heartbeat:Iprule params
>>> from=1.2.3.82 table=126 meta migration-threshold=2 op monitor
>>> timeout=20s interval=10s on-fail=restart --group ClusterNetworking
>>> --after ClusterIPRoute_02
>>> pcs constraint location ClusterIP_02 prefers lb01=INFINITY
>>> pcs constraint location ClusterIP_02 prefers lb02=INFINITY
>>> pcs constraint location ClusterIPRoute_02 prefers lb01=INFINITY
>>> pcs constraint location ClusterIPRoute_02 prefers lb02=INFINITY
>>> pcs constraint location ClusterIPRule_02 prefers lb01=INFINITY
>>> pcs constraint location ClusterIPRule_02 prefers lb02=INFINITY
>>> ################################################################################################################################################################
>>>
>>>
>>> # NGINX
>>> ################################################################################################################################################################
>>>
>>>
>>> pcs resource create webserver ocf:heartbeat:nginx httpd=/usr/sbin/nginx
>>> configfile=/etc/nginx/nginx.conf meta migration-threshold=2 op monitor
>>> timeout=5s interval=5s on-fail=restart
>>> pcs constraint colocation add webserver with ClusterNetworking INFINITY
>>> pcs constraint order ClusterNetworking then webserver
>>> pcs constraint location webserver prefers lb01=INFINITY
>>> pcs constraint location webserver prefers lb02=INFINITY
>>> ================================================================================================================================================================
More information about the Users
mailing list