[ClusterLabs] Problems with corosync and pacemaker with error scenarios
Gerhard Wiesinger
lists at wiesinger.com
Mon Jan 16 12:18:54 EST 2017
Hello Ken,
thank you for the answers.
On 16.01.2017 16:43, Ken Gaillot wrote:
> On 01/16/2017 08:56 AM, Gerhard Wiesinger wrote:
>> Hello,
>>
>> I'm new to corosync and pacemaker and I want to setup a nginx cluster
>> with quorum.
>>
>> Requirements:
>> - 3 Linux maschines
>> - On 2 maschines floating IP should be handled and nginx as a load
>> balancing proxy
>> - 3rd maschine is for quorum only, no services must run there
>>
>> Installed on all 3 nodes corosync/pacemaker, firewall ports openend are:
>> 5404, 5405, 5406 for udp in both directions
> If you're using firewalld, the easiest configuration is:
>
> firewall-cmd --permanent --add-service=high-availability
>
> If not, depending on what you're running, you may also want to open TCP
> ports 2224 (pcsd), 3121 (Pacemaker Remote), and 21064 (DLM).
I'm using shorewall on the lb01/lb02 nodes and firewalld on kvm01.
pcs status
Cluster name: lbcluster
Stack: corosync
Current DC: lb01 (version 1.1.16-1.fc25-94ff4df) - partition with quorum
Last updated: Mon Jan 16 16:46:52 2017
Last change: Mon Jan 16 15:07:59 2017 by root via cibadmin on lb01
3 nodes configured
40 resources configured
Online: [ kvm01 lb01 lb02 ]
Full list of resources:
...
Daemon Status:
corosync: active/enabled
pacemaker: active/enabled
pcsd: inactive/disabled
BTW: I'm not running pcsd, as far as I know it is for UI configuration
only So ports ports 2224 (pcsd), 3121 (Pacemaker Remote), and 21064
(DLM) are closed. Shouldn't be a problem, right?
>
>> OS: Fedora 25
>>
>> Configuration of corosync (only the bindnetaddr is different on every
>> maschine) and pacemaker below.
> FYI you don't need a different bindnetaddr. You can (and generally
> should) use the *network* address, which is the same on all hosts.
Only lb01 and lb02 are on the same network, kvm01 is on a different
location and network therefore.
>> Configuration works so far but error test scenarios don't work like
>> expected:
>> 1.) I had cases in testing without qourum and quorum again where the
>> cluster kept in Stopped state
>> I had to restart the whole stack to get it online again (killall -9
>> corosync;systemctl restart corosync;systemctl restart pacemaker)
>> Any ideas?
> It will be next to impossible to say without logs. It's definitely not
> expected behavior. Stopping is the correct response to losing quorum;
> perhaps quorum is not being properly restored for some reason. What is
> your test methodology?
I had it when I rebooted just one node.
Testing scenarios are:
*) Rebooting
*) Starting/stopping corosync
*) network down simulation on lb01/lb02
*) putting an interface down with ifconfig eth1:1 down (simulation of
loosing an IP address)
*) see also below
Tested now again with all nodes up (I've configured 13 ip adresses for
the sake of getting a faster overview I posted only the config for 2 ip
adresses):
No automatic recovery happens.
e.g. ifconfig eth1:1 down
Resource Group: ClusterNetworking
ClusterIP_01 (ocf::heartbeat:IPaddr2): FAILED lb02
ClusterIPRoute_01 (ocf::heartbeat:Route): FAILED lb02
ClusterIPRule_01 (ocf::heartbeat:Iprule): Started lb02
ClusterIP_02 (ocf::heartbeat:IPaddr2): FAILED lb02
ClusterIPRoute_02 (ocf::heartbeat:Route): FAILED lb02 (blocked)
ClusterIPRule_02 (ocf::heartbeat:Iprule): Stopped
ClusterIP_03 (ocf::heartbeat:IPaddr2): Stopped
ClusterIPRoute_03 (ocf::heartbeat:Route): Stopped
ClusterIPRule_03 (ocf::heartbeat:Iprule): Stopped
...
ClusterIP_13 (ocf::heartbeat:IPaddr2): Stopped
ClusterIPRoute_13 (ocf::heartbeat:Route): Stopped
ClusterIPRule_13 (ocf::heartbeat:Iprule): Stopped
webserver (ocf::heartbeat:nginx): Stopped
Failed Actions:
* ClusterIP_01_monitor_10000 on lb02 'not running' (7): call=176,
status=complete, exitreason='none',
last-rc-change='Mon Jan 16 16:53:49 2017', queued=0ms, exec=0ms
* ClusterIP_02_monitor_10000 on lb02 'not running' (7): call=182,
status=complete, exitreason='none',
last-rc-change='Mon Jan 16 16:54:01 2017', queued=0ms, exec=0ms
Only this helps:
killall -9 corosync;systemctl restart corosync;systemctl restart pacemaker
================================================================================================================================================================
also had now on a test:
Failed Actions:
* webserver_start_0 on lb02 'not configured' (6): call=499,
status=complete, exitreason='none',
last-rc-change='Mon Jan 16 17:04:13 2017', queued=0ms, exec=4120ms
Why is it not configured now?
killall -9 corosync;systemctl restart corosync;systemctl restart pacemaker
got to the next situation below where some of them are FAILED and
blocked on lb02 and cluster didn't start up (why is it started on
different clusters anyway???)
ClusterIP_05 (ocf::heartbeat:IPaddr2): Started lb01
ClusterIPRoute_05 (ocf::heartbeat:Route): Started lb01
ClusterIPRule_05 (ocf::heartbeat:Iprule): Started lb01
ClusterIP_06 (ocf::heartbeat:IPaddr2): Started lb01
ClusterIPRoute_06 (ocf::heartbeat:Route): FAILED lb02 (blocked)
ClusterIPRule_06 (ocf::heartbeat:Iprule): Stopped
ClusterIP_07 (ocf::heartbeat:IPaddr2): Stopped
ClusterIPRoute_07 (ocf::heartbeat:Route): Stopped
ClusterIPRule_07 (ocf::heartbeat:Iprule): Stopped
>
>> 2.) Restarting pacemaker on inactive node also restarts resources on the
>> other active node:
>> a.) Everything up & ok
>> b.) lb01 handles all resources
>> c.) on lb02 which handles no resrouces: systemctl restart pacemaker:
>> All resources will also be restart with a short outage on lb01 (state
>> is Stopped, Started[ lb01 lb02 ] and then Started lb02)
>> How can this be avoided?
> This is not expected behavior, except with clones, which I don't see you
> using.
Is my config OK? I'm not using any clones ...
>
>> 3.) Stopping and starting corosync doesn't awake the node up again:
>> systemctl stop corosync;sleep 10;systemctl restart corosync
>> Online: [ kvm01 lb01 ]
>> OFFLINE: [ lb02 ]
>> Stays in that state until pacemaker is restarted: systemctl restart
>> pacemaker
>> Bug?
> No, pacemaker should always restart if corosync restarts. That is
> specified in the systemd units, so I'm not sure why pacemaker didn't
> automatically restart in your case.
OK, can you reproduce it?
>
>> 4.) "systemctl restart corosync" hangs sometimes (waiting 2 min)
>> needs a
>> killall -9 corosync;systemctl restart corosync;systemctl restart
>> pacemaker
>> sequence to get it up gain
>>
>> 5.) Simulation of split brain: Disabling/reenabling local firewall
>> (ports 5404, 5405, 5406) on node lb01 and lb02 for the following ports
> FYI for an accurate simulation, be sure to block both incoming and
> outgoing traffic on the corosync ports.
Yes I'm blocking both incoming and outgoing:
?IF 1
CORO1(ACCEPT) net $FW
CORO1(ACCEPT) $FW net
CORO2(ACCEPT) net $FW
CORO2(ACCEPT) $FW net
CORO3(ACCEPT) net $FW
CORO3(ACCEPT) $FW net
?ELSE
CORO1(DROP) net $FW
CORO1(DROP) $FW net
CORO2(DROP) net $FW
CORO2(DROP) $FW net
CORO3(DROP) net $FW
CORO3(DROP) $FW net
?ENDIF
Where CORO1-CORO3 are macros for ports 5404, 5405, 5406.
>
>> doesn't bring corosync up again after reenabling lb02 firewall
>> partition WITHOUT quorum
>> Online: [ kvm01 ]
>> OFFLINE: [ lb01 lb02 ]
>> NOK: restart on lb02: systemctl restart corosync;systemctl restart
>> pacemaker
>> OK: restart on lb02 and kvm01 (quorum host): systemctl restart
>> corosync;systemctl restart pacemaker
>> I also see that non enabled hosts (quorum hosts) are also tried to be
>> started on kvm01
>> Started[ kvm01 lb02 ]
>> Started lb02
>> Any ideas?
>>
>> I've also written a new ocf:heartbeat:Iprule to modify "ip rule"
>> accordingly.
>>
>> Versions are:
>> corosync: 2.4.2
>> pacemaker: 1.1.16
>> Kernel: 4.9.3-200.fc25.x86_64
>>
>> Thnx.
>>
>> Ciao,
>> Gerhard
>>
>> Corosync config:
>> ================================================================================================================================================================
>>
>> totem {
>> version: 2
>> cluster_name: lbcluster
>> crypto_cipher: aes256
>> crypto_hash: sha512
>> interface {
>> ringnumber: 0
>> bindnetaddr: 1.2.3.35
>> mcastport: 5405
>> }
>> transport: udpu
>> }
>> logging {
>> fileline: off
>> to_logfile: yes
>> to_syslog: yes
>> logfile: /var/log/cluster/corosync.log
>> debug: off
>> timestamp: on
>> logger_subsys {
>> subsys: QUORUM
>> debug: off
>> }
>> }
>> nodelist {
>> node {
>> ring0_addr: lb01
>> nodeid: 1
>> }
>> node {
>> ring0_addr: lb02
>> nodeid: 2
>> }
>> node {
>> ring0_addr: kvm01
>> nodeid: 3
>> }
>> }
>> quorum {
>> # Enable and configure quorum subsystem (default: off)
>> # see also corosync.conf.5 and votequorum.5
>> #provider: corosync_votequorum
>> provider: corosync_votequorum
>> # Only for 2 node setup!
>> # two_node: 1
>> }
>> ================================================================================================================================================================
>>
>> ################################################################################################################################################################
>>
>> # Default properties
>> ################################################################################################################################################################
>>
>> pcs property set stonith-enabled=false
> FYI fencing is the only way to recover from certain failure scenarios,
> so be aware you'll have problems if those happen.
>
> E.g. if one of the lb's experiences crippling CPU or I/O load, it will
> be unable to function as a member of the cluster (including stopping
> resources), but the cluster will be unable to recover resources
> elsewhere because it can't be sure they are not still active.
I'm aware of that, I can add this later on.
>
>> pcs property set no-quorum-policy=stop
>> pcs property set default-resource-stickiness=100
>> pcs property set symmetric-cluster=false
>> ################################################################################################################################################################
>>
>> # Delete & cleanup resources
>> ################################################################################################################################################################
>>
>> pcs resource delete webserver
>> pcs resource cleanup webserver
>> pcs resource delete ClusterIP_01
>> pcs resource cleanup ClusterIP_01
>> pcs resource delete ClusterIPRoute_01
>> pcs resource cleanup ClusterIPRoute_01
>> pcs resource delete ClusterIPRule_01
>> pcs resource cleanup ClusterIPRule_01
>> pcs resource delete ClusterIP_02
>> pcs resource cleanup ClusterIP_02
>> pcs resource delete ClusterIPRoute_02
>> pcs resource cleanup ClusterIPRoute_02
>> pcs resource delete ClusterIPRule_02
>> pcs resource cleanup ClusterIPRule_02
>> ################################################################################################################################################################
>>
>> # Create resources
>> ################################################################################################################################################################
>>
>> pcs resource create ClusterIP_01 ocf:heartbeat:IPaddr2 ip=1.2.3.81
>> nic=eth1 cidr_netmask=28 broadcast=1.2.3.95 iflabel=1 meta
>> migration-threshold=2 op monitor timeout=20s interval=10s
>> on-fail=restart --group ClusterNetworking
>> pcs resource create ClusterIPRoute_01 ocf:heartbeat:Route params
>> device=eth1 source=1.2.3.81 destination=default gateway=1.2.3.94
>> table=125 meta migration-threshold=2 op monitor timeout=20s interval=10s
>> on-fail=restart --group ClusterNetworking --after ClusterIP_01
>> pcs resource create ClusterIPRule_01 ocf:heartbeat:Iprule params
>> from=1.2.3.81 table=125 meta migration-threshold=2 op monitor
>> timeout=20s interval=10s on-fail=restart --group ClusterNetworking
>> --after ClusterIPRoute_01
>> pcs constraint location ClusterIP_01 prefers lb01=INFINITY
>> pcs constraint location ClusterIP_01 prefers lb02=INFINITY
>> pcs constraint location ClusterIPRoute_01 prefers lb01=INFINITY
>> pcs constraint location ClusterIPRoute_01 prefers lb02=INFINITY
>> pcs constraint location ClusterIPRule_01 prefers lb01=INFINITY
>> pcs constraint location ClusterIPRule_01 prefers lb02=INFINITY
>> pcs resource create ClusterIP_02 ocf:heartbeat:IPaddr2 ip=1.2.3.82
>> nic=eth1 cidr_netmask=28 broadcast=1.2.3.95 iflabel=2 meta
>> migration-threshold=2 op monitor timeout=20s interval=10s
>> on-fail=restart --group ClusterNetworking
>> pcs resource create ClusterIPRoute_02 ocf:heartbeat:Route params
>> device=eth1 source=1.2.3.82 destination=default gateway=1.2.3.94
>> table=126 meta migration-threshold=2 op monitor timeout=20s interval=10s
>> on-fail=restart --group ClusterNetworking --after ClusterIP_02
>> pcs resource create ClusterIPRule_02 ocf:heartbeat:Iprule params
>> from=1.2.3.82 table=126 meta migration-threshold=2 op monitor
>> timeout=20s interval=10s on-fail=restart --group ClusterNetworking
>> --after ClusterIPRoute_02
>> pcs constraint location ClusterIP_02 prefers lb01=INFINITY
>> pcs constraint location ClusterIP_02 prefers lb02=INFINITY
>> pcs constraint location ClusterIPRoute_02 prefers lb01=INFINITY
>> pcs constraint location ClusterIPRoute_02 prefers lb02=INFINITY
>> pcs constraint location ClusterIPRule_02 prefers lb01=INFINITY
>> pcs constraint location ClusterIPRule_02 prefers lb02=INFINITY
>> ################################################################################################################################################################
>>
>> # NGINX
>> ################################################################################################################################################################
>>
>> pcs resource create webserver ocf:heartbeat:nginx httpd=/usr/sbin/nginx
>> configfile=/etc/nginx/nginx.conf meta migration-threshold=2 op monitor
>> timeout=5s interval=5s on-fail=restart
>> pcs constraint colocation add webserver with ClusterNetworking INFINITY
>> pcs constraint order ClusterNetworking then webserver
>> pcs constraint location webserver prefers lb01=INFINITY
>> pcs constraint location webserver prefers lb02=INFINITY
>> ================================================================================================================================================================
>
> _______________________________________________
> Users mailing list: Users at clusterlabs.org
> http://lists.clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
More information about the Users
mailing list