[ClusterLabs] Antw: Expected recovery behavior of remote-node guest when corosync ring0 is lost in a passive mode RRP config?
Ulrich Windl
Ulrich.Windl at rz.uni-regensburg.de
Thu Mar 2 02:37:52 EST 2017
>>> "Scott Greenlese" <swgreenl at us.ibm.com> schrieb am 01.03.2017 um 22:07 in
Nachricht
<OFFC50C6DC.1138528D-ON002580D6.006F49AA-852580D6.00740858 at notes.na.collabserv.c
m>:
> Hi..
>
> I am running a few corosync "passive mode" Redundant Ring Protocol (RRP)
> failure scenarios, where
> my cluster has several remote-node VirtualDomain resources running on each
> node in the cluster,
> which have been configured to allow Live Guest Migration (LGM) operations.
>
> While both corosync rings are active, if I drop ring0 on a given node where
> I have remote node (guests) running,
> I noticed that the guest will be shutdown / re-started on the same host,
> after which the connection is re-established
> and the guest proceeds to run on that same cluster node.
Could it be you forgot "allow-migrate=true" at the resource level or some migration IP address at the node level?
I only have SLES11 here...
>
> I am wondering why pacemaker doesn't try to "live" migrate the remote node
> (guest) to a different node, instead
> of rebooting the guest? Is there some way to configure the remote nodes
> such that the recovery action is
> LGM instead of reboot when the host-to-remote_node connect is lost in an
> RRP situation? I guess the
> next question is, is it even possible to LGM a remote node guest if the
> corosync ring fails over from ring0 to ring1
> (or vise-versa)?
>
> # For example, here's a remote node's VirtualDomain resource definition.
>
> [root at zs95kj]# pcs resource show zs95kjg110102_res
> Resource: zs95kjg110102_res (class=ocf provider=heartbeat
> type=VirtualDomain)
> Attributes: config=/guestxml/nfs1/zs95kjg110102.xml
> hypervisor=qemu:///system migration_transport=ssh
> Meta Attrs: allow-migrate=true remote-node=zs95kjg110102
> remote-addr=10.20.110.102
> Operations: start interval=0s timeout=480
> (zs95kjg110102_res-start-interval-0s)
> stop interval=0s timeout=120
> (zs95kjg110102_res-stop-interval-0s)
> monitor interval=30s (zs95kjg110102_res-monitor-interval-30s)
> migrate-from interval=0s timeout=1200
> (zs95kjg110102_res-migrate-from-interval-0s)
> migrate-to interval=0s timeout=1200
> (zs95kjg110102_res-migrate-to-interval-0s)
> [root at zs95kj VD]#
>
>
>
>
> # My RRP rings are active, and configured "rrp_mode="passive"
>
> [root at zs95kj ~]# corosync-cfgtool -s
> Printing ring status.
> Local node ID 2
> RING ID 0
> id = 10.20.93.12
> status = ring 0 active with no faults
> RING ID 1
> id = 10.20.94.212
> status = ring 1 active with no faults
>
>
>
> # Here's the corosync.conf ..
>
> [root at zs95kj ~]# cat /etc/corosync/corosync.conf
> totem {
> version: 2
> secauth: off
> cluster_name: test_cluster_2
> transport: udpu
> rrp_mode: passive
> }
>
> nodelist {
> node {
> ring0_addr: zs95kjpcs1
> ring1_addr: zs95kjpcs2
> nodeid: 2
> }
>
> node {
> ring0_addr: zs95KLpcs1
> ring1_addr: zs95KLpcs2
> nodeid: 3
> }
>
> node {
> ring0_addr: zs90kppcs1
> ring1_addr: zs90kppcs2
> nodeid: 4
> }
>
> node {
> ring0_addr: zs93KLpcs1
> ring1_addr: zs93KLpcs2
> nodeid: 5
> }
>
> node {
> ring0_addr: zs93kjpcs1
> ring1_addr: zs93kjpcs2
> nodeid: 1
> }
> }
>
> quorum {
> provider: corosync_votequorum
> }
>
> logging {
> to_logfile: yes
> logfile: /var/log/corosync/corosync.log
> timestamp: on
> syslog_facility: daemon
> to_syslog: yes
> debug: on
>
> logger_subsys {
> debug: off
> subsys: QUORUM
> }
> }
>
>
>
>
> # Here's the vlan / route situation on cluster node zs95kj:
>
> ring0 is on vlan1293
> ring1 is on vlan1294
>
> [root at zs95kj ~]# route -n
> Kernel IP routing table
> Destination Gateway Genmask Flags Metric Ref Use
> Iface
> 0.0.0.0 10.20.93.254 0.0.0.0 UG 400 0 0
> vlan1293 << default route to guests from ring0
> 9.0.0.0 9.12.23.1 255.0.0.0 UG 400 0 0
> vlan508
> 9.12.23.0 0.0.0.0 255.255.255.0 U 400 0 0
> vlan508
> 10.20.92.0 0.0.0.0 255.255.255.0 U 400 0 0
> vlan1292
> 10.20.93.0 0.0.0.0 255.255.255.0 U 0 0 0
> vlan1293 << ring0 IPs
> 10.20.93.0 0.0.0.0 255.255.255.0 U 400 0 0
> vlan1293
> 10.20.94.0 0.0.0.0 255.255.255.0 U 0 0 0
> vlan1294 << ring1 IPs
> 10.20.94.0 0.0.0.0 255.255.255.0 U 400 0 0
> vlan1294
> 10.20.101.0 0.0.0.0 255.255.255.0 U 400 0 0
> vlan1298
> 10.20.109.0 10.20.94.254 255.255.255.0 UG 400 0 0
> vlan1294 << Route to guests on 10.20.109 from ring1
> 10.20.110.0 10.20.94.254 255.255.255.0 UG 400 0 0
> vlan1294 << Route to guests on 10.20.110 from ring1
> 169.254.0.0 0.0.0.0 255.255.0.0 U 1007 0 0
> enccw0.0.02e0
> 169.254.0.0 0.0.0.0 255.255.0.0 U 1016 0 0
> ovsbridge1
> 192.168.122.0 0.0.0.0 255.255.255.0 U 0 0 0
> virbr0
>
>
>
> # On remote node, you can see we have a connection back to the host.
>
> Feb 28 14:30:22 [928] zs95kjg110102 pacemaker_remoted: info:
> crm_log_init: Changed active directory to /var/lib/heartbeat/cores/root
> Feb 28 14:30:22 [928] zs95kjg110102 pacemaker_remoted: info:
> qb_ipcs_us_publish: server name: lrmd
> Feb 28 14:30:22 [928] zs95kjg110102 pacemaker_remoted: notice:
> lrmd_init_remote_tls_server: Starting a tls listener on port 3121.
> Feb 28 14:30:22 [928] zs95kjg110102 pacemaker_remoted: notice:
> bind_and_listen: Listening on address ::
> Feb 28 14:30:22 [928] zs95kjg110102 pacemaker_remoted: info:
> qb_ipcs_us_publish: server name: cib_ro
> Feb 28 14:30:22 [928] zs95kjg110102 pacemaker_remoted: info:
> qb_ipcs_us_publish: server name: cib_rw
> Feb 28 14:30:22 [928] zs95kjg110102 pacemaker_remoted: info:
> qb_ipcs_us_publish: server name: cib_shm
> Feb 28 14:30:22 [928] zs95kjg110102 pacemaker_remoted: info:
> qb_ipcs_us_publish: server name: attrd
> Feb 28 14:30:22 [928] zs95kjg110102 pacemaker_remoted: info:
> qb_ipcs_us_publish: server name: stonith-ng
> Feb 28 14:30:22 [928] zs95kjg110102 pacemaker_remoted: info:
> qb_ipcs_us_publish: server name: crmd
> Feb 28 14:30:22 [928] zs95kjg110102 pacemaker_remoted: info: main:
> Starting
> Feb 28 14:30:27 [928] zs95kjg110102 pacemaker_remoted: notice:
> lrmd_remote_listen: LRMD client connection established. 0x9ec18b50 id:
> 93e25ef0-4ff8-45ac-a6ed-f13b64588326
>
> zs95kjg110102:~ # netstat -anp
> Active Internet connections (servers and established)
> Proto Recv-Q Send-Q Local Address Foreign Address State
> PID/Program name
> tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
> 946/sshd
> tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN
> 1022/master
> tcp 0 0 0.0.0.0:5666 0.0.0.0:* LISTEN
> 931/xinetd
> tcp 0 0 0.0.0.0:5801 0.0.0.0:* LISTEN
> 931/xinetd
> tcp 0 0 0.0.0.0:5901 0.0.0.0:* LISTEN
> 931/xinetd
> tcp 0 0 :::21 :::* LISTEN
> 926/vsftpd
> tcp 0 0 :::22 :::* LISTEN
> 946/sshd
> tcp 0 0 ::1:25 :::* LISTEN
> 1022/master
> tcp 0 0 :::44931 :::* LISTEN
> 1068/xdm
> tcp 0 0 :::80 :::* LISTEN
> 929/httpd-prefork
> tcp 0 0 :::3121 :::* LISTEN
> 928/pacemaker_remot
> tcp 0 0 10.20.110.102:3121 10.20.93.12:46425
> ESTABLISHED 928/pacemaker_remot
> udp 0 0 :::177 :::*
> 1068/xdm
>
>
>
>
> ## Drop the ring0 (vlan1293) interface on cluster node zs95kj, causing fail
> over to ring1 (vlan1294)
>
> [root at zs95kj]# date;ifdown vlan1293
> Tue Feb 28 15:54:11 EST 2017
> Device 'vlan1293' successfully disconnected.
>
>
>
> ## Confirm that ring0 is now offline (a.k.a. "FAULTY")
>
> [root at zs95kj]# date;corosync-cfgtool -s
> Tue Feb 28 15:54:49 EST 2017
> Printing ring status.
> Local node ID 2
> RING ID 0
> id = 10.20.93.12
> status = Marking ringid 0 interface 10.20.93.12 FAULTY
> RING ID 1
> id = 10.20.94.212
> status = ring 1 active with no faults
> [root at zs95kj VD]#
>
>
>
>
> # See that the resource stayed local to cluster node zs95kj.
>
> [root at zs95kj]# date;pcs resource show |grep zs95kjg110102
> Tue Feb 28 15:55:32 EST 2017
> zs95kjg110102_res (ocf::heartbeat:VirtualDomain): Started zs95kjpcs1
> You have new mail in /var/spool/mail/root
>
>
>
> # On the remote node, show new entries in pacemaker.log showing connection
> re-established.
>
> Feb 28 15:55:17 [928] zs95kjg110102 pacemaker_remoted: notice:
> crm_signal_dispatch: Invoking handler for signal 15: Terminated
> Feb 28 15:55:17 [928] zs95kjg110102 pacemaker_remoted: info:
> lrmd_shutdown: Terminating with 1 clients
> Feb 28 15:55:17 [928] zs95kjg110102 pacemaker_remoted: info:
> qb_ipcs_us_withdraw: withdrawing server sockets
> Feb 28 15:55:17 [928] zs95kjg110102 pacemaker_remoted: info:
> crm_xml_cleanup: Cleaning up memory from libxml2
> Feb 28 15:55:37 [942] zs95kjg110102 pacemaker_remoted: info:
> crm_log_init: Changed active directory to /var/lib/heartbeat/cores/root
> Feb 28 15:55:37 [942] zs95kjg110102 pacemaker_remoted: info:
> qb_ipcs_us_publish: server name: lrmd
> Feb 28 15:55:37 [942] zs95kjg110102 pacemaker_remoted: notice:
> lrmd_init_remote_tls_server: Starting a tls listener on port 3121.
> Feb 28 15:55:37 [942] zs95kjg110102 pacemaker_remoted: notice:
> bind_and_listen: Listening on address ::
> Feb 28 15:55:37 [942] zs95kjg110102 pacemaker_remoted: info:
> qb_ipcs_us_publish: server name: cib_ro
> Feb 28 15:55:37 [942] zs95kjg110102 pacemaker_remoted: info:
> qb_ipcs_us_publish: server name: cib_rw
> Feb 28 15:55:37 [942] zs95kjg110102 pacemaker_remoted: info:
> qb_ipcs_us_publish: server name: cib_shm
> Feb 28 15:55:37 [942] zs95kjg110102 pacemaker_remoted: info:
> qb_ipcs_us_publish: server name: attrd
> Feb 28 15:55:37 [942] zs95kjg110102 pacemaker_remoted: info:
> qb_ipcs_us_publish: server name: stonith-ng
> Feb 28 15:55:37 [942] zs95kjg110102 pacemaker_remoted: info:
> qb_ipcs_us_publish: server name: crmd
> Feb 28 15:55:37 [942] zs95kjg110102 pacemaker_remoted: info: main:
> Starting
> Feb 28 15:55:38 [942] zs95kjg110102 pacemaker_remoted: notice:
> lrmd_remote_listen: LRMD client connection established. 0xbed1ab50 id:
> b19ed532-6f61-4d9c-9439-ffb836eea34f
> zs95kjg110102:~ #
>
>
>
> zs95kjg110102:~ # netstat -anp |less
> Active Internet connections (servers and established)
> Proto Recv-Q Send-Q Local Address Foreign Address State
> PID/Program name
> tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
> 961/sshd
> tcp 0 0 127.0.0.1:25 0.0.0.0:* LISTEN
> 1065/master
> tcp 0 0 0.0.0.0:5666 0.0.0.0:* LISTEN
> 946/xinetd
> tcp 0 0 0.0.0.0:5801 0.0.0.0:* LISTEN
> 946/xinetd
> tcp 0 0 0.0.0.0:5901 0.0.0.0:* LISTEN
> 946/xinetd
> tcp 0 0 10.20.110.102:22 10.20.94.32:57749
> ESTABLISHED 1134/0
> tcp 0 0 :::21 :::* LISTEN
> 941/vsftpd
> tcp 0 0 :::22 :::* LISTEN
> 961/sshd
> tcp 0 0 ::1:25 :::* LISTEN
> 1065/master
> tcp 0 0 :::80 :::* LISTEN
> 944/httpd-prefork
> tcp 0 0 :::3121 :::* LISTEN
> 942/pacemaker_remot
> tcp 0 0 :::34836 :::* LISTEN
> 1070/xdm
> tcp 0 0 10.20.110.102:3121 10.20.94.212:49666
> ESTABLISHED 942/pacemaker_remot
> udp 0 0 :::177 :::*
> 1070/xdm
>
>
>
> ## On host node, zs95kj show system messages indicating remote node (guest)
> shutdown / start ... (but no attempt to LGM).
>
> [root at zs95kj ~]# grep "Feb 28" /var/log/messages |grep zs95kjg110102
>
> Feb 28 15:55:07 zs95kj crmd[121380]: error: Operation
> zs95kjg110102_monitor_30000: Timed Out (node=zs95kjpcs1, call=2,
> timeout=30000ms)
> Feb 28 15:55:07 zs95kj crmd[121380]: error: Unexpected disconnect on
> remote-node zs95kjg110102
> Feb 28 15:55:17 zs95kj crmd[121380]: notice: Operation
> zs95kjg110102_stop_0: ok (node=zs95kjpcs1, call=38, rc=0, cib-update=370,
> confirmed=true)
> Feb 28 15:55:17 zs95kj attrd[121378]: notice: Removing all zs95kjg110102
> attributes for zs95kjpcs1
> Feb 28 15:55:17 zs95kj VirtualDomain(zs95kjg110102_res)[173127]: INFO:
> Issuing graceful shutdown request for domain zs95kjg110102.
> Feb 28 15:55:23 zs95kj systemd-machined: Machine qemu-38-zs95kjg110102
> terminated.
> Feb 28 15:55:23 zs95kj crmd[121380]: notice: Operation
> zs95kjg110102_res_stop_0: ok (node=zs95kjpcs1, call=858, rc=0,
> cib-update=378, confirmed=true)
> Feb 28 15:55:24 zs95kj systemd-machined: New machine qemu-64-zs95kjg110102.
> Feb 28 15:55:24 zs95kj systemd: Started Virtual Machine
> qemu-64-zs95kjg110102.
> Feb 28 15:55:24 zs95kj systemd: Starting Virtual Machine
> qemu-64-zs95kjg110102.
> Feb 28 15:55:25 zs95kj crmd[121380]: notice: Operation
> zs95kjg110102_res_start_0: ok (node=zs95kjpcs1, call=859, rc=0,
> cib-update=385, confirmed=true)
> Feb 28 15:55:38 zs95kj crmd[121380]: notice: Operation
> zs95kjg110102_start_0: ok (node=zs95kjpcs1, call=44, rc=0, cib-update=387,
> confirmed=true)
> [root at zs95kj ~]#
>
>
> Once the remote node established re-connection, there was no further remote
> node / resource instability.
>
> Anyway, just wondering why there was no attempt to migrate this remote node
> guest as opposed to a reboot? Is it necessary to reboot the guest in
> order to be managed
> by pacemaker and corosync over the ring1 interface if ring0 goes down?
> Is live guest migration even possible if ring0 goes away and ring1 takes
> over?
>
> Thanks in advance..
>
> Scott Greenlese ... KVM on System Z - Solutions Test, IBM Poughkeepsie,
> N.Y.
> INTERNET: swgreenl at us.ibm.com
More information about the Users
mailing list