<html dir="ltr">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1">
<style id="owaParaStyle" type="text/css">P {margin-top:0;margin-bottom:0;}</style>
</head>
<body ocsi="0" fpstyle="1">
<div style="direction: ltr;font-family: Tahoma;color: #000000;font-size: 10pt;">Hello,<br>
<br>
I have set up a DRBD-Corosync-Pacemaker cluster following the instructions from <a href="https://wiki.ubuntu.com/ClusterStack/Natty">
https://wiki.ubuntu.com/ClusterStack/Natty</a> adapting them to CentOS 7 (e.g: using systemd). After testing it in Virtual Machines it seemed to be working fine, so it is now implemented in physical machines, and I have noticed that the failover works fine
as long as I kill the master by pulling the AC cable, but not if I issue the halt, reboot or shutdown commands, that makes the cluster get in a situation like this:
<br>
<br>
Last updated: Tue Aug 30 16:55:58 2016 Last change: Tue Aug 23 11:49:43 2016 by hacluster via crmd on nfsha2<br>
Stack: corosync<br>
Current DC: nfsha2 (version 1.1.13-10.el7_2.4-44eb2dd) - partition with quorum<br>
2 nodes and 9 resources configured<br>
<br>
Online: [ nfsha1 nfsha2 ]<br>
<br>
Master/Slave Set: ms_drbd_export [res_drbd_export]<br>
Masters: [ nfsha2 ]<br>
Slaves: [ nfsha1 ]<br>
Resource Group: rg_export<br>
res_fs (ocf::heartbeat:Filesystem): Started nfsha2<br>
res_exportfs_export1 (ocf::heartbeat:exportfs): FAILED nfsha2 (unmanaged)<br>
res_ip (ocf::heartbeat:IPaddr2): Stopped<br>
Clone Set: cl_nfsserver [res_nfsserver]<br>
Started: [ nfsha1 ]<br>
Clone Set: cl_exportfs_root [res_exportfs_root]<br>
res_exportfs_root (ocf::heartbeat:exportfs): FAILED nfsha2<br>
Started: [ nfsha1 ]<br>
<br>
Migration Summary:<br>
* Node 2:<br>
res_exportfs_export1: migration-threshold=1000000 fail-count=1000000 last-failure='Tue Aug 30 16:55:50 2016'<br>
res_exportfs_root: migration-threshold=1000000 fail-count=1 last-failure='Tue Aug 30 16:55:48 2016'<br>
* Node 1:<br>
<br>
Failed Actions:<br>
* res_exportfs_export1_stop_0 on nfsha2 'unknown error' (1): call=134, status=Timed Out, exitreason='non<br>
e',<br>
last-rc-change='Tue Aug 30 16:55:30 2016', queued=0ms, exec=20001ms<br>
* res_exportfs_root_monitor_30000 on nfsha2 'not running' (7): call=126, status=complete, exitreason='no<br>
ne',<br>
last-rc-change='Tue Aug 30 16:55:48 2016', queued=0ms, exec=0ms<br>
<br>
This of course blocks it, because the IP and the NFS exports are down. It doesn't even recognize that the other node is down. I am then forced to do "crm_resource -P" to get it back to a working state.<br>
<br>
Even when unplugging the master, and booting it up again, trying to get it back in the cluster executing "service pacemaker start" on the node that was unplugged will sometimes just cause the exportfs_root resource on the slave to fail (but the service is still
up):<br>
<br>
Master/Slave Set: ms_drbd_export [res_drbd_export]<br>
Masters: [ nfsha1 ]<br>
Slaves: [ nfsha2 ]<br>
Resource Group: rg_export<br>
res_fs (ocf::heartbeat:Filesystem): Started nfsha1<br>
res_exportfs_export1 (ocf::heartbeat:exportfs): Started nfsha1<br>
res_ip (ocf::heartbeat:IPaddr2): Started nfsha1<br>
Clone Set: cl_nfsserver [res_nfsserver]<br>
Started: [ nfsha1 nfsha2 ]<br>
Clone Set: cl_exportfs_root [res_exportfs_root]<br>
Started: [ nfsha1 nfsha2 ]<br>
<br>
Migration Summary:<br>
* Node nfsha2:<br>
res_exportfs_root: migration-threshold=1000000 fail-count=1 last-failure='Tue Aug 30 17:18:17 2016'<br>
* Node nfsha1:<br>
<br>
Failed Actions:<br>
* res_exportfs_root_monitor_30000 on nfsha2 'not running' (7): call=34, status=complete, exitreason='non<br>
e',<br>
last-rc-change='Tue Aug 30 17:18:17 2016', queued=0ms, exec=33ms<br>
<br>
BTW I notice that the node attributes are changed: <br>
<br>
Node Attributes:<br>
* Node nfsha1:<br>
+ master-res_drbd_export : 10000<br>
* Node nfsha2:<br>
+ master-res_drbd_export : 1000<br>
<br>
Usually both would have the same weight (10000), so running "crm_resource -P" restores that.<br>
<br>
Some other times it will instead cause a service disruption: <br>
<br>
Online: [ nfsha1 nfsha2 ]<br>
<br>
Master/Slave Set: ms_drbd_export [res_drbd_export]<br>
Masters: [ nfsha2 ]<br>
Slaves: [ nfsha1 ]<br>
Resource Group: rg_export<br>
res_fs (ocf::heartbeat:Filesystem): Started nfsha2<br>
res_exportfs_export1 (ocf::heartbeat:exportfs): FAILED (unmanaged)[ nfsha2 nfsha1 ]<br>
res_ip (ocf::heartbeat:IPaddr2): Stopped<br>
Clone Set: cl_nfsserver [res_nfsserver]<br>
Started: [ nfsha1 nfsha2 ]<br>
Clone Set: cl_exportfs_root [res_exportfs_root]<br>
Started: [ nfsha1 nfsha2]<br>
<br>
Migration Summary:<br>
* Node nfsha2:<br>
res_exportfs_export1: migration-threshold=1000000 fail-count=1000000 last-failure='Tue Aug 30 17:31:01 2016'<br>
* Node nfsha1:<br>
res_exportfs_export1: migration-threshold=1000000 fail-count=1000000 last-failure='Tue Aug 30 17:31:01 2016'<br>
res_exportfs_root: migration-threshold=1000000 fail-count=1 last-failure='Tue Aug 30 17:31:11 2016'<br>
<br>
Failed Actions:<br>
* res_exportfs_export1_stop_0 on nfsha2 'unknown error' (1): call=86, status=Timed Out, exitreason='none<br>
',<br>
last-rc-change='Tue Aug 30 17:30:41 2016', queued=0ms, exec=20002ms<br>
* res_exportfs_export1_stop_0 on nfsha1 'unknown error' (1): call=32, status=Timed Out, exitreason='none<br>
',<br>
last-rc-change='Tue Aug 30 17:30:41 2016', queued=0ms, exec=20002ms<br>
* res_exportfs_root_monitor_30000 on nfsha1 'not running' (7): call=29, status=complete, exitreason='non<br>
e',<br>
last-rc-change='Tue Aug 30 17:31:11 2016', queued=0ms, exec=0ms<br>
<br>
Then executing "crm_resource -P" brings it back to life, but if that command is not executed the cluster remains blocked until after around 10 mins when it sometimes gets magically back (like an auto execution of crm_resource -P).<br>
<br>
In case it helps, the CRM configuration is this one: <br>
<br>
node 1: nfsha1<br>
node 2: nfsha2 \<br>
attributes standby=off<br>
primitive res_drbd_export ocf:linbit:drbd \<br>
params drbd_resource=export<br>
primitive res_exportfs_export1 exportfs \<br>
params fsid=1 directory="/mnt/export/export1" options="rw,root_squash,mountpoint" clientspec="*.0/255.255.255.0" wait_for_leasetime_on_stop=true \<br>
op monitor interval=30s \<br>
meta target-role=Started<br>
primitive res_exportfs_root exportfs \<br>
params fsid=0 directory="/mnt/export" options="rw,crossmnt" clientspec="*.0/255.255.255.0" \<br>
op monitor interval=30s \<br>
meta target-role=Started<br>
primitive res_fs Filesystem \<br>
params device="/dev/drbd0" directory="/mnt/export" fstype=ext3 \<br>
meta target-role=Started<br>
primitive res_ip IPaddr2 \<br>
params ip=*.46 cidr_netmask=24 nic=eno1<br>
primitive res_nfsserver systemd:nfs-server \<br>
op monitor interval=30s<br>
group rg_export res_fs res_exportfs_export1 res_ip<br>
ms ms_drbd_export res_drbd_export \<br>
meta notify=true master-max=1 master-node-max=1 clone-max=2 clone-node-max=1<br>
clone cl_exportfs_root res_exportfs_root<br>
clone cl_nfsserver res_nfsserver<br>
colocation c_export_on_drbd inf: rg_export ms_drbd_export:Master<br>
colocation c_nfs_on_root inf: rg_export cl_exportfs_root<br>
order o_drbd_before_nfs inf: ms_drbd_export:promote rg_export:start<br>
order o_root_before_nfs inf: cl_exportfs_root rg_export:start<br>
property cib-bootstrap-options: \<br>
maintenance-mode=false \<br>
stonith-enabled=false \<br>
no-quorum-policy=ignore \<br>
have-watchdog=false \<br>
dc-version=1.1.13-10.el7_2.4-44eb2dd \<br>
cluster-infrastructure=corosync \<br>
cluster-name=nfsha<br>
<br>
And the corosync.conf:<br>
<br>
totem {<br>
version: 2<br>
# Corosync itself works without a cluster name, but DLM needs one.<br>
# The cluster name is also written into the VG metadata of newly<br>
# created shared LVM volume groups, if lvmlockd uses DLM locking.<br>
# It is also used for computing mcastaddr, unless overridden below.<br>
cluster_name: nfsha<br>
# How long before declaring a token lost (ms)<br>
token: 3000<br>
# How many token retransmits before forming a new configuration<br>
token_retransmits_before_loss_const: 10<br>
# Limit generated nodeids to 31-bits (positive signed integers)<br>
clear_node_high_bit: yes<br>
# crypto_cipher and crypto_hash: Used for mutual node authentication.<br>
# If you choose to enable this, then do remember to create a shared<br>
# secret with "corosync-keygen".<br>
# enabling crypto_cipher, requires also enabling of crypto_hash.<br>
# crypto_cipher and crypto_hash should be used instead of deprecated<br>
# secauth parameter.<br>
# Valid values for crypto_cipher are none (no encryption), aes256, aes192,<br>
# aes128 and 3des. Enabling crypto_cipher, requires also enabling of<br>
# crypto_hash.<br>
crypto_cipher: none<br>
# Valid values for crypto_hash are none (no authentication), md5, sha1,<br>
# sha256, sha384 and sha512.<br>
crypto_hash: none<br>
# Optionally assign a fixed node id (integer)<br>
# nodeid: 1234<br>
transport: udpu<br>
}<br>
nodelist {<br>
node {<br>
ring0_addr: *.50<br>
nodeid: 1<br>
}<br>
node {<br>
ring0_addr:*.51<br>
nodeid: 2<br>
}<br>
}<br>
logging {<br>
to_syslog: yes<br>
}<br>
<br>
quorum {<br>
# Enable and configure quorum subsystem (default: off)<br>
# see also corosync.conf.5 and votequorum.5<br>
provider: corosync_votequorum<br>
expected_votes: 2<br>
}<br>
<br>
So as you can imagine I am really puzzled about all this and would certainly welcome any help about what might be wrong with the current configuration.<br>
<br>
Thank you very much, kind regards<br>
<br>
Pablo<br>
<br>
</div>
</body>
</html>