[ClusterLabs] Active-Active NFS cluster failover test - system hangs (VirtualBox)

ArekW arkaduis at gmail.com
Mon Jul 10 10:59:54 UTC 2017


Hi,
I've created 2-node active-active HA Cluster with NFS resource. The
resources are active on both nodes. The Cluster passes failover test with
pcs standby command but does not work when "real" node shutdown occure.

Test scenario with cluster standby:
- start cluster
- mount nfs share on client1
- start copy file from client1 to nfs share
- during the copy put node1/node2 to standby mode (pcs cluster standby
nfsnode2)
- the copy continue
- unstandby node1/node2
- the copy continue and the storage re-sync (drbd)
- the copy finish with no errors

I can standby and unstandby the cluster many times and it works. The
problem begins when I do a "true" failover test by hard-shutting down one
of the nodes. Test results:
- start cluster
- mount nfs share on client1
- start copy file from client1 to nfs share
- during the copy shutdown node2 by stoping the node's virtual machine
(hard stop)
- the system hangs:

<Start copy file at client1>
# rsync -a --bwlimit=2000 /root/testfile.dat /mnt/nfsshare/

<everything works ok. There is temp file .testfile.dat.9780fH>

[root at nfsnode1 nfs]# ls -lah
razem 9,8M
drwxr-xr-x 2 root root 3,8K 07-10 11:07 .
drwxr-xr-x 4 root root 3,8K 07-10 08:20 ..
-rw-r--r-- 1 root root    9 07-10 08:20 client1.txt
-rw-r----- 1 root root    0 07-10 11:07 .rmtab
-rw------- 1 root root 9,8M 07-10 11:07 .testfile.dat.9780fH

[root at nfsnode1 nfs]# pcs status
Cluster name: nfscluster
Stack: corosync
Current DC: nfsnode2 (version 1.1.15-11.el7_3.5-e174ec8) - partition with
quorum
Last updated: Mon Jul 10 11:07:29 2017          Last change: Mon Jul 10
10:28:12 2017 by root via crm_attribute on nfsnode1

2 nodes and 15 resources configured

Online: [ nfsnode1 nfsnode2 ]

Full list of resources:

 Master/Slave Set: StorageClone [Storage]
     Masters: [ nfsnode1 nfsnode2 ]
 Clone Set: dlm-clone [dlm]
     Started: [ nfsnode1 nfsnode2 ]
 vbox-fencing   (stonith:fence_vbox):   Started nfsnode1
 Clone Set: ClusterIP-clone [ClusterIP] (unique)
     ClusterIP:0        (ocf::heartbeat:IPaddr2):       Started nfsnode2
     ClusterIP:1        (ocf::heartbeat:IPaddr2):       Started nfsnode1
 Clone Set: StorageFS-clone [StorageFS]
     Started: [ nfsnode1 nfsnode2 ]
 Clone Set: WebSite-clone [WebSite]
     Started: [ nfsnode1 nfsnode2 ]
 Clone Set: nfs-group-clone [nfs-group]
     Started: [ nfsnode1 nfsnode2 ]

<Hard poweroff vm machine: nfsnode2>

[root at nfsnode1 nfs]# pcs status
Cluster name: nfscluster
Stack: corosync
Current DC: nfsnode1 (version 1.1.15-11.el7_3.5-e174ec8) - partition with
quorum
Last updated: Mon Jul 10 11:07:43 2017          Last change: Mon Jul 10
10:28:12 2017 by root via crm_attribute on nfsnode1

2 nodes and 15 resources configured

Node nfsnode2: UNCLEAN (offline)
Online: [ nfsnode1 ]

Full list of resources:

 Master/Slave Set: StorageClone [Storage]
     Storage    (ocf::linbit:drbd):     Master nfsnode2 (UNCLEAN)
     Masters: [ nfsnode1 ]
 Clone Set: dlm-clone [dlm]
     dlm        (ocf::pacemaker:controld):      Started nfsnode2 (UNCLEAN)
     Started: [ nfsnode1 ]
 vbox-fencing   (stonith:fence_vbox):   Started nfsnode1
 Clone Set: ClusterIP-clone [ClusterIP] (unique)
     ClusterIP:0        (ocf::heartbeat:IPaddr2):       Started nfsnode2
(UNCLEAN)
     ClusterIP:1        (ocf::heartbeat:IPaddr2):       Started nfsnode1
 Clone Set: StorageFS-clone [StorageFS]
     StorageFS  (ocf::heartbeat:Filesystem):    Started nfsnode2 (UNCLEAN)
     Started: [ nfsnode1 ]
 Clone Set: WebSite-clone [WebSite]
     WebSite    (ocf::heartbeat:apache):        Started nfsnode2 (UNCLEAN)
     Started: [ nfsnode1 ]
 Clone Set: nfs-group-clone [nfs-group]
     Resource Group: nfs-group:1
         nfs    (ocf::heartbeat:nfsserver):     Started nfsnode2 (UNCLEAN)
         nfs-export     (ocf::heartbeat:exportfs):      Started nfsnode2
(UNCLEAN)
     Started: [ nfsnode1 ]

<ssh console hangs on client1>
[root at nfsnode1 nfs]# ls -lah
<nothing happen>

<drbd status is ok in this situation>
[root at nfsnode1 ~]# drbdadm status
storage role:Primary
  disk:UpToDate
  nfsnode2 connection:Connecting

<the nfs export is still active on node1>
[root at nfsnode1 ~]# exportfs
/mnt/drbd/nfs   10.0.2.0/255.255.255.0

<After ssh to client1 the nfs mount is not accessible>
login as: root
root at 127.0.0.1's password:
Last login: Mon Jul 10 07:48:17 2017 from 10.0.2.2
# cd /mnt/
# ls
<console hangs>

# mount
10.0.2.7:/ on /mnt/nfsshare type nfs4
(rw,relatime,vers=4.0,rsize=131072,wsize=131072,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=10.0.2.20,local_lock=none,addr=10.0.2.7)

<Power on vm machine nfsnode2>
<After nfsnode2 boot, console an nfsnode1 start respond but coping is not
proceeding>
<The temp file is visible but not active>
[root at nfsnode1 ~]# ls -lah
razem 9,8M
drwxr-xr-x 2 root root 3,8K 07-10 11:07 .
drwxr-xr-x 4 root root 3,8K 07-10 08:20 ..
-rw-r--r-- 1 root root    9 07-10 08:20 client1.txt
-rw-r----- 1 root root    0 07-10 11:16 .rmtab
-rw------- 1 root root 9,8M 07-10 11:07 .testfile.dat.9780fH

<Coping at client1 hangs>

<Cluster status:>
[root at nfsnode1 ~]# pcs status
Cluster name: nfscluster
Stack: corosync
Current DC: nfsnode1 (version 1.1.15-11.el7_3.5-e174ec8) - partition with
quorum
Last updated: Mon Jul 10 11:17:19 2017          Last change: Mon Jul 10
10:28:12 2017 by root via crm_attribute on nfsnode1

2 nodes and 15 resources configured

Online: [ nfsnode1 nfsnode2 ]

Full list of resources:

 Master/Slave Set: StorageClone [Storage]
     Masters: [ nfsnode1 ]
     Stopped: [ nfsnode2 ]
 Clone Set: dlm-clone [dlm]
     Started: [ nfsnode1 nfsnode2 ]
 vbox-fencing   (stonith:fence_vbox):   Started nfsnode1
 Clone Set: ClusterIP-clone [ClusterIP] (unique)
     ClusterIP:0        (ocf::heartbeat:IPaddr2):       Stopped
     ClusterIP:1        (ocf::heartbeat:IPaddr2):       Started nfsnode1
 Clone Set: StorageFS-clone [StorageFS]
     Started: [ nfsnode1 ]
     Stopped: [ nfsnode2 ]
 Clone Set: WebSite-clone [WebSite]
     Started: [ nfsnode1 ]
     Stopped: [ nfsnode2 ]
 Clone Set: nfs-group-clone [nfs-group]
     Resource Group: nfs-group:0
         nfs    (ocf::heartbeat:nfsserver):     Started nfsnode1
         nfs-export     (ocf::heartbeat:exportfs):      FAILED nfsnode1
     Stopped: [ nfsnode2 ]

Failed Actions:
* nfs-export_monitor_30000 on nfsnode1 'unknown error' (1): call=61,
status=Timed Out, exitreason='none',
    last-rc-change='Mon Jul 10 11:11:50 2017', queued=0ms, exec=0ms
* vbox-fencing_monitor_60000 on nfsnode1 'unknown error' (1): call=22,
status=Error, exitreason='none',
    last-rc-change='Mon Jul 10 11:06:41 2017', queued=0ms, exec=11988ms

<Try to cleanup>

# pcs resource cleanup
# pcs status
Cluster name: nfscluster
Stack: corosync
Current DC: nfsnode1 (version 1.1.15-11.el7_3.5-e174ec8) - partition with
quorum
Last updated: Mon Jul 10 11:20:38 2017          Last change: Mon Jul 10
10:28:12 2017 by root via crm_attribute on nfsnode1

2 nodes and 15 resources configured

Online: [ nfsnode1 nfsnode2 ]

Full list of resources:

 Master/Slave Set: StorageClone [Storage]
     Masters: [ nfsnode1 ]
     Stopped: [ nfsnode2 ]
 Clone Set: dlm-clone [dlm]
     Started: [ nfsnode1 nfsnode2 ]
 vbox-fencing   (stonith:fence_vbox):   Stopped
 Clone Set: ClusterIP-clone [ClusterIP] (unique)
     ClusterIP:0        (ocf::heartbeat:IPaddr2):       Stopped
     ClusterIP:1        (ocf::heartbeat:IPaddr2):       Stopped
 Clone Set: StorageFS-clone [StorageFS]
     Stopped: [ nfsnode1 nfsnode2 ]
 Clone Set: WebSite-clone [WebSite]
     Stopped: [ nfsnode1 nfsnode2 ]
 Clone Set: nfs-group-clone [nfs-group]
     Stopped: [ nfsnode1 nfsnode2 ]

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

<Reboot of both nfsnode1 and nfsnode2>
<After reboot:>

# pcs status
Cluster name: nfscluster
Stack: corosync
Current DC: nfsnode2 (version 1.1.15-11.el7_3.5-e174ec8) - partition with
quorum
Last updated: Mon Jul 10 11:24:10 2017          Last change: Mon Jul 10
10:28:12 2017 by root via crm_attribute on nfsnode1

2 nodes and 15 resources configured

Online: [ nfsnode1 nfsnode2 ]

Full list of resources:

 Master/Slave Set: StorageClone [Storage]
     Slaves: [ nfsnode2 ]
     Stopped: [ nfsnode1 ]
 Clone Set: dlm-clone [dlm]
     Started: [ nfsnode1 nfsnode2 ]
 vbox-fencing   (stonith:fence_vbox):   Stopped
 Clone Set: ClusterIP-clone [ClusterIP] (unique)
     ClusterIP:0        (ocf::heartbeat:IPaddr2):       Stopped
     ClusterIP:1        (ocf::heartbeat:IPaddr2):       Stopped
 Clone Set: StorageFS-clone [StorageFS]
     Stopped: [ nfsnode1 nfsnode2 ]
 Clone Set: WebSite-clone [WebSite]
     Stopped: [ nfsnode1 nfsnode2 ]
 Clone Set: nfs-group-clone [nfs-group]
     Stopped: [ nfsnode1 nfsnode2 ]

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

<Eventually the cluster was recovered after:>
pcs cluster stop --all
<Solve drbd split-brain>
pcs cluster start --all

The client1 could not be rebooted with 'reboot' due to mount hung (as I
preasume). It has to be rebooted hard-way by virtualbox hypervisor.
What's wrong with this configuration? I can send CIB configuration if
necessary.

---------------
Full cluster configuration (working state):

# pcs status --full
Cluster name: nfscluster
Stack: corosync
Current DC: nfsnode1 (1) (version 1.1.15-11.el7_3.5-e174ec8) - partition
with quorum
Last updated: Mon Jul 10 12:44:03 2017          Last change: Mon Jul 10
11:37:13 2017 by root via crm_attribute on nfsnode1

2 nodes and 15 resources configured

Online: [ nfsnode1 (1) nfsnode2 (2) ]

Full list of resources:

 Master/Slave Set: StorageClone [Storage]
     Storage    (ocf::linbit:drbd):     Master nfsnode1
     Storage    (ocf::linbit:drbd):     Master nfsnode2
     Masters: [ nfsnode1 nfsnode2 ]
 Clone Set: dlm-clone [dlm]
     dlm        (ocf::pacemaker:controld):      Started nfsnode1
     dlm        (ocf::pacemaker:controld):      Started nfsnode2
     Started: [ nfsnode1 nfsnode2 ]
 vbox-fencing   (stonith:fence_vbox):   Started nfsnode1
 Clone Set: ClusterIP-clone [ClusterIP] (unique)
     ClusterIP:0        (ocf::heartbeat:IPaddr2):       Started nfsnode2
     ClusterIP:1        (ocf::heartbeat:IPaddr2):       Started nfsnode1
 Clone Set: StorageFS-clone [StorageFS]
     StorageFS  (ocf::heartbeat:Filesystem):    Started nfsnode1
     StorageFS  (ocf::heartbeat:Filesystem):    Started nfsnode2
     Started: [ nfsnode1 nfsnode2 ]
 Clone Set: WebSite-clone [WebSite]
     WebSite    (ocf::heartbeat:apache):        Started nfsnode1
     WebSite    (ocf::heartbeat:apache):        Started nfsnode2
     Started: [ nfsnode1 nfsnode2 ]
 Clone Set: nfs-group-clone [nfs-group]
     Resource Group: nfs-group:0
         nfs    (ocf::heartbeat:nfsserver):     Started nfsnode1
         nfs-export     (ocf::heartbeat:exportfs):      Started nfsnode1
     Resource Group: nfs-group:1
         nfs    (ocf::heartbeat:nfsserver):     Started nfsnode2
         nfs-export     (ocf::heartbeat:exportfs):      Started nfsnode2
     Started: [ nfsnode1 nfsnode2 ]

Node Attributes:
* Node nfsnode1 (1):
    + master-Storage                    : 10000
* Node nfsnode2 (2):
    + master-Storage                    : 10000

Migration Summary:
* Node nfsnode1 (1):
* Node nfsnode2 (2):

PCSD Status:
  nfsnode1: Online
  nfsnode2: Online

Daemon Status:
  corosync: active/enabled
  pacemaker: active/enabled
  pcsd: active/enabled

]# pcs resource --full
 Master: StorageClone
  Meta Attrs: master-node-max=1 clone-max=2 notify=true master-max=2
clone-node-max=1
  Resource: Storage (class=ocf provider=linbit type=drbd)
   Attributes: drbd_resource=storage
   Operations: start interval=0s timeout=240 (Storage-start-interval-0s)
               promote interval=0s timeout=90 (Storage-promote-interval-0s)
               demote interval=0s timeout=90 (Storage-demote-interval-0s)
               stop interval=0s timeout=100 (Storage-stop-interval-0s)
               monitor interval=60s (Storage-monitor-interval-60s)
 Clone: dlm-clone
  Meta Attrs: clone-max=2 clone-node-max=1
  Resource: dlm (class=ocf provider=pacemaker type=controld)
   Operations: start interval=0s timeout=90 (dlm-start-interval-0s)
               stop interval=0s timeout=100 (dlm-stop-interval-0s)
               monitor interval=60s (dlm-monitor-interval-60s)
 Clone: ClusterIP-clone
  Meta Attrs: clona-node-max=2 clone-max=2 globally-unique=true
clone-node-max=2
  Resource: ClusterIP (class=ocf provider=heartbeat type=IPaddr2)
   Attributes: ip=10.0.2.7 cidr_netmask=32 clusterip_hash=sourceip
   Meta Attrs: resource-stickiness=0
   Operations: start interval=0s timeout=20s (ClusterIP-start-interval-0s)
               stop interval=0s timeout=20s (ClusterIP-stop-interval-0s)
               monitor interval=5s (ClusterIP-monitor-interval-5s)
 Clone: StorageFS-clone
  Resource: StorageFS (class=ocf provider=heartbeat type=Filesystem)
   Attributes: device=/dev/drbd1 directory=/mnt/drbd fstype=gfs2
   Operations: start interval=0s timeout=60 (StorageFS-start-interval-0s)
               stop interval=0s timeout=60 (StorageFS-stop-interval-0s)
               monitor interval=20 timeout=40
(StorageFS-monitor-interval-20)
 Clone: WebSite-clone
  Resource: WebSite (class=ocf provider=heartbeat type=apache)
   Attributes: configfile=/etc/httpd/conf/httpd.conf statusurl=
http://localhost/server-status
   Operations: start interval=0s timeout=40s (WebSite-start-interval-0s)
               stop interval=0s timeout=60s (WebSite-stop-interval-0s)
               monitor interval=1min (WebSite-monitor-interval-1min)
 Clone: nfs-group-clone
  Meta Attrs: interleave=true
  Group: nfs-group
   Resource: nfs (class=ocf provider=heartbeat type=nfsserver)
    Attributes: nfs_ip=10.0.2.7 nfs_no_notify=true
    Operations: start interval=0s timeout=40 (nfs-start-interval-0s)
                stop interval=0s timeout=20s (nfs-stop-interval-0s)
                monitor interval=30s (nfs-monitor-interval-30s)
   Resource: nfs-export (class=ocf provider=heartbeat type=exportfs)
    Attributes: clientspec=10.0.2.0/255.255.255.0
options=rw,sync,no_root_squash directory=/mnt/drbd/nfs fsid=0
    Operations: start interval=0s timeout=40 (nfs-export-start-interval-0s)
                stop interval=0s timeout=120 (nfs-export-stop-interval-0s)
                monitor interval=30s (nfs-export-monitor-interval-30s)

# pcs constraint --full
Location Constraints:
Ordering Constraints:
  start ClusterIP-clone then start WebSite-clone (kind:Mandatory)
(id:order-ClusterIP-WebSite-mandatory)
  promote StorageClone then start StorageFS-clone (kind:Mandatory)
(id:order-StorageClone-StorageFS-mandatory)
  start StorageFS-clone then start WebSite-clone (kind:Mandatory)
(id:order-StorageFS-WebSite-mandatory)
  start dlm-clone then start StorageFS-clone (kind:Mandatory)
(id:order-dlm-clone-StorageFS-mandatory)
  start StorageFS-clone then start nfs-group-clone (kind:Mandatory)
(id:order-StorageFS-clone-nfs-group-clone-mandatory)
Colocation Constraints:
  WebSite-clone with ClusterIP-clone (score:INFINITY)
(id:colocation-WebSite-ClusterIP-INFINITY)
  StorageFS-clone with StorageClone (score:INFINITY) (with-rsc-role:Master)
(id:colocation-StorageFS-StorageClone-INFINITY)
  WebSite-clone with StorageFS-clone (score:INFINITY)
(id:colocation-WebSite-StorageFS-INFINITY)
  StorageFS-clone with dlm-clone (score:INFINITY)
(id:colocation-StorageFS-dlm-clone-INFINITY)
  StorageFS-clone with nfs-group-clone (score:INFINITY)
(id:colocation-StorageFS-clone-nfs-group-clone-INFINITY)
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20170710/5769d3d0/attachment-0002.html>


More information about the Users mailing list