[ClusterLabs] epic fail
Dimitri Maziuk
dmaziuk at bmrb.wisc.edu
Mon Jul 24 13:24:08 EDT 2017
OK, how about this:
> Jul 22 14:03:41 zebrafish nfsserver(server_nfs)[6398]: INFO: Status: rpcbind
> Jul 22 14:03:41 zebrafish nfsserver(server_nfs)[6398]: INFO: Status: nfs-mountd
> Jul 22 14:03:41 zebrafish nfsserver(server_nfs)[6398]: INFO: Status: nfs-idmapd
> Jul 22 14:03:41 zebrafish nfsserver(server_nfs)[6398]: INFO: Status: rpc-statd
...
> Jul 22 14:03:45 zebrafish crmd[1078]: notice: Transition aborted by status-1-master-drbd_storage doing create master-drbd_storage=1000: Transient attribute change
> Jul 22 14:03:45 zebrafish crmd[1078]: notice: Initiating notify operation drbd_storage_post_notify_start_0 locally on zebrafish
> Jul 22 14:03:45 zebrafish crmd[1078]: notice: Initiating notify operation drbd_storage:1_post_notify_start_0 on lionfish
> Jul 22 14:03:45 zebrafish crmd[1078]: notice: Result of notify operation for drbd_storage on zebrafish: 0 (ok)
> Jul 22 14:03:45 zebrafish crmd[1078]: notice: Transition 1 (Complete=21, Pending=0, Fired=0, Skipped=1, Incomplete=1, Source=/var/lib/pacemaker/pengine/pe-input-255.bz2): Stopped
> Jul 22 14:03:45 zebrafish pengine[1077]: notice: On loss of CCM Quorum: Ignore
> Jul 22 14:03:45 zebrafish pengine[1077]: notice: Demote drbd_storage:0#011(Master -> Slave zebrafish)
> Jul 22 14:03:45 zebrafish pengine[1077]: notice: Promote drbd_storage:1#011(Slave -> Master lionfish)
> Jul 22 14:03:45 zebrafish pengine[1077]: notice: Move drbd_filesystem#011(Started zebrafish -> lionfish)
> Jul 22 14:03:45 zebrafish pengine[1077]: notice: Move symlink_home#011(Started zebrafish -> lionfish)
> Jul 22 14:03:45 zebrafish pengine[1077]: notice: Move floating_ip#011(Started zebrafish -> lionfish)
> Jul 22 14:03:45 zebrafish pengine[1077]: notice: Move server_nfs#011(Started zebrafish -> lionfish)
> Jul 22 14:03:45 zebrafish pengine[1077]: notice: Move nfsshare_home#011(Started zebrafish -> lionfish)
> Jul 22 14:03:45 zebrafish pengine[1077]: notice: Move nfs_notify#011(Started zebrafish -> lionfish)
> Jul 22 14:03:45 zebrafish pengine[1077]: notice: Move symlink_etc_pki#011(Started zebrafish -> lionfish)
> Jul 22 14:03:45 zebrafish pengine[1077]: notice: Move symlink_etc_dovecot#011(Started zebrafish -> lionfish)
> Jul 22 14:03:45 zebrafish pengine[1077]: notice: Move symlink_var_dovecot#011(Started zebrafish -> lionfish)
> Jul 22 14:03:45 zebrafish pengine[1077]: notice: Move server_dovecot#011(Started zebrafish -> lionfish)
> Jul 22 14:03:45 zebrafish pengine[1077]: notice: Calculated transition 2, saving inputs in /var/lib/pacemaker/pengine/pe-input-256.bz2
> Jul 22 14:03:45 zebrafish crmd[1078]: notice: Initiating cancel operation drbd_storage_monitor_29000 locally on zebrafish
> Jul 22 14:03:45 zebrafish crmd[1078]: notice: Initiating stop operation nfs_notify_stop_0 locally on zebrafish
> Jul 22 14:03:45 zebrafish crmd[1078]: notice: Initiating stop operation server_dovecot_stop_0 locally on zebrafish
> Jul 22 14:03:45 zebrafish crmd[1078]: notice: Initiating notify operation drbd_storage_pre_notify_demote_0 locally on zebrafish
> Jul 22 14:03:45 zebrafish systemd: Reloading.
> Jul 22 14:03:45 zebrafish crmd[1078]: notice: Initiating notify operation drbd_storage_pre_notify_demote_0 on lionfish
> Jul 22 14:03:46 zebrafish nfsnotify(nfs_notify)[6511]: INFO: previous sm-notify processes terminated before stop action.
> Jul 22 14:03:46 zebrafish crmd[1078]: notice: Result of stop operation for nfs_notify on zebrafish: 0 (ok)
> Jul 22 14:03:46 zebrafish crmd[1078]: notice: Initiating stop operation nfsshare_home_stop_0 locally on zebrafish
> Jul 22 14:03:46 zebrafish crmd[1078]: notice: Result of notify operation for drbd_storage on zebrafish: 0 (ok)
> Jul 22 14:03:46 zebrafish systemd: Stopping Dovecot IMAP/POP3 email server...
> Jul 22 14:03:46 zebrafish systemd: Stopped Dovecot IMAP/POP3 email server.
> Jul 22 14:03:46 zebrafish exportfs(nfsshare_home)[6558]: INFO: Un-exporting file system ...
> Jul 22 14:03:46 zebrafish kernel: drbd raid: Handshake successful: Agreed network protocol version 101
> Jul 22 14:03:46 zebrafish kernel: drbd raid: Feature flags enabled on protocol level: 0x7 TRIM THIN_RESYNC WRITE_SAME.
> Jul 22 14:03:46 zebrafish kernel: drbd raid: conn( WFConnection -> WFReportParams )
> Jul 22 14:03:46 zebrafish kernel: drbd raid: Starting ack_recv thread (from drbd_r_raid [1626])
> Jul 22 14:03:46 zebrafish exportfs(nfsshare_home)[6558]: INFO: unexporting 144.92.167.128/25:/raid/home
> Jul 22 14:03:46 zebrafish exportfs(nfsshare_home)[6558]: INFO: Un-exported file system
> Jul 22 14:03:46 zebrafish crmd[1078]: notice: Result of stop operation for nfsshare_home on zebrafish: 0 (ok)
> Jul 22 14:03:46 zebrafish kernel: block drbd0: drbd_sync_handshake:
> Jul 22 14:03:46 zebrafish kernel: block drbd0: self F5C467DDAA8CAD23:BEA8E935E70AC8E8:30542C9D8D243C3C:30532C9D8D243C3D bits:9327 flags:0
> Jul 22 14:03:46 zebrafish kernel: block drbd0: peer BEA8E935E70AC8E8:0000000000000000:30542C9D8D243C3D:30532C9D8D243C3D bits:1266688 flags:2
> Jul 22 14:03:46 zebrafish kernel: block drbd0: uuid_compare()=1 by rule 70
> Jul 22 14:03:46 zebrafish crmd[1078]: notice: Initiating stop operation symlink_home_stop_0 locally on zebrafish
> Jul 22 14:03:46 zebrafish crmd[1078]: notice: Initiating stop operation server_nfs_stop_0 locally on zebrafish
> Jul 22 14:03:46 zebrafish kernel: block drbd0: peer( Unknown -> Secondary ) conn( WFReportParams -> WFBitMapS ) pdsk( DUnknown -> Consistent )
> Jul 22 14:03:46 zebrafish symlink(symlink_home)[6613]: INFO: removed '/home'
> Jul 22 14:03:46 zebrafish crmd[1078]: notice: Result of stop operation for symlin> k_home on zebrafish: 0 (ok)
> Jul 22 14:03:46 zebrafish kernel: block drbd0: drbd_sync_handshake:
> Jul 22 14:03:46 zebrafish kernel: block drbd0: self F5C467DDAA8CAD23:BEA8E935E70AC8E8:30542C9D8D243C3C:30532C9D8D243C3D bits:9327 flags:0
> Jul 22 14:03:46 zebrafish kernel: block drbd0: peer BEA8E935E70AC8E8:0000000000000000:30542C9D8D243C3D:30532C9D8D243C3D bits:1266688 flags:2
> Jul 22 14:03:46 zebrafish kernel: block drbd0: uuid_compare()=1 by rule 70
> Jul 22 14:03:46 zebrafish crmd[1078]: notice: Initiating stop operation symlink_home_stop_0 locally on zebrafish
> Jul 22 14:03:46 zebrafish crmd[1078]: notice: Initiating stop operation server_nfs_stop_0 locally on zebrafish
> Jul 22 14:03:46 zebrafish kernel: block drbd0: peer( Unknown -> Secondary ) conn( WFReportParams -> WFBitMapS ) pdsk( DUnknown -> Consistent )
> Jul 22 14:03:46 zebrafish symlink(symlink_home)[6613]: INFO: removed '/home'
> Jul 22 14:03:46 zebrafish crmd[1078]: notice: Result of stop operation for symlink_home on zebrafish: 0 (ok)
> Jul 22 14:03:46 zebrafish kernel: block drbd0: send bitmap stats [Bytes(packets)]: plain 0(0), RLE 1458(1), total 1458; compression: 100.0%
> Jul 22 14:03:46 zebrafish kernel: block drbd0: receive bitmap stats [Bytes(packets)]: plain 0(0), RLE 4700(2), total 4700; compression: 100.0%
> Jul 22 14:03:46 zebrafish kernel: block drbd0: helper command: /sbin/drbdadm before-resync-source minor-0
> Jul 22 14:03:46 zebrafish kernel: block drbd0: helper command: /sbin/drbdadm before-resync-source minor-0 exit code 0 (0x0)
> Jul 22 14:03:46 zebrafish kernel: block drbd0: conn( WFBitMapS -> SyncSource ) pdsk( Consistent -> Inconsistent )
> Jul 22 14:03:46 zebrafish kernel: block drbd0: Began resync as SyncSource (will sync 5084788 KB [1271197 bits set]).
> Jul 22 14:03:46 zebrafish kernel: block drbd0: updated sync UUID F5C467DDAA8CAD23:BEA9E935E70AC8E8:BEA8E935E70AC8E8:30542C9D8D243C3C
> Jul 22 14:03:46 zebrafish nfsserver(server_nfs)[6614]: INFO: Stopping NFS server ...
> Jul 22 14:03:46 zebrafish systemd: Stopping NFS server and services...
> Jul 22 14:03:46 zebrafish systemd: Stopped NFS server and services.
> Jul 22 14:03:46 zebrafish systemd: Stopping NFS Mount Daemon...
> Jul 22 14:03:46 zebrafish systemd: Stopping NFSv4 ID-name mapping service...
> Jul 22 14:03:46 zebrafish rpc.mountd[2655]: Caught signal 15, un-registering and exiting.
> Jul 22 14:03:46 zebrafish systemd: Stopped NFSv4 ID-name mapping service.
> Jul 22 14:03:46 zebrafish systemd: Stopped NFS Mount Daemon.
> Jul 22 14:03:46 zebrafish nfsserver(server_nfs)[6614]: INFO: Stop: threads
> Jul 22 14:03:46 zebrafish kernel: nfsd: last server has exited, flushing export cache
> Jul 22 14:03:46 zebrafish systemd: Stopping NFS status monitor for NFSv2/3 locking....
> Jul 22 14:03:46 zebrafish systemd: Stopped NFS status monitor for NFSv2/3 locking..
> Jul 22 14:03:46 zebrafish nfsserver(server_nfs)[6614]: INFO: Stop: rpc-statd
> Jul 22 14:03:46 zebrafish nfsserver(server_nfs)[6614]: INFO: Stop: nfs-idmapd
> Jul 22 14:03:46 zebrafish nfsserver(server_nfs)[6614]: INFO: Stop: nfs-mountd
> Jul 22 14:03:46 zebrafish systemd: Stopping RPC bind service...
> Jul 22 14:03:46 zebrafish systemd: Stopped RPC bind service.
> Jul 22 14:03:46 zebrafish nfsserver(server_nfs)[6614]: INFO: Stop: rpcbind
> Jul 22 14:03:46 zebrafish nfsserver(server_nfs)[6614]: INFO: Stop: rpc-gssd
> Jul 22 14:03:46 zebrafish nfsserver(server_nfs)[6614]: INFO: Stop: umount (1/10 attempts)
> Jul 22 14:03:47 zebrafish nfsserver(server_nfs)[6614]: INFO: NFS server stopped
> Jul 22 14:03:47 zebrafish crmd[1078]: notice: Result of stop operation for server_nfs on zebrafish: 0 (ok)
> Jul 22 14:03:47 zebrafish crmd[1078]: notice: Initiating stop operation floating_ip_stop_0 locally on zebrafish
> Jul 22 14:03:48 zebrafish crmd[1078]: notice: Result of stop operation for server_dovecot on zebrafish: 0 (ok)
> Jul 22 14:03:48 zebrafish crmd[1078]: notice: Initiating stop operation symlink_etc_pki_stop_0 locally on zebrafish
> Jul 22 14:03:48 zebrafish IPaddr2(floating_ip)[6769]: INFO: IP status = ok, IP_CIP=
> Jul 22 14:03:48 zebrafish crmd[1078]: notice: Initiating stop operation symlink_var_dovecot_stop_0 locally on zebrafish
> Jul 22 14:03:48 zebrafish crmd[1078]: notice: Result of stop operation for floating_ip on zebrafish: 0 (ok)
> Jul 22 14:03:48 zebrafish symlink(symlink_etc_pki)[6821]: INFO: removed '/etc/pki'
> Jul 22 14:03:48 zebrafish symlink(symlink_var_dovecot)[6822]: INFO: removed '/var/spool/dovecot'
> Jul 22 14:03:48 zebrafish crmd[1078]: notice: Result of stop operation for symlink_var_dovecot on zebrafish: 0 (ok)
> Jul 22 14:03:48 zebrafish crmd[1078]: notice: Initiating stop operation symlink_etc_dovecot_stop_0 locally on zebrafish
> Jul 22 14:03:48 zebrafish crmd[1078]: notice: Result of stop operation for symlink_etc_pki on zebrafish: 0 (ok)
> Jul 22 14:03:48 zebrafish symlink(symlink_etc_dovecot)[6863]: INFO: removed '/etc/dovecot'
> Jul 22 14:03:48 zebrafish crmd[1078]: notice: Result of stop operation for symlink_etc_dovecot on zebrafish: 0 (ok)
> Jul 22 14:03:48 zebrafish crmd[1078]: notice: Initiating stop operation drbd_filesystem_stop_0 locally on zebrafish
> Jul 22 14:03:48 zebrafish Filesystem(drbd_filesystem)[6886]: INFO: Running stop for /dev/drbd0 on /raid
> Jul 22 14:03:48 zebrafish Filesystem(drbd_filesystem)[6886]: INFO: Trying to unmount /raid
> Jul 22 14:03:48 zebrafish Filesystem(drbd_filesystem)[6886]: ERROR: Couldn't unmount /raid; trying cleanup with TERM
> Jul 22 14:03:48 zebrafish Filesystem(drbd_filesystem)[6886]: INFO: No processes on /raid were signalled. force_unmount is set to 'yes'
> Jul 22 14:03:49 zebrafish Filesystem(drbd_filesystem)[6886]: ERROR: Couldn't unmount /raid; trying cleanup with TERM
> Jul 22 14:03:49 zebrafish Filesystem(drbd_filesystem)[6886]: INFO: No processes on /raid were signalled. force_unmount is set to 'yes'
> Jul 22 14:03:50 zebrafish ntpd[596]: Deleting interface #8 enp2s0f0, 144.92.167.221#123, interface stats: received=0, sent=0, dropped=0, active_time=260 secs
> Jul 22 14:03:50 zebrafish Filesystem(drbd_filesystem)[6886]: ERROR: Couldn't unmount /raid; trying cleanup with TERM
> Jul 22 14:03:50 zebrafish Filesystem(drbd_filesystem)[6886]: INFO: No processes on /raid were signalled. force_unmount is set to 'yes'
> Jul 22 14:03:51 zebrafish Filesystem(drbd_filesystem)[6886]: ERROR: Couldn't unmount /raid; trying cleanup with KILL
> Jul 22 14:03:51 zebrafish Filesystem(drbd_filesystem)[6886]: INFO: No processes on /raid were signalled. force_unmount is set to 'yes'
> Jul 22 14:03:52 zebrafish Filesystem(drbd_filesystem)[6886]: ERROR: Couldn't unmount /raid; trying cleanup with KILL
> Jul 22 14:03:53 zebrafish Filesystem(drbd_filesystem)[6886]: INFO: No processes on /raid were signalled. force_unmount is set to 'yes'
> Jul 22 14:03:54 zebrafish Filesystem(drbd_filesystem)[6886]: ERROR: Couldn't unmount /raid; trying cleanup with KILL
> Jul 22 14:03:54 zebrafish Filesystem(drbd_filesystem)[6886]: INFO: No processes on /raid were signalled. force_unmount is set to 'yes'
> Jul 22 14:03:55 zebrafish Filesystem(drbd_filesystem)[6886]: ERROR: Couldn't unmount /raid, giving up!
> Jul 22 14:03:55 zebrafish lrmd[1075]: notice: drbd_filesystem_stop_0:6886:stderr [ umount: /raid: target is busy. ]
> Jul 22 14:03:55 zebrafish lrmd[1075]: notice: drbd_filesystem_stop_0:6886:stderr [ (In some cases useful info about processes that use ]
> Jul 22 14:03:55 zebrafish lrmd[1075]: notice: drbd_filesystem_stop_0:6886:stderr [ the device is found by lsof(8) or fuser(1)) ]
> Jul 22 14:03:55 zebrafish lrmd[1075]: notice: drbd_filesystem_stop_0:6886:stderr [ ocf-exit-reason:Couldn't unmount /raid; trying cleanup with TERM ]
> Jul 22 14:03:55 zebrafish lrmd[1075]: notice: drbd_filesystem_stop_0:6886:stderr [ umount: /raid: target is busy. ]
> Jul 22 14:03:55 zebrafish lrmd[1075]: notice: drbd_filesystem_stop_0:6886:stderr [ (In some cases useful info about processes that use ]
> Jul 22 14:03:55 zebrafish lrmd[1075]: notice: drbd_filesystem_stop_0:6886:stderr [ the device is found by lsof(8) or fuser(1)) ]
> Jul 22 14:03:55 zebrafish lrmd[1075]: notice: drbd_filesystem_stop_0:6886:stderr [ ocf-exit-reason:Couldn't unmount /raid; trying cleanup with TERM ]
> Jul 22 14:03:55 zebrafish lrmd[1075]: notice: drbd_filesystem_stop_0:6886:stderr [ umount: /raid: target is busy. ]
> Jul 22 14:03:55 zebrafish lrmd[1075]: notice: drbd_filesystem_stop_0:6886:stderr [ (In some cases useful info about processes that use ]
> Jul 22 14:03:55 zebrafish lrmd[1075]: notice: drbd_filesystem_stop_0:6886:stderr [ the device is found by lsof(8) or fuser(1)) ]
> Jul 22 14:03:55 zebrafish lrmd[1075]: notice: drbd_filesystem_stop_0:6886:stderr [ ocf-exit-reason:Couldn't unmount /raid; trying cleanup with TERM ]
> Jul 22 14:03:55 zebrafish lrmd[1075]: notice: drbd_filesystem_stop_0:6886:stderr [ umount: /raid: target is busy. ]
> Jul 22 14:03:55 zebrafish lrmd[1075]: notice: drbd_filesystem_stop_0:6886:stderr [ (In some cases useful info about processes that use ]
> Jul 22 14:03:55 zebrafish lrmd[1075]: notice: drbd_filesystem_stop_0:6886:stderr [ the device is found by lsof(8) or fuser(1)) ]
> Jul 22 14:03:55 zebrafish lrmd[1075]: notice: drbd_filesystem_stop_0:6886:stderr [ ocf-exit-reason:Couldn't unmount /raid; trying cleanup with KILL ]
> Jul 22 14:03:55 zebrafish lrmd[1075]: notice: drbd_filesystem_stop_0:6886:stderr [ umount: /raid: target is busy. ]
> Jul 22 14:03:55 zebrafish lrmd[1075]: notice: drbd_filesystem_stop_0:6886:stderr [ (In some cases useful info about processes that use ]
> Jul 22 14:03:55 zebrafish lrmd[1075]: notice: drbd_filesystem_stop_0:6886:stderr [ the device is found by lsof(8) or fuser(1)) ]
> Jul 22 14:03:55 zebrafish lrmd[1075]: notice: drbd_filesystem_stop_0:6886:stderr [ ocf-exit-reason:Couldn't unmount /raid; trying cleanup with KILL ]
> Jul 22 14:03:55 zebrafish lrmd[1075]: notice: drbd_filesystem_stop_0:6886:stderr [ umount: /raid: target is busy. ]
> Jul 22 14:03:55 zebrafish lrmd[1075]: notice: drbd_filesystem_stop_0:6886:stderr [ (In some cases useful info about processes that use ]
> Jul 22 14:03:55 zebrafish lrmd[1075]: notice: drbd_filesystem_stop_0:6886:stderr [ the device is found by lsof(8) or fuser(1)) ]
> Jul 22 14:03:55 zebrafish lrmd[1075]: notice: drbd_filesystem_stop_0:6886:stderr [ ocf-exit-reason:Couldn't unmount /raid; trying cleanup with KILL ]
> Jul 22 14:03:55 zebrafish lrmd[1075]: notice: drbd_filesystem_stop_0:6886:stderr [ ocf-exit-reason:Couldn't unmount /raid, giving up! ]
> Jul 22 14:03:55 zebrafish crmd[1078]: notice: Result of stop operation for drbd_filesystem on zebrafish: 1 (unknown error)
> Jul 22 14:03:55 zebrafish crmd[1078]: notice: zebrafish-drbd_filesystem_stop_0:101 [ umount: /raid: target is busy.\n (In some cases useful info about processes that use\n the device is found by lsof(8) or fuser(1))\nocf-exit-reason:Couldn't unmount /raid; trying cleanup with TERM\numount: /raid: target is busy.\n (In some cases useful info about processes that use\n the device is found by lsof(8) or fuser(1))\nocf-exit-reason:Couldn't unmount /raid; trying cleanup with TERM\numount: /raid: target is busy.\n
> Jul 22 14:03:55 zebrafish crmd[1078]: warning: Action 45 (drbd_filesystem_stop_0) on zebrafish failed (target: 0 vs. rc: 1): Error
> Jul 22 14:03:55 zebrafish crmd[1078]: notice: Transition aborted by operation drbd_filesystem_stop_0 'modify' on zebrafish: Event failed
> Jul 22 14:03:55 zebrafish crmd[1078]: warning: Action 45 (drbd_filesystem_stop_0) on zebrafish failed (target: 0 vs. rc: 1): Error
> Jul 22 14:03:55 zebrafish crmd[1078]: notice: Transition 2 (Complete=21, Pending=0, Fired=0, Skipped=0, Incomplete=43, Source=/var/lib/pacemaker/pengine/pe-input-256.bz2): Complete
> Jul 22 14:03:55 zebrafish pengine[1077]: notice: On loss of CCM Quorum: Ignore
> Jul 22 14:03:55 zebrafish pengine[1077]: warning: Processing failed op stop for drbd_filesystem on zebrafish: unknown error (1)
> Jul 22 14:03:55 zebrafish pengine[1077]: warning: Processing failed op stop for drbd_filesystem on zebrafish: unknown error (1)
> Jul 22 14:03:55 zebrafish pengine[1077]: notice: Demote drbd_storage:0#011(Master -> Slave zebrafish)
> Jul 22 14:03:55 zebrafish pengine[1077]: notice: Promote drbd_storage:1#011(Slave -> Master lionfish)
> Jul 22 14:03:55 zebrafish pengine[1077]: notice: Start floating_ip#011(lionfish - blocked)
> Jul 22 14:03:55 zebrafish pengine[1077]: notice: Start server_nfs#011(lionfish - blocked)
> Jul 22 14:03:55 zebrafish pengine[1077]: notice: Start nfsshare_home#011(lionfish - blocked)
> Jul 22 14:03:55 zebrafish pengine[1077]: notice: Start nfs_notify#011(lionfish - blocked)
> Jul 22 14:03:55 zebrafish pengine[1077]: notice: Start symlink_etc_pki#011(lionfish - blocked)
> Jul 22 14:03:55 zebrafish pengine[1077]: notice: Start symlink_etc_dovecot#011(lionfish - blocked)
> Jul 22 14:03:55 zebrafish pengine[1077]: notice: Start symlink_var_dovecot#011(lionfish - blocked)
> Jul 22 14:03:55 zebrafish pengine[1077]: notice: Start server_dovecot#011(lionfish - blocked)
> Jul 22 14:03:55 zebrafish pengine[1077]: notice: Calculated transition 3, saving inputs in /var/lib/pacemaker/pengine/pe-input-257.bz2
> Jul 22 14:03:55 zebrafish pengine[1077]: notice: On loss of CCM Quorum: Ignore
> Jul 22 14:03:55 zebrafish pengine[1077]: warning: Processing failed op stop for drbd_filesystem on zebrafish: unknown error (1)
> Jul 22 14:03:55 zebrafish pengine[1077]: warning: Processing failed op stop for drbd_filesystem on zebrafish: unknown error (1)
> Jul 22 14:03:55 zebrafish pengine[1077]: warning: Forcing drbd_filesystem away from zebrafish after 1000000 failures (max=1000000)
> Jul 22 14:03:55 zebrafish pengine[1077]: notice: Demote drbd_storage:0#011(Master -> Slave zebrafish)
> Jul 22 14:03:55 zebrafish pengine[1077]: notice: Promote drbd_storage:1#011(Slave -> Master lionfish)
> Jul 22 14:03:55 zebrafish pengine[1077]: notice: Start floating_ip#011(lionfish - blocked)
> Jul 22 14:03:55 zebrafish pengine[1077]: notice: Start server_nfs#011(lionfish - blocked)
> Jul 22 14:03:55 zebrafish pengine[1077]: notice: Start nfsshare_home#011(lionfish - blocked)
> Jul 22 14:03:55 zebrafish pengine[1077]: notice: Start nfs_notify#011(lionfish - blocked)
> Jul 22 14:03:55 zebrafish pengine[1077]: notice: Start symlink_etc_pki#011(lionfish - blocked)
> Jul 22 14:03:55 zebrafish pengine[1077]: notice: Start symlink_etc_dovecot#011(lionfish - blocked)
> Jul 22 14:03:55 zebrafish pengine[1077]: notice: Start symlink_var_dovecot#011(lionfish - blocked)
> Jul 22 14:03:55 zebrafish pengine[1077]: notice: Start server_dovecot#011(lionfish - blocked)
> Jul 22 14:03:55 zebrafish pengine[1077]: notice: Calculated transition 4, saving inputs in /var/lib/pacemaker/pengine/pe-input-258.bz2
> Jul 22 14:03:55 zebrafish crmd[1078]: notice: Initiating demote operation drbd_storage_demote_0 locally on zebrafish
> Jul 22 14:03:55 zebrafish crmd[1078]: notice: Initiating notify operation drbd_storage_pre_notify_demote_0 locally on zebrafish
> Jul 22 14:03:55 zebrafish crmd[1078]: notice: Initiating notify operation drbd_storage_pre_notify_demote_0 on lionfish
> Jul 22 14:03:56 zebrafish kernel: block drbd0: State change failed: Device is held open by someone
> Jul 22 14:03:56 zebrafish kernel: block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r----- }
> Jul 22 14:03:56 zebrafish kernel: block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r----- }
> Jul 22 14:03:56 zebrafish drbd(drbd_storage)[7093]: ERROR: raid: Called drbdadm -c /etc/drbd.conf secondary raid
> Jul 22 14:03:56 zebrafish drbd(drbd_storage)[7093]: ERROR: raid: Exit code 11
> Jul 22 14:03:56 zebrafish drbd(drbd_storage)[7093]: ERROR: raid: Command output:
> Jul 22 14:03:56 zebrafish kernel: block drbd0: State change failed: Device is held open by someone
> Jul 22 14:03:56 zebrafish kernel: block drbd0: state = { cs:SyncSource ro:Primary/Secondary ds:UpToDate/Inconsistent r----- }
> Jul 22 14:03:56 zebrafish kernel: block drbd0: wanted = { cs:SyncSource ro:Secondary/Secondary ds:UpToDate/Inconsistent r----- }
> Jul 22 14:03:56 zebrafish drbd(drbd_storage)[7093]: ERROR: raid: Called drbdadm -c /etc/drbd.conf secondary raid
> Jul 22 14:03:56 zebrafish drbd(drbd_storage)[7093]: ERROR: raid: Exit code 11
> Jul 22 14:03:56 zebrafish drbd(drbd_storage)[7093]: ERROR: raid: Command output:
...
this last bit: "kernel: block drbd 0"x3 and "drbd(drbd_storage)"x3 goes
on until the power-cycle.
--
Dimitri Maziuk
Programmer/sysadmin
BioMagResBank, UW-Madison -- http://www.bmrb.wisc.edu
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 190 bytes
Desc: OpenPGP digital signature
URL: <https://lists.clusterlabs.org/pipermail/users/attachments/20170724/138083d7/attachment-0003.sig>
More information about the Users
mailing list