[ClusterLabs] A stop job is running for pacemaker high availability cluster manager

[Ken Gaillot]

On 02/02/2017 03:06 PM, Oscar Segarra wrote:
> Hi Ken, 
> 
> I have checked the /var/log/cluster/corosync.log and there no
> information about why system hangs stopping... 
> 
> ¿Can you be more specific about what logs to check?
> 
> Thanks a lot.

There, and /var/log/messages sometimes has relevant messages from
non-cluster components.

You'd want to look for messages like "Caught 'Terminated' signal" and
"Shutting down", as well as resources being stopped ("_stop_0"), then
various "Disconnect" and "Stopping" messages as individual daemons exit.

> 2017-02-02 21:10 GMT+01:00 Ken Gaillot <kgaillot at redhat.com
> <mailto:kgaillot at redhat.com>>:
> 
>     On 02/02/2017 12:35 PM, Oscar Segarra wrote:
>     > Hi,
>     >
>     > I have a two node cluster... when I try to shutdown the physical host I
>     > get the following message in console: "a stop job is running for
>     > pacemaker high availability cluster manager" and never stops...
> 
>     That would be a message from systemd. You'll need to check the pacemaker
>     status and/or logs to see why pacemaker can't shut down.
> 
>     Without stonith enabled, pacemaker will be unable to recover if a
>     resource fails to stop. That could lead to a hang.
> 
>     > This is my configuration:
>     >
>     > [root at vdicnode01 ~]# pcs config
>     > Cluster Name: vdic-cluster
>     > Corosync Nodes:
>     >  vdicnode01-priv vdicnode02-priv
>     > Pacemaker Nodes:
>     >  vdicnode01-priv vdicnode02-priv
>     >
>     > Resources:
>     >  Resource: nfs-vdic-mgmt-vm-vip (class=ocf provider=heartbeat
>     type=IPaddr)
>     >   Attributes: ip=192.168.100.200 cidr_netmask=24
>     >   Operations: start interval=0s timeout=20s
>     > (nfs-vdic-mgmt-vm-vip-start-interval-0s)
>     >               stop interval=0s timeout=20s
>     > (nfs-vdic-mgmt-vm-vip-stop-interval-0s)
>     >               monitor interval=10s
>     > (nfs-vdic-mgmt-vm-vip-monitor-interval-10s)
>     >  Clone: nfs_setup-clone
>     >   Resource: nfs_setup (class=ocf provider=heartbeat type=ganesha_nfsd)
>     >    Attributes: ha_vol_mnt=/var/run/gluster/shared_storage
>     >    Operations: start interval=0s timeout=5s
>     (nfs_setup-start-interval-0s)
>     >                stop interval=0s timeout=5s
>     (nfs_setup-stop-interval-0s)
>     >                monitor interval=0 timeout=5s
>     (nfs_setup-monitor-interval-0)
>     >  Clone: nfs-mon-clone
>     >   Resource: nfs-mon (class=ocf provider=heartbeat type=ganesha_mon)
>     >    Operations: start interval=0s timeout=40s
>     (nfs-mon-start-interval-0s)
>     >                stop interval=0s timeout=40s (nfs-mon-stop-interval-0s)
>     >                monitor interval=10s timeout=10s
>     > (nfs-mon-monitor-interval-10s)
>     >  Clone: nfs-grace-clone
>     >   Meta Attrs: notify=true
>     >   Resource: nfs-grace (class=ocf provider=heartbeat
>     type=ganesha_grace)
>     >    Meta Attrs: notify=true
>     >    Operations: start interval=0s timeout=40s
>     (nfs-grace-start-interval-0s)
>     >                stop interval=0s timeout=40s
>     (nfs-grace-stop-interval-0s)
>     >                monitor interval=5s timeout=10s
>     > (nfs-grace-monitor-interval-5s)
>     >  Resource: vm-vdicone01 (class=ocf provider=heartbeat
>     type=VirtualDomain)
>     >   Attributes: hypervisor=qemu:///system
>     > config=/mnt/nfs-vdic-mgmt-vm/vdicone01.xml
>     > migration_network_suffix=tcp:// migration_transport=ssh
>     >   Meta Attrs: allow-migrate=true target-role=Stopped
>     >   Utilization: cpu=1 hv_memory=512
>     >   Operations: start interval=0s timeout=90
>     (vm-vdicone01-start-interval-0s)
>     >               stop interval=0s timeout=90
>     (vm-vdicone01-stop-interval-0s)
>     >               monitor interval=20s role=Stopped
>     > (vm-vdicone01-monitor-interval-20s)
>     >               monitor interval=30s (vm-vdicone01-monitor-interval-30s)
>     >  Resource: vm-vdicsunstone01 (class=ocf provider=heartbeat
>     > type=VirtualDomain)
>     >   Attributes: hypervisor=qemu:///system
>     > config=/mnt/nfs-vdic-mgmt-vm/vdicsunstone01.xml
>     > migration_network_suffix=tcp:// migration_transport=ssh
>     >   Meta Attrs: allow-migrate=true target-role=Stopped
>     >   Utilization: cpu=1 hv_memory=1024
>     >   Operations: start interval=0s timeout=90
>     > (vm-vdicsunstone01-start-interval-0s)
>     >               stop interval=0s timeout=90
>     > (vm-vdicsunstone01-stop-interval-0s)
>     >               monitor interval=20s role=Stopped
>     > (vm-vdicsunstone01-monitor-interval-20s)
>     >               monitor interval=30s
>     (vm-vdicsunstone01-monitor-interval-30s)
>     >  Resource: vm-vdicdb01 (class=ocf provider=heartbeat
>     type=VirtualDomain)
>     >   Attributes: hypervisor=qemu:///system
>     > config=/mnt/nfs-vdic-mgmt-vm/vdicdb01.xml
>     > migration_network_suffix=tcp:// migration_transport=ssh
>     >   Meta Attrs: allow-migrate=true target-role=Stopped
>     >   Utilization: cpu=1 hv_memory=512
>     >   Operations: start interval=0s timeout=90
>     (vm-vdicdb01-start-interval-0s)
>     >               stop interval=0s timeout=90
>     (vm-vdicdb01-stop-interval-0s)
>     >               monitor interval=20s role=Stopped
>     > (vm-vdicdb01-monitor-interval-20s)
>     >               monitor interval=30s (vm-vdicdb01-monitor-interval-30s)
>     >  Clone: nfs-vdic-images-vip-clone
>     >   Resource: nfs-vdic-images-vip (class=ocf provider=heartbeat
>     type=IPaddr)
>     >    Attributes: ip=192.168.100.201 cidr_netmask=24
>     >    Operations: start interval=0s timeout=20s
>     > (nfs-vdic-images-vip-start-interval-0s)
>     >                stop interval=0s timeout=20s
>     > (nfs-vdic-images-vip-stop-interval-0s)
>     >                monitor interval=10s
>     > (nfs-vdic-images-vip-monitor-interval-10s)
>     >  Resource: vm-vdicudsserver (class=ocf provider=heartbeat
>     > type=VirtualDomain)
>     >   Attributes: hypervisor=qemu:///system
>     > config=/mnt/nfs-vdic-mgmt-vm/vdicudsserver.xml
>     > migration_network_suffix=tcp:// migration_transport=ssh
>     >   Meta Attrs: allow-migrate=true target-role=Stopped
>     >   Utilization: cpu=1 hv_memory=1024
>     >   Operations: start interval=0s timeout=90
>     > (vm-vdicudsserver-start-interval-0s)
>     >               stop interval=0s timeout=90
>     > (vm-vdicudsserver-stop-interval-0s)
>     >               monitor interval=20s role=Stopped
>     > (vm-vdicudsserver-monitor-interval-20s)
>     >               monitor interval=30s
>     (vm-vdicudsserver-monitor-interval-30s)
>     >  Resource: vm-vdicudstuneler (class=ocf provider=heartbeat
>     > type=VirtualDomain)
>     >   Attributes: hypervisor=qemu:///system
>     > config=/mnt/nfs-vdic-mgmt-vm/vdicudstuneler.xml
>     > migration_network_suffix=tcp:// migration_transport=ssh
>     >   Meta Attrs: allow-migrate=true target-role=Stopped
>     >   Utilization: cpu=1 hv_memory=1024
>     >   Operations: start interval=0s timeout=90
>     > (vm-vdicudstuneler-start-interval-0s)
>     >               stop interval=0s timeout=90
>     > (vm-vdicudstuneler-stop-interval-0s)
>     >               monitor interval=20s role=Stopped
>     > (vm-vdicudstuneler-monitor-interval-20s)
>     >               monitor interval=30s
>     (vm-vdicudstuneler-monitor-interval-30s)
>     >
>     > Stonith Devices:
>     > Fencing Levels:
>     >
>     > Location Constraints:
>     >   Resource: nfs-grace-clone
>     >     Constraint: location-nfs-grace-clone
>     >       Rule: score=-INFINITY  (id:location-nfs-grace-clone-rule)
>     >         Expression: grace-active ne 1
>     >  (id:location-nfs-grace-clone-rule-expr)
>     >   Resource: nfs-vdic-images-vip-clone
>     >     Constraint: location-nfs-vdic-images-vip
>     >       Rule: score=-INFINITY  (id:location-nfs-vdic-images-vip-rule)
>     >         Expression: ganesha-active ne 1
>     >  (id:location-nfs-vdic-images-vip-rule-expr)
>     >   Resource: nfs-vdic-mgmt-vm-vip
>     >     Constraint: location-nfs-vdic-mgmt-vm-vip
>     >       Rule: score=-INFINITY  (id:location-nfs-vdic-mgmt-vm-vip-rule)
>     >         Expression: ganesha-active ne 1
>     >  (id:location-nfs-vdic-mgmt-vm-vip-rule-expr)
>     > Ordering Constraints:
>     > Colocation Constraints:
>     >   nfs-vdic-mgmt-vm-vip with nfs-vdic-images-vip-clone (score:-1)
>     > (id:colocation-nfs-vdic-mgmt-vm-vip-nfs-vdic-images-vip-INFINITY)
>     >   vm-vdicone01 with vm-vdicdb01 (score:-10)
>     > (id:colocation-vm-vdicone01-vm-vdicdb01-INFINITY)
>     >   vm-vdicsunstone01 with vm-vdicone01 (score:-10)
>     > (id:colocation-vm-vdicsunstone01-vm-vdicone01-INFINITY)
>     >   vm-vdicsunstone01 with vm-vdicdb01 (score:-10)
>     > (id:colocation-vm-vdicsunstone01-vm-vdicdb01-INFINITY)
>     > Ticket Constraints:
>     >
>     > Alerts:
>     >  No alerts defined
>     >
>     > Resources Defaults:
>     >  No defaults set
>     > Operations Defaults:
>     >  No defaults set
>     >
>     > Cluster Properties:
>     >  cluster-infrastructure: corosync
>     >  cluster-name: vdic-cluster
>     >  dc-version: 1.1.15-11.el7_3.2-e174ec8
>     >  have-watchdog: false
>     >  last-lrm-refresh: 1485628578
>     >  start-failure-is-fatal: false
>     >  stonith-enabled: false
>     > Node Attributes:
>     >  vdicnode01-priv: grace-active=1
>     >  vdicnode02-priv: grace-active=1
>     >
>     > Quorum:
>     >   Options:
>     > [root at vdicnode01 ~]#
>     >
>     > Any help will be welcome!
>     >
>     > Thanks a lot.