[ClusterLabs] Colocation and ordering with live migration

Mon Oct 10 11:32:22 UTC 2016

On 10/10/2016 10:17 AM, Pavel Levshin wrote:
> Hello.
>
> We are trying to migrate our services to relatively fresh version of
> cluster software. It is RHEL 7 with pacemaker 1.1.13-10. I’ve faced a
> problem when live migration of virtual machines is allowed. In short,
> I need to manage libvirtd, and I cannot set proper ordering
> constraints to stop libvirtd only after all virtual machines are
> stopped. This leads to failing migration and STONITH when I try to
> shutdown one of the nodes.
>
> If live migration Is disabled, then all works well.
Why are the order-constraints between libvirt & vms optional?
>
> Cluster config follows:
>
> ==
>
> Cluster Name: smartbvcluster
>
> Corosync Nodes:
>
> bvnode1 bvnode2
>
> Pacemaker Nodes:
>
> bvnode1 bvnode2
>
> Resources:
>
> Clone: dlm-clone
>
>   Meta Attrs: interleave=true ordered=true
>
>   Resource: dlm (class=ocf provider=pacemaker type=controld)
>
>    Operations: start interval=0s timeout=90 (dlm-start-interval-0s)
>
>                stop interval=0s timeout=100 (dlm-stop-interval-0s)
>
>                monitor interval=30s (dlm-monitor-interval-30s)
>
> Clone: clvmd-clone
>
>   Meta Attrs: interleave=true ordered=true
>
>   Resource: clvmd (class=ocf provider=heartbeat type=clvm)
>
>    Operations: start interval=0s timeout=90 (clvmd-start-interval-0s)
>
>                stop interval=0s timeout=90 (clvmd-stop-interval-0s)
>
>                monitor interval=30s (clvmd-monitor-interval-30s)
>
> Clone: cluster-config-clone
>
>   Meta Attrs: interleave=true
>
>   Resource: cluster-config (class=ocf provider=heartbeat type=Filesystem)
>
>    Attributes: device=/dev/vg_bv_shared/cluster-config
> directory=/opt/cluster-config fstype=gfs2 options=noatime
>
>    Operations: start interval=0s timeout=60
> (cluster-config-start-interval-0s)
>
>                stop interval=0s timeout=60
> (cluster-config-stop-interval-0s)
>
>                monitor interval=10s on-fail=fence OCF_CHECK_LEVEL=20
> (cluster-config-monitor-interval-10s)
>
> Resource: vm_smartbv1 (class=ocf provider=heartbeat type=VirtualDomain)
>
>   Attributes: config=/opt/cluster-config/libvirt/qemu/smartbv1.xml
> hypervisor=qemu:///system migration_transport=tcp
>
>   Meta Attrs: allow-migrate=true
>
>   Operations: start interval=0s timeout=90
> (vm_smartbv1-start-interval-0s)
>
>               stop interval=0s timeout=90 (vm_smartbv1-stop-interval-0s)
>
>               monitor interval=10 timeout=30
> (vm_smartbv1-monitor-interval-10)
>
> Resource: vm_smartbv2 (class=ocf provider=heartbeat type=VirtualDomain)
>
>   Attributes: config=/opt/cluster-config/libvirt/qemu/smartbv2.xml
> hypervisor=qemu:///system migration_transport=tcp
>
>   Meta Attrs: target-role=started allow-migrate=true
>
>   Operations: start interval=0s timeout=90
> (vm_smartbv2-start-interval-0s)
>
>               stop interval=0s timeout=90 (vm_smartbv2-stop-interval-0s)
>
>               monitor interval=10 timeout=30
> (vm_smartbv2-monitor-interval-10)
>
> Clone: libvirtd-clone
>
>   Meta Attrs: interleave=true
>
>   Resource: libvirtd (class=systemd type=libvirtd)
>
>    Operations: monitor interval=60s (libvirtd-monitor-interval-60s)
>
> Stonith Devices:
>
> Resource: ilo.bvnode2 (class=stonith type=fence_ilo4)
>
>   Attributes: ipaddr=ilo.bvnode2 login=hacluster passwd=s
> pcmk_host_list=bvnode2 privlvl=operator
>
>   Operations: monitor interval=60s (ilo.bvnode2-monitor-interval-60s)
>
> Resource: ilo.bvnode1 (class=stonith type=fence_ilo4)
>
>   Attributes: ipaddr=ilo.bvnode1 login=hacluster passwd=s
> pcmk_host_list=bvnode1 privlvl=operator
>
>   Operations: monitor interval=60s (ilo.bvnode1-monitor-interval-60s)
>
> Fencing Levels:
>
> Node: bvnode1
>
>   Level 10 - ilo.bvnode1
>
> Node: bvnode2
>
>   Level 10 - ilo.bvnode2
>
> Location Constraints:
>
>   Resource: ilo.bvnode1
>
>     Disabled on: bvnode1 (score:-INFINITY)
> (id:location-ilo.bvnode1-bvnode1--INFINITY)
>
>   Resource: ilo.bvnode2
>
>     Disabled on: bvnode2 (score:-INFINITY)
> (id:location-ilo.bvnode2-bvnode2--INFINITY)
>
> Ordering Constraints:
>
>   start dlm-clone then start clvmd-clone (kind:Mandatory)
> (id:order-dlm-clone-clvmd-clone-mandatory)
>
>   start clvmd-clone then start cluster-config-clone (kind:Mandatory)
> (id:order-clvmd-clone-cluster-config-clone-mandatory)
>
>   start cluster-config-clone then start libvirtd-clone
> (kind:Mandatory) (id:order-cluster-config-clone-libvirtd-clone-mandatory)
>
>   start libvirtd-clone then start vm_smartbv2 (kind:Optional)
> (id:order-libvirtd-clone-vm_smartbv2-Optional)
>
>   start libvirtd-clone then start vm_smartbv1 (kind:Optional)
> (id:order-libvirtd-clone-vm_smartbv1-Optional)
>
> Colocation Constraints:
>
>   clvmd-clone with dlm-clone (score:INFINITY)
> (id:colocation-clvmd-clone-dlm-clone-INFINITY)
>
>   cluster-config-clone with clvmd-clone (score:INFINITY)
> (id:colocation-cluster-config-clone-clvmd-clone-INFINITY)
>
>   libvirtd-clone with cluster-config-clone (score:INFINITY)
> (id:colocation-libvirtd-clone-cluster-config-clone-INFINITY)
>
>   vm_smartbv1 with libvirtd-clone (score:INFINITY)
> (id:colocation-vm_smartbv1-libvirtd-clone-INFINITY)
>
>   vm_smartbv2 with libvirtd-clone (score:INFINITY)
> (id:colocation-vm_smartbv2-libvirtd-clone-INFINITY)
>
> Resources Defaults:
>
> No defaults set
>
> Operations Defaults:
>
> No defaults set
>
> Cluster Properties:
>
> cluster-infrastructure: corosync
>
> cluster-name: smartbvcluster
>
> dc-version: 1.1.13-10.el7_2.4-44eb2dd
>
> have-watchdog: false
>
> last-lrm-refresh: 1475752213
>
> maintenance-mode: false
>
> no-quorum-policy: freeze
>
> start-failure-is-fatal: false
>
> stonith-enabled: true
>
> ==
>
> Crm_simulate output is attached. What exactly means “dashed-arrows
> indicate dependencies that are not present in the transition graph”?
> What shall I do to make them present?
>
>
> -- 
>
> Pavel Levshin
>
>
>
>
> _______________________________________________
> Users mailing list: Users at clusterlabs.org
> http://clusterlabs.org/mailman/listinfo/users
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org