[Pacemaker] Help with colocation and order of primitives

Jake Smith jsmith at argotec.com
Mon Feb 13 15:22:09 EST 2012 

----- Original Message ----- 

> From: "Andrew Martin" <amartin at xes-inc.com>
> To: "The Pacemaker cluster resource manager"
> <pacemaker at oss.clusterlabs.org>
> Sent: Monday, February 13, 2012 2:11:12 PM
> Subject: Re: [Pacemaker] Help with colocation and order of primitives

> Hi Jake,

> Thanks, that helps to clarify how to properly create the colocation
> and order constraints and how the crm shell corresponds to the XML.
> You mentioned that I should "verify the statement didn't create a
> resource set and change the outcome a little" - can you elaborate on
> this a bit more? After I re-created the contraints, the XML lists
> two resource sets in each constraint:

Example colocation:

colocation c_test2_on_test1_on_drbd_master inf: p_test2 p_test1 ms_drbd_samba:Master

XML output:

      <rsc_colocation id="c_test2_on_test1_on_drbd_master" score="INFINITY">
        <resource_set id="c_test2_on_test1_on_drbd_master-0">
          <resource_ref id="p_test2"/>
          <resource_ref id="p_test1"/>
        </resource_set>
        <resource_set id="c_test2_on_test1_on_drbd_master-1" role="Master">
          <resource_ref id="ms_drbd_samba"/>
        </resource_set>

Everything looks fine... test2 on test1 on drbd:Master.  Now if you run "crm resource stop p_test2" you would expect p_test1 still running on drbd:Master... but that's not the case:

crm_mon -fr:
 p_test1        (ocf::pacemaker:Dummy): Started Condor
 p_test2        (ocf::pacemaker:Dummy): Started Condor
crm resource stop p_test2
crm_mon -fr:
 p_test1        (ocf::pacemaker:Dummy): Stopped
 p_test2        (ocf::pacemaker:Dummy): Stopped

The reason is that when the resource_set is created it acts as a group so it colocates in reverse of how it's written i.e. in my example it's actually
test1 on test2 on drbd:Master
p_test2, p_test1 are a group (resource set) now.

Correct syntax for the outcome to be test2 on test1 on drbd:Master would be:

colocation c_test2_on_test1_on_drbd_master inf: p_test1 p_test2 ms_drbd_samba:Master

Keep in mind that anytime you have more than 2 resources in a colocation statement depending upon their types they *may* create resource sets and when they do it reverses their colocation direction



> <constraints>
> <rsc_order id="o_drbd-fs-vm" score="INFINITY">
> <resource_set action="promote" id="o_drbd-fs-vm-0">
> <resource_ref id="ms_drbd_r0"/>
> <resource_ref id="ms_drbd_r1"/>
> <resource_ref id="ms_drbd_r2"/>
> </resource_set>
> <resource_set action="start" id="o_drbd-fs-vm-1">
> <resource_ref id="p_fs_r0"/>
> <resource_ref id="p_virtualdomain_vm0"/>
> </resource_set>
> </rsc_order>
> <rsc_colocation id="c_drbd_libvirt_vm" score="INFINITY">
> <resource_set id="c_drbd_libvirt_vm-0" role="Master">
> <resource_ref id="ms_drbd_r0"/>
> <resource_ref id="ms_drbd_r1"/>
> <resource_ref id="ms_drbd_r2"/>
> </resource_set>

these will colocate this way:
r2 on r1 on r0

> <resource_set id="c_drbd_libvirt_vm-1">
> <resource_ref id="p_fs_r0"/>
> <resource_ref id="p_virtualdomain_vm0"/>
> </resource_set>

and these:
vm0 on fs_r0

> </rsc_colocation>
> </constraints>

which all together is:

vm0 on fs_r0 on r2 on r1 on r0

> Is this the desired behavior, or will having two resource sets cause
> the resources to not start linearly/in order as desired?

It's looks to me like this is the behavior you want... but hopefully my explaination above will allow you to make the right decision

HTH

Jake

> Thanks,

> Andrew
> ----- Original Message -----

> From: "Jake Smith" <jsmith at argotec.com>
> To: "The Pacemaker cluster resource manager"
> <pacemaker at oss.clusterlabs.org>
> Sent: Friday, February 10, 2012 12:06:15 PM
> Subject: Re: [Pacemaker] Help with colocation and order of primitives

> ----- Original Message -----

> > From: "Andrew Martin" <amartin at xes-inc.com>
> > To: "The Pacemaker cluster resource manager"
> > <pacemaker at oss.clusterlabs.org>
> > Sent: Friday, February 10, 2012 10:33:44 AM
> > Subject: [Pacemaker] Help with colocation and order of primitives

> > Hello,

> > I am working on configuring a 2-node cluster with 3 DRBD devices,
> > their corresponding filesystem mounts, and a VirtualDomain resource
> > agent. I want to add the appropriate constraints to pacemaker so
> > that these resources only start together on the same node, and that
> > they start in the proper order. I have added the DRBD devices to a
> > Master/Slave object, as outlined in
> > http://www.linbit.com/fileadmin/tech-guides/ha-nfs.pdf . I would
> > like them to start in the following order (and stop in the reverse
> > order):
> > ms_drbd_r0
> > ms_drbd_r1
> > ms_drbd_r2
> > p_filesystem_r0
> > p_filesystem_r1
> > p_filesystem_r2
> > p_virtualdomain_vm0

> > I added the 3 filesystem primitives to a group, g_filesystems,
> > which
> > simplifies this somewhat. Are primitives in a group started in the
> > order they are listed in a group, and stopped in the opposite
> > order?

> Yes and colocated in reverse...

> group test A B C
> order is A B C
> colocation is C B A
> Starts A then B then C, Stops C, B, A.

> > I read through the Pacemaker Explained documentation on colocation
> > and order,
> > http://www.clusterlabs.org/doc/en-US/Pacemaker/1.1/html/Pacemaker_Explained/s-resource-sets-ordering.html

> It shows XML but when adding via the CRM CLI the rules work the same.
> The CRM CLI is just a(n easier) front-end for the XML (config is
> still written in XML)
> Guide for CRM CLI is here:
> http://www.clusterlabs.org/doc/crm_cli.html

> > however it seems to mostly address the XML and not how to add
> > constraints via the crm console (crm configure). It seems that I
> > cannot add the Master/Slave objects to a group, so I will need to
> > specify each of them individual in the constraints. Is it possible
> > to include more than 2 resources to a constraint, e.g.
> > colocation c_all_on_one inf: ms_drbd_r0:Master ms_drbd_r1:Master
> > ms_drbd_r2:Master g_filesystems p_virtualdomain_vm0

> Yes you can put more than one primitive in a colocation or order
> constraint - just be careful when adding multiples of different
> types of resource. You may want to do "crm configure show xml" to
> verify the statement didn't create a resource set and change the
> outcome a little.
> Also with order statements make sure you specify the action when
> combining different types otherwise the first action will apply to
> all resources i.e. ms_drbd_r0:promote g_filesystems:start

> > Or, can a single colocation or order constraint only contain 2
> > resources? If so would I need to chain several constraints
> > together,
> > like this:
> > colocation c_r0-r1 inf: ms_drbd_r0:Master ms_drbd_r1:Master
> > colocation c_r1-r2 inf: ms_drbd_r1:Master ms_drbd:r2:Master
> > colocation c_r2-fs inf: ms_drbd_r2:Master g_filesystems
> > colocation c_fs_vm0 inf: g_filesystems p_virtualdomain_vm0

> > Which syntax is supported and how can I achieve these colocation
> > and
> > order constraints?

> > Thanks,

> > Andrew
> > _______________________________________________
> > Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> > http://oss.clusterlabs.org/mailman/listinfo/pacemaker

> > Project Home: http://www.clusterlabs.org
> > Getting started:
> > http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> > Bugs: http://bugs.clusterlabs.org

> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker

> Project Home: http://www.clusterlabs.org
> Getting started:
> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org

> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker

> Project Home: http://www.clusterlabs.org
> Getting started:
> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org

More information about the Pacemaker mailing list