[Pacemaker] Configuration for FS over DRBD over LVM

[Bob Schatz]

Hi,

I am trying to configure an FS running on top of DRBD on top of LVM or:

     FS
     |
    DRBD
     |
    LVM

I am using Pacemaker 1.0.8, Ubuntu 10.04 and DRBD 8.3.7.

Reviewing all the manuals (Pacemaker Explained 1.0, DRBD 8.4 User Guide, etc) I came up with this Pacemaker configuration:

node cnode-1-3-5
node cnode-1-3-6
primitive glance-drbd ocf:linbit:drbd \
        params drbd_resource="glance-repos-drbd" \
        op start interval="0" timeout="240" \
        op stop interval="0" timeout="100" \
        op monitor interval="59s" role="Master" timeout="30s" \
        op monitor interval="61s" role="Slave" timeout="30s"
primitive glance-fs ocf:heartbeat:Filesystem \
        params device="/dev/drbd1" directory="/glance-mount" fstype="ext4" \
        op start interval="0" timeout="60" \
        op monitor interval="60" timeout="60" OCF_CHECK_LEVEL="20" \
        op stop interval="0" timeout="120"
primitive glance-repos ocf:heartbeat:LVM \
        params volgrpname="glance-repos" exclusive="true" \
        op start interval="0" timeout="30" \
        op stop interval="0" timeout="30"
group glance-repos-fs-group glance-fs
ms ms_drbd glance-drbd \
        meta master-node-max="1" clone-max="2" clone-node-max="1" globally-unique="false" notify="true" target-role="Master"
location drbd_on_node1 ms_drbd \
        rule $id="drbd_on_node1-rule" $role="Master" 100: #uname eq cnode-1-3-5
colocation coloc-rule-w-master inf: glance-repos ms_drbd:Master
order glance-order-fs-after-drbd inf: glance-repos:start ms_drbd:promote glance-repos-fs-group:start
property $id="cib-bootstrap-options" \
        dc-version="1.0.8-042548a451fce8400660f6031f4da6f0223dd5dd" \
        cluster-infrastructure="openais" \
        expected-quorum-votes="1" \
        stonith-enabled="false" \
        no-quorum-policy="ignore" \
        last-lrm-refresh="1310768814"

On one node, things come up cleanly.  In fact, debug messages in the agent show that the start() functions for the agent are called and exited in order (LVM start, DRBD start and Filesystem start).

The problem occurs when I do "/etc/init.d/corosync stop" on a single node.   What happens is that the stop() functions are called in this order:

1. LVM stop
2. Filesystem stop
3. DRBD stop

What I have tried:

1. I tried setting the score of the order to "500" assuming that this would mean the colocation rule would hit first.  Still the same problem.
2. I tried leaving off the ":start" and ":promote" options on the "order" line.   The stop order was still LVM, Filesystem, and DRBD
3. I tried adding another colocation rule "colocation coloc-rule-w-master2 inf: ms_drbd:Master glance-repos-fs-group" to tie glance-repos-fs-group to the same node as DRBD.   Stop still had the same issue.   I assume that I will still need this rule when I add a second node to the test.

Any suggestions would be appreciated.

A side note, the reason I have a group for the file system is that I would like to add an application and IP address to the group once I get this working.   Also, the reason I have LVM under DRBD is that I want to be able to grow the LVM volume as needed and then expand the DRBD volume.


Thanks in advance,

Bob
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.clusterlabs.org/pipermail/pacemaker/attachments/20110719/215e468f/attachment-0002.html>