[ClusterLabs] Error When Creating LVM Resource

Jason A Ramsey jason at eramsey.org
Fri Aug 26 11:32:51 EDT 2016


So, I’m setting up a two node cluster that will eventually (hopefully) serve as an HA iSCSI Target (Active/Passive) on RHEL 6. I’m using the [incredibly poorly written] guide I found on Linbit’s website (“Highly available iSCSI storage with DRBD and Pacemaker”). I have somehow gotten pretty far through it, but I’ve hit a couple of snags.

Here’s my drbd.conf (/etc/drbd.d/global_common.conf):

global {
                usage-count yes;
                # minor-count dialog-refresh disable-ip-verification
}

common {
                protocol C;

                handlers {
                                # These are EXAMPLE handlers only.
                                # They may have severe implications,
                                # like hard resetting the node under certain circumstances.
                                # Be careful when chosing your poison.

                                # pri-on-incon-degr "/usr/lib/drbd/notify-pri-on-incon-degr.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
                                # pri-lost-after-sb "/usr/lib/drbd/notify-pri-lost-after-sb.sh; /usr/lib/drbd/notify-emergency-reboot.sh; echo b > /proc/sysrq-trigger ; reboot -f";
                                # local-io-error "/usr/lib/drbd/notify-io-error.sh; /usr/lib/drbd/notify-emergency-shutdown.sh; echo o > /proc/sysrq-trigger ; halt -f";
                                # fence-peer "/usr/lib/drbd/crm-fence-peer.sh";
                                # split-brain "/usr/lib/drbd/notify-split-brain.sh root";
                                # out-of-sync "/usr/lib/drbd/notify-out-of-sync.sh root";
                                # before-resync-target "/usr/lib/drbd/snapshot-resync-target-lvm.sh -p 15 -- -c 16k";
                                # after-resync-target /usr/lib/drbd/unsnapshot-resync-target-lvm.sh;
                }

                startup {
                                # wfc-timeout degr-wfc-timeout outdated-wfc-timeout wait-after-sb
                }

                disk {
                                # on-io-error fencing use-bmbv no-disk-barrier no-disk-flushes
                                # no-disk-drain no-md-flushes max-bio-bvecs
                }

                net {
                                # sndbuf-size rcvbuf-size timeout connect-int ping-int ping-timeout max-buffers
                                # max-epoch-size ko-count allow-two-primaries cram-hmac-alg shared-secret
                                # after-sb-0pri after-sb-1pri after-sb-2pri data-integrity-alg no-tcp-cork
                }

                syncer {
                                # rate after al-extents use-rle cpu-mask verify-alg csums-alg
                }
}

And here’s my targetfs.res config (/etc/drbd.d/targetfs.res):

resource targetfs {
protocol C;
meta-disk internal;
device /dev/drbd1;
disk /dev/xvdf;
syncer {
  verify-alg sha1;
  c-plan-ahead 0;
  rate 32M;
}
net {
  allow-two-primaries;
}
on node1 {
  address  10.130.96.120:7789;
}
on node2 {
  address  10.130.97.165:7789;
}
}

These, of course, live on both nodes.

Once I create the drbd md and sync the nodes:

(node1)# drbdadm create-md targetfs
(node2)# drbdadm create-md targetfs
(node1)# drbdadm up targetfs
(node2)# drbdadm up targetfs
(node2)# drbdadm invalidate targetfs
(node1)# cat /proc/drbd
version: 8.3.16 (api:88/proto:86-97)
GIT-hash: a798fa7e274428a357657fb52f0ecf40192c1985 build by phil at Build64R6, 2014-11-24 14:51:37

1: cs:Connected ro:Primary/Secondary ds:UpToDate/UpToDate C r-----
    ns:134213632 nr:0 dw:36 dr:134215040 al:1 bm:8192 lo:0 pe:0 ua:0 ap:0 ep:1 wo:f oos:0

I run a pvcreate and an lvcreate on node1:

(node1)# pvcreate /dev/drbd/by-res/targetfs
(node1)# vgcreate targetfs /dev/drbd/by-res/targetfs
(node1)# pvs && vgs
  PV         VG       Fmt  Attr PSize   PFree
  /dev/drbd1 targetfs lvm2 a--u 127.99g 127.99g
  VG       #PV #LV #SN Attr   VSize   VFree
  targetfs   1   0   0 wz--n- 127.99g 127.99g

pcs cluster configuration goes well enough for a bit:

# pcs cluster setup --name gctvanas node1 node2 --transport udpu
# pcs cluster start --all
# pcs property set stonith-enabled=false
# pcs property set no-quorum-policy=ignore
# pcs property set default-resource-stickiness="200"
# pcs resource create gctvanas-vip ocf:heartbeat:IPaddr2 ip=10.30.96.100 cidr_netmask=32 nic=eth0 op monitor interval=30s
# pcs cluster cib drbd_cfg
# pcs -f drbd_cfg resource create gctvanas-fs1o ocf:linbit:drbd drbd_resource=targetfs op monitor interval=10s
# pcs -f drbd_cfg resource master gctvanas-fs2o gctvanas-fs1o master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true
# pcs cluster cib-push drbd_cfg
# pcs status
Cluster name: gctvanas
Stack: cman
Current DC: node1 (version 1.1.15-1.9a34920.git.el6-9a34920) - partition with quorum
Last updated: Fri Aug 26 11:29:11 2016
Last change: Fri Aug 26 11:29:07 2016 by root via cibadmin on node1

2 nodes configured
3 resources configured

Online: [ node1 node2 ]

Full list of resources:

Master/Slave Set: gctvanas-fs2o [gctvanas-fs1o]
     Masters: [ node1 ]
     Slaves: [ node2 ]
gctvanas-vip        (ocf::heartbeat:IPaddr2): Started node1

PCSD Status:
  node1: Online
  node2: Online

And then I do this:

# pcs resource create gctvanas-lvm ocf:heartbeat:LVM params volgrpname=targetfs op monitor interval="30s”

and the wheels come off the cart :-|

# pcs status
Cluster name: gctvanas
Stack: cman
Current DC: node1 (version 1.1.15-1.9a34920.git.el6-9a34920) - partition with quorum
Last updated: Fri Aug 26 11:27:29 2016
Last change: Fri Aug 26 10:57:21 2016 by root via cibadmin on node1

2 nodes configured
4 resources configured

Online: [ node1 node2 ]

Full list of resources:

Master/Slave Set: gctvanas-fs2o [gctvanas-fs1o]
     Masters: [ node1 ]
     Slaves: [ node2 ]
gctvanas-vip        (ocf::heartbeat:IPaddr2): Started node1
gctvanas-lvm      (ocf::heartbeat:LVM):       Stopped

Failed Actions:
* gctvanas-lvm_start_0 on node1 'not running' (7): call=42, status=complete, exitreason='LVM: targetfs did not activate correctly',
    last-rc-change='Fri Aug 26 10:57:22 2016', queued=0ms, exec=577ms
* gctvanas-lvm_start_0 on node2 'unknown error' (1): call=34, status=complete, exitreason='Volume group [targetfs] does not exist or contains error!   Volume group "targetfs" not found',
    last-rc-change='Fri Aug 26 10:57:21 2016', queued=0ms, exec=322ms


PCSD Status:
  node1: Online
  node2: Online

I’m not seeing anything obvious on node1 that indicates what the issue is there, but the error on node2 makes a little more sense since node2 doesn’t actually know about the pv/vg created on the drbd disk (so I feel like I’m missing a step that somehow lets node2 know what’s going on there). Any ideas?


Here’s my current cluster.conf

<cluster config_version="9" name="gctvanas">
  <fence_daemon/>
  <clusternodes>
    <clusternode name="node1" nodeid="1">
      <fence>
        <method name="pcmk-method">
          <device name="pcmk-redirect" port="node1"/>
        </method>
      </fence>
    </clusternode>
    <clusternode name="node2" nodeid="2">
      <fence>
        <method name="pcmk-method">
          <device name="pcmk-redirect" port="node2"/>
        </method>
      </fence>
    </clusternode>
  </clusternodes>
  <cman broadcast="no" expected_votes="1" transport="udpu" two_node="1"/>
  <fencedevices>
    <fencedevice agent="fence_pcmk" name="pcmk-redirect"/>
  </fencedevices>
  <rm>
    <failoverdomains/>
    <resources/>
  </rm>
</cluster>

Here’s my current cib dump:

<cib crm_feature_set="3.0.11" validate-with="pacemaker-2.6" epoch="56" num_updates="10" admin_epoch="0" cib-last-written="Fri Aug 26 10:57:21 2016" update-origin="node1" update-client="cibadmin" update-user="root" have-quorum="1" dc-uuid="node1">
  <configuration>
    <crm_config>
      <cluster_property_set id="cib-bootstrap-options">
        <nvpair id="cib-bootstrap-options-have-watchdog" name="have-watchdog" value="false"/>
        <nvpair id="cib-bootstrap-options-dc-version" name="dc-version" value="1.1.15-1.9a34920.git.el6-9a34920"/>
        <nvpair id="cib-bootstrap-options-cluster-infrastructure" name="cluster-infrastructure" value="cman"/>
        <nvpair id="cib-bootstrap-options-stonith-enabled" name="stonith-enabled" value="false"/>
        <nvpair id="cib-bootstrap-options-no-quorum-policy" name="no-quorum-policy" value="ignore"/>
        <nvpair id="cib-bootstrap-options-default-resource-stickiness" name="default-resource-stickiness" value="200"/>
        <nvpair id="cib-bootstrap-options-last-lrm-refresh" name="last-lrm-refresh" value="1472203780"/>
      </cluster_property_set>
    </crm_config>
    <nodes>
      <node id="node2" uname="node2"/>
      <node id="node1" uname="node1"/>
    </nodes>
    <resources>
      <master id="gctvanas-fs2o">
        <primitive class="ocf" id="gctvanas-fs1o" provider="linbit" type="drbd">
          <instance_attributes id="gctvanas-fs1o-instance_attributes">
            <nvpair id="gctvanas-fs1o-instance_attributes-drbd_resource" name="drbd_resource" value="targetfs"/>
          </instance_attributes>
          <operations>
            <op id="gctvanas-fs1o-start-interval-0s" interval="0s" name="start" timeout="240"/>
            <op id="gctvanas-fs1o-promote-interval-0s" interval="0s" name="promote" timeout="90"/>
            <op id="gctvanas-fs1o-demote-interval-0s" interval="0s" name="demote" timeout="90"/>
            <op id="gctvanas-fs1o-stop-interval-0s" interval="0s" name="stop" timeout="100"/>
            <op id="gctvanas-fs1o-monitor-interval-10s" interval="10s" name="monitor"/>
          </operations>
        </primitive>
        <meta_attributes id="gctvanas-fs2o-meta_attributes">
          <nvpair id="gctvanas-fs2o-meta_attributes-master-max" name="master-max" value="1"/>
          <nvpair id="gctvanas-fs2o-meta_attributes-master-node-max" name="master-node-max" value="1"/>
          <nvpair id="gctvanas-fs2o-meta_attributes-clone-max" name="clone-max" value="2"/>
          <nvpair id="gctvanas-fs2o-meta_attributes-clone-node-max" name="clone-node-max" value="1"/>
          <nvpair id="gctvanas-fs2o-meta_attributes-notify" name="notify" value="true"/>
        </meta_attributes>
      </master>
      <primitive class="ocf" id="gctvanas-vip" provider="heartbeat" type="IPaddr2">
        <instance_attributes id="gctvanas-vip-instance_attributes">
         <nvpair id="gctvanas-vip-instance_attributes-ip" name="ip" value="10.30.96.100"/>
          <nvpair id="gctvanas-vip-instance_attributes-cidr_netmask" name="cidr_netmask" value="32"/>
          <nvpair id="gctvanas-vip-instance_attributes-nic" name="nic" value="eth0"/>
        </instance_attributes>
        <operations>
          <op id="gctvanas-vip-start-interval-0s" interval="0s" name="start" timeout="20s"/>
          <op id="gctvanas-vip-stop-interval-0s" interval="0s" name="stop" timeout="20s"/>
          <op id="gctvanas-vip-monitor-interval-30s" interval="30s" name="monitor"/>
        </operations>
      </primitive>
      <primitive class="ocf" id="gctvanas-lvm" provider="heartbeat" type="LVM">
        <instance_attributes id="gctvanas-lvm-instance_attributes">
          <nvpair id="gctvanas-lvm-instance_attributes-volgrpname" name="volgrpname" value="targetfs"/>
        </instance_attributes>
        <operations>
          <op id="gctvanas-lvm-start-interval-0s" interval="0s" name="start" timeout="30"/>
          <op id="gctvanas-lvm-stop-interval-0s" interval="0s" name="stop" timeout="30"/>
          <op id="gctvanas-lvm-monitor-interval-30s" interval="30s" name="monitor"/>
        </operations>
      </primitive>
    </resources>
    <constraints/>
    <rsc_defaults>
      <meta_attributes id="rsc_defaults-options">
        <nvpair id="rsc_defaults-options-resource-stickiness" name="resource-stickiness" value="100"/>
      </meta_attributes>
    </rsc_defaults>
  </configuration>
  <status>
    <node_state id="node1" uname="node1" in_ccm="true" crmd="online" crm-debug-origin="do_update_resource" join="member" expected="member">
      <lrm id="node1">
        <lrm_resources>
          <lrm_resource id="gctvanas-vip" type="IPaddr2" class="ocf" provider="heartbeat">
            <lrm_rsc_op id="gctvanas-vip_last_0" operation_key="gctvanas-vip_start_0" operation="start" crm-debug-origin="do_update_resource" crm_feature_set="3.0.11" transition-key="32:1:0:681b3ca7-f83d-4396-a249-d6d80e0efe16" transition-magic="0:0;32:1:0:681b3ca7-f83d-4396-a249-d6d80e0efe16" on_node="node1" call-id="11" rc-code="0" op-status="0" interval="0" last-run="1472203004" last-rc-change="1472203004" exec-time="74" queue-time="0" op-digest="15b4ba230497d33ad5d77f05e4b9a83e"/>
            <lrm_rsc_op id="gctvanas-vip_monitor_30000" operation_key="gctvanas-vip_monitor_30000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.11" transition-key="33:1:0:681b3ca7-f83d-4396-a249-d6d80e0efe16" transition-magic="0:0;33:1:0:681b3ca7-f83d-4396-a249-d6d80e0efe16" on_node="node1" call-id="14" rc-code="0" op-status="0" interval="30000" last-rc-change="1472203004" exec-time="72" queue-time="0" op-digest="205179ac48c643694ee24512cc3b1429"/>
          </lrm_resource>
          <lrm_resource id="gctvanas-fs1o" type="drbd" class="ocf" provider="linbit">
            <lrm_rsc_op id="gctvanas-fs1o_last_failure_0" operation_key="gctvanas-fs1o_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.11" transition-key="3:9:7:681b3ca7-f83d-4396-a249-d6d80e0efe16" transition-magic="0:8;3:9:7:681b3ca7-f83d-4396-a249-d6d80e0efe16" on_node="node1" call-id="37" rc-code="8" op-status="0" interval="0" last-run="1472203781" last-rc-change="1472203781" exec-time="37" queue-time="1" op-digest="fb1e24e691d75f64117224686c0f806b"/>
            <lrm_rsc_op id="gctvanas-fs1o_last_0" operation_key="gctvanas-fs1o_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.11" transition-key="3:9:7:681b3ca7-f83d-4396-a249-d6d80e0efe16" transition-magic="0:8;3:9:7:681b3ca7-f83d-4396-a249-d6d80e0efe16" on_node="node1" call-id="37" rc-code="8" op-status="0" interval="0" last-run="1472203781" last-rc-change="1472203781" exec-time="37" queue-time="1" op-digest="fb1e24e691d75f64117224686c0f806b"/>
          </lrm_resource>
          <lrm_resource id="gctvanas-lvm" type="LVM" class="ocf" provider="heartbeat">
            <lrm_rsc_op id="gctvanas-lvm_last_0" operation_key="gctvanas-lvm_stop_0" operation="stop" crm-debug-origin="do_update_resource" crm_feature_set="3.0.11" transition-key="2:36:0:681b3ca7-f83d-4396-a249-d6d80e0efe16" transition-magic="0:0;2:36:0:681b3ca7-f83d-4396-a249-d6d80e0efe16" on_node="node1" call-id="43" rc-code="0" op-status="0" interval="0" last-run="1472223443" last-rc-change="1472223443" exec-time="369" queue-time="0" op-digest="df48f11b20123e34ddf99999ce9f3f1c" exit-reason="LVM: targetfs did not activate correctly"/>
            <lrm_rsc_op id="gctvanas-lvm_last_failure_0" operation_key="gctvanas-lvm_start_0" operation="start" crm-debug-origin="do_update_resource" crm_feature_set="3.0.11" transition-key="37:35:0:681b3ca7-f83d-4396-a249-d6d80e0efe16" transition-magic="0:7;37:35:0:681b3ca7-f83d-4396-a249-d6d80e0efe16" exit-reason="LVM: targetfs did not activate correctly" on_node="node1" call-id="42" rc-code="7" op-status="0" interval="0" last-run="1472223442" last-rc-change="1472223442" exec-time="577" queue-time="0" op-digest="df48f11b20123e34ddf99999ce9f3f1c"/>
          </lrm_resource>
        </lrm_resources>
      </lrm>
      <transient_attributes id="node1">
        <instance_attributes id="status-node1">
          <nvpair id="status-node1-shutdown" name="shutdown" value="0"/>
          <nvpair id="status-node1-last-failure-gctvanas-fs1o" name="last-failure-gctvanas-fs1o" value="1472203145"/>
          <nvpair id="status-node1-master-gctvanas-fs1o" name="master-gctvanas-fs1o" value="10000"/>
          <nvpair id="status-node1-fail-count-gctvanas-lvm" name="fail-count-gctvanas-lvm" value="INFINITY"/>
          <nvpair id="status-node1-last-failure-gctvanas-lvm" name="last-failure-gctvanas-lvm" value="1472223442"/>
        </instance_attributes>
      </transient_attributes>
    </node_state>
    <node_state id="node2" uname="node2" in_ccm="true" crmd="online" crm-debug-origin="do_update_resource" join="member" expected="member">
      <lrm id="node2">
        <lrm_resources>
          <lrm_resource id="gctvanas-vip" type="IPaddr2" class="ocf" provider="heartbeat">
            <lrm_rsc_op id="gctvanas-vip_last_0" operation_key="gctvanas-vip_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.11" transition-key="5:0:7:681b3ca7-f83d-4396-a249-d6d80e0efe16" transition-magic="0:7;5:0:7:681b3ca7-f83d-4396-a249-d6d80e0efe16" on_node="node2" call-id="10" rc-code="7" op-status="0" interval="0" last-run="1472203004" last-rc-change="1472203004" exec-time="63" queue-time="0" op-digest="15b4ba230497d33ad5d77f05e4b9a83e"/>
          </lrm_resource>
          <lrm_resource id="gctvanas-fs1o" type="drbd" class="ocf" provider="linbit">
            <lrm_rsc_op id="gctvanas-fs1o_last_failure_0" operation_key="gctvanas-fs1o_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.11" transition-key="4:9:7:681b3ca7-f83d-4396-a249-d6d80e0efe16" transition-magic="0:0;4:9:7:681b3ca7-f83d-4396-a249-d6d80e0efe16" on_node="node2" call-id="28" rc-code="0" op-status="0" interval="0" last-run="1472203781" last-rc-change="1472203781" exec-time="39" queue-time="0" op-digest="fb1e24e691d75f64117224686c0f806b"/>
            <lrm_rsc_op id="gctvanas-fs1o_last_0" operation_key="gctvanas-fs1o_monitor_0" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.11" transition-key="4:9:7:681b3ca7-f83d-4396-a249-d6d80e0efe16" transition-magic="0:0;4:9:7:681b3ca7-f83d-4396-a249-d6d80e0efe16" on_node="node2" call-id="28" rc-code="0" op-status="0" interval="0" last-run="1472203781" last-rc-change="1472203781" exec-time="39" queue-time="0" op-digest="fb1e24e691d75f64117224686c0f806b"/>
            <lrm_rsc_op id="gctvanas-fs1o_monitor_10000" operation_key="gctvanas-fs1o_monitor_10000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.11" transition-key="9:10:0:681b3ca7-f83d-4396-a249-d6d80e0efe16" transition-magic="0:0;9:10:0:681b3ca7-f83d-4396-a249-d6d80e0efe16" on_node="node2" call-id="29" rc-code="0" op-status="0" interval="10000" last-rc-change="1472203781" exec-time="45" queue-time="0" op-digest="1a2c711d2933e74557b4c1a13ec62162"/>
          </lrm_resource>
          <lrm_resource id="gctvanas-lvm" type="LVM" class="ocf" provider="heartbeat">
            <lrm_rsc_op id="gctvanas-lvm_last_0" operation_key="gctvanas-lvm_stop_0" operation="stop" crm-debug-origin="do_update_resource" crm_feature_set="3.0.11" transition-key="3:35:0:681b3ca7-f83d-4396-a249-d6d80e0efe16" transition-magic="0:0;3:35:0:681b3ca7-f83d-4396-a249-d6d80e0efe16" on_node="node2" call-id="35" rc-code="0" op-status="0" interval="0" last-run="1472223442" last-rc-change="1472223442" exec-time="151" queue-time="0" op-digest="df48f11b20123e34ddf99999ce9f3f1c" exit-reason="Volume group [targetfs] does not exist or contains error!   Volume group "targetfs" not found"/>
            <lrm_rsc_op id="gctvanas-lvm_last_failure_0" operation_key="gctvanas-lvm_start_0" operation="start" crm-debug-origin="do_update_resource" crm_feature_set="3.0.11" transition-key="38:33:0:681b3ca7-f83d-4396-a249-d6d80e0efe16" transition-magic="0:1;38:33:0:681b3ca7-f83d-4396-a249-d6d80e0efe16" exit-reason="Volume group [targetfs] does not exist or contains error!   Volume group "targetfs" not found" on_node="node2" call-id="34" rc-code="1" op-status="0" interval="0" last-run="1472223441" last-rc-change="1472223441" exec-time="322" queue-time="0" op-digest="df48f11b20123e34ddf99999ce9f3f1c"/>
          </lrm_resource>
        </lrm_resources>
      </lrm>
      <transient_attributes id="node2">
        <instance_attributes id="status-node2">
          <nvpair id="status-node2-shutdown" name="shutdown" value="0"/>
          <nvpair id="status-node2-last-failure-gctvanas-fs1o" name="last-failure-gctvanas-fs1o" value="1472203144"/>
          <nvpair id="status-node2-master-gctvanas-fs1o" name="master-gctvanas-fs1o" value="10000"/>
          <nvpair id="status-node2-fail-count-gctvanas-lvm" name="fail-count-gctvanas-lvm" value="INFINITY"/>
          <nvpair id="status-node2-last-failure-gctvanas-lvm" name="last-failure-gctvanas-lvm" value="1472223441"/>
        </instance_attributes>
      </transient_attributes>
    </node_state>
  </status>
</cib>

--

[ jR ]
  @: jason at eramsey.org<mailto:jason at eramsey.org>

  there is no path to greatness; greatness is the path
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20160826/c41fbc2a/attachment-0002.html>


More information about the Users mailing list