[Pacemaker] Trouble Starting Filesystem
Jake Smith
jsmith at argotec.com
Tue Dec 11 10:04:03 EST 2012
----- Original Message -----
> From: "Art Zemon" <art at hens-teeth.net>
> To: "The Pacemaker cluster resource manager" <pacemaker at oss.clusterlabs.org>
> Sent: Monday, December 10, 2012 6:07:04 PM
> Subject: Re: [Pacemaker] Trouble Starting Filesystem
>
> Folks,
>
> I am still struggling with this problem. At the moment, I cannot get
> my OCSF2 filesystem to start at all. OCFS2 worked until I expanded
> my cluster from 2 nodes to 4 nodes.
>
> I see this in /var/log/syslog. In particular, note the "FATAL: Module
> scsi_hostadapter not found." on the last line.
Pretty sure this doesn't have anything to do with your problem. I had this when testing OCFS2 in the cluster and it never caused any issues... when I asked about it someone mentioned it was something not cleaned up somewhere but harmless
> Dec 10 16:48:03 aztestc1 crmd: [2416]: info: do_lrm_rsc_op:
> Performing key=71:14:0:a766cb8e-4813-483e-a127-d67cf25979ea
> op=p_fs_share_plesk:0_start_0 )
> Dec 10 16:48:03 aztestc1 lrmd: [2413]: debug: on_msg_perform_op:2396:
> copying parameters for rsc p_fs_share_plesk:0
> Dec 10 16:48:03 aztestc1 lrmd: [2413]: debug: on_msg_perform_op: add
> an operation operation start[29] on p_fs_share_plesk:0 for client
> 2416, its parameters:
> CRM_meta_notify_start_resource=[p_fs_share_plesk:0
> p_fs_share_plesk:1 ] CRM_meta_notify_stop_resource=[ ]
> fstype=[ocfs2] CRM_meta_notify_demote_resource=[ ]
> CRM_meta_notify_master_uname=[ ] CRM_meta_notify_promote_uname=[ ]
> CRM_meta_timeout=[60000] options=[rw,noatime] CRM_meta_name=[start]
> CRM_meta_notify_inactive_resource=[p_fs_share_plesk:0
> p_fs_share_plesk:1 ] CRM_meta_notify_start_uname=[aztestc1 aztestc2
> ] crm_feature_set=[3.0 to the operation list.
> Dec 10 16:48:03 aztestc1 lrmd: [2413]: info: rsc:p_fs_share_plesk:0
> start[29] (pid 4528)
> Dec 10 16:48:03 aztestc1 crmd: [2416]: debug: get_xpath_object: No
> match for //cib_update_result//diff-added//crm_config in
> /notify/cib_update_result/diff
> Dec 10 16:48:03 aztestc1 crmd: [2416]: debug: get_xpath_object: No
> match for //cib_update_result//diff-added//crm_config in
> /notify/cib_update_result/diff
> Dec 10 16:48:03 aztestc1 lrmd: [2413]: debug:
> rsc:p_drbd_share_plesk:1 monitor[16] (pid 4530)
> Dec 10 16:48:03 aztestc1 crmd: [2416]: debug: get_xpath_object: No
> match for //cib_update_result//diff-added//crm_config in
> /notify/cib_update_result/diff
> Dec 10 16:48:03 aztestc1 Filesystem[4528]: INFO: Running start for
> /dev/drbd/by-res/shareplesk on /shareplesk
> Dec 10 16:48:03 aztestc1 drbd[4530]: DEBUG: shareplesk: Calling
> /usr/sbin/crm_master -Q -l reboot -v 10000
> Dec 10 16:48:03 aztestc1 lrmd: [2413]: info: RA output:
> (p_fs_share_plesk:0:start:stderr) FATAL: Module scsi_hostadapter not
> found.
>
>
>
> DRBD is running in dual-primary mode:
>
> root at aztestc1:~# service drbd status
> drbd driver loaded OK; device status:
> version: 8.3.11 (api:88/proto:86-96)
> srcversion: 71955441799F513ACA6DA60
> m:res cs ro ds p
> mounted fstype
> 1:shareplesk Connected Primary/Primary UpToDate/UpToDate C
>
>
>
> Everything looks happy:
>
> root at aztestc1:~# crm_mon -1
> ============
> Last updated: Mon Dec 10 16:59:40 2012
> Last change: Mon Dec 10 16:48:02 2012 via crmd on aztestc3
> Stack: cman
> Current DC: aztestc3 - partition with quorum
> Version: 1.1.6-9971ebba4494012a93c03b40a2c58ec0eb60f50c
> 4 Nodes configured, unknown expected votes
> 10 Resources configured.
> ============
>
> Online: [ aztestc3 aztestc4 aztestc1 aztestc2 ]
>
> Clone Set: cl_fencing [p_stonith]
> Started: [ aztestc2 aztestc1 aztestc4 aztestc3 ]
> Clone Set: cl_o2cb [p_o2cb]
> Started: [ aztestc1 aztestc2 ]
> Master/Slave Set: ms_drbd_share_plesk [p_drbd_share_plesk]
> Masters: [ aztestc2 aztestc1 ]
>
> Failed actions:
> p_fs_share_plesk:1_start_0 (node=aztestc2, call=31, rc=1,
> status=complete): unknown error
> p_fs_share_plesk:0_start_0 (node=aztestc1, call=29, rc=1,
> status=complete): unknown error
Would help if you pastebin log output a little before and after the failed filesystem actions from aztest1/2. I don't know what a return code 1 from filesystem RA signifies but that would be a good start.
>
>
>
> Here is my complete configuration, which does not work:
>
> node aztestc1 \
> attributes standby="off"
> node aztestc2 \
> attributes standby="off"
> node aztestc3 \
> attributes standby="off"
> node aztestc4 \
> attributes standby="off"
> primitive p_drbd_share_plesk ocf:linbit:drbd \
> params drbd_resource="shareplesk" \
> op monitor interval="15s" role="Master" timeout="20s" \
> op monitor interval="20s" role="Slave" timeout="20s" \
> op start interval="0" timeout="240s" \
> op stop interval="0" timeout="100s"
> primitive p_fs_share_plesk ocf:heartbeat:Filesystem \
> params device="/dev/drbd/by-res/shareplesk" directory="/shareplesk"
> fstype="ocfs2" options="rw,noatime" \
> op start interval="0" timeout="60" \
> op stop interval="0" timeout="60" \
> op monitor interval="20" timeout="40"
> primitive p_o2cb ocf:pacemaker:o2cb \
> params stack="cman" \
> op start interval="0" timeout="90" \
> op stop interval="0" timeout="100" \
> op monitor interval="10" timeout="20"
> primitive p_stonith stonith:fence_ec2 \
> params pcmk_host_check="static-list" pcmk_host_list="aztestc1
> aztestc2 aztestc3 aztestc4" \
> op monitor interval="600s" timeout="300s" \
> op start start-delay="10s" interval="0"
> ms ms_drbd_share_plesk p_drbd_share_plesk \
> meta master-max="2" notify="true" interleave="true" clone-max="2"
> is-managed="true" target-role="Started"
> clone cl_fencing p_stonith \
> meta target-role="Started"
> clone cl_fs_share_plesk p_fs_share_plesk \
> meta clone-max="2" interleave="true" notify="true"
> globally-unique="false" target-role="Started"
> clone cl_o2cb p_o2cb \
> meta clone-max="2" interleave="true" globally-unique="false"
> target-role="Started"
> location lo_drbd_plesk3 ms_drbd_share_plesk -inf: aztestc3
> location lo_drbd_plesk4 ms_drbd_share_plesk -inf: aztestc4
> location lo_fs_plesk3 cl_fs_share_plesk -inf: aztestc3
> location lo_fs_plesk4 cl_fs_share_plesk -inf: aztestc4
> location lo_o2cb3 cl_o2cb -inf: aztestc3
> location lo_o2cb4 cl_o2cb -inf: aztestc4
> order o_20plesk inf: ms_drbd_share_plesk:promote cl_o2cb:start
> order o_40fs_plesk inf: cl_o2cb cl_fs_share_plesk
> property $id="cib-bootstrap-options" \
> stonith-enabled="true" \
> stonith-timeout="180s" \
> no-quorum-policy="freeze" \
> dc-version="1.1.6-9971ebba4494012a93c03b40a2c58ec0eb60f50c" \
> cluster-infrastructure="cman" \
> last-lrm-refresh="1355179514"
> rsc_defaults $id="rsc-options" \
> resource-stickiness="100"
>
Not sure it will fix your issues completely but I would add a couple collocation statements and remove a few location statements. The collocation statements basically say resource A must run where resource B runs; if resource B isn't running on a particular node then resource A cannot run there. This way once you've located DRBD then the other resources can only run on the nodes that DRBD is running on:
Remove these:
> location lo_fs_plesk3 cl_fs_share_plesk -inf: aztestc3
> location lo_fs_plesk4 cl_fs_share_plesk -inf: aztestc4
> location lo_o2cb3 cl_o2cb -inf: aztestc3
> location lo_o2cb4 cl_o2cb -inf: aztestc4
Add these:
collocation c_o2cb_on_drbd_master inf: cl_o2cb ms_drbd_share_plesk:Master
collocation c_fs_on_o2cb inf: cl_fs_share_plesk cl_o2cb
If you wanted to make it more concise you could group a few resources which would reduce the number of statements. This assumes you didn't make the changes listed above yet:
Remove these:
> clone cl_fs_share_plesk p_fs_share_plesk \
> meta clone-max="2" interleave="true" notify="true"
> globally-unique="false" target-role="Started"
> clone cl_o2cb p_o2cb \
> meta clone-max="2" interleave="true" globally-unique="false"
> target-role="Started"
> location lo_fs_plesk3 cl_fs_share_plesk -inf: aztestc3
> location lo_fs_plesk4 cl_fs_share_plesk -inf: aztestc4
> location lo_o2cb3 cl_o2cb -inf: aztestc3
> location lo_o2cb4 cl_o2cb -inf: aztestc4
> order o_20plesk inf: ms_drbd_share_plesk:promote cl_o2cb:start
> order o_40fs_plesk inf: cl_o2cb cl_fs_share_plesk
Add these (you can't multi-state resources in a group FYI):
group g_o2cb_fs p_o2cb p_fs_share_plesk
clone cl_o2cb_fs g_o2cb_fs meta clone-max="2" interleave="true" notify="true" globally-unique="false"
order o_drbd_then_ocfs inf: ms_drbd_share_plesk:promote cl_o2cb_fs:start
collocation c_ocfs_on_drbd_master inf: cl_o2cb_fs ms_drbd_share_plesk:Master
>
>
> and here is my previous 2-node configuration, which worked "mostly."
> Sometimes I had to manually "crm resource cleanup cl_fs_share" to
> get the filesystem to mount but otherwise eveyrthing was fine.
I'm surprised you don't have any collocation statements in this 'working' config - maybe why you had occasional issues.
HTH
Jake
>
> node aztestc1 \
> attributes standby="off"
> node aztestc2 \
> attributes standby="off"
> primitive p_drbd_share ocf:linbit:drbd \
> params drbd_resource="share" \
> op monitor interval="15s" role="Master" timeout="20s" \
> op monitor interval="20s" role="Slave" timeout="20s" \
> op start interval="0" timeout="240s" \
> op stop interval="0" timeout="100s"
> primitive p_fs_share ocf:heartbeat:Filesystem \
> params device="/dev/drbd/by-res/share" directory="/share"
> fstype="ocfs2" options="rw,noatime" \
> op start interval="0" timeout="60" \
> op stop interval="0" timeout="60" \
> op monitor interval="20" timeout="40"
> primitive p_o2cb ocf:pacemaker:o2cb \
> params stack="cman" \
> op start interval="0" timeout="90" \
> op stop interval="0" timeout="100" \
> op monitor interval="10" timeout="20"
> primitive p_stonith stonith:fence_ec2 \
> params pcmk_host_check="static-list" pcmk_host_list="aztestc1
> aztestc2" \
> op monitor interval="600s" timeout="300s" \
> op start start-delay="10s" interval="0"
> ms ms_drbd_share p_drbd_share \
> meta master-max="2" notify="true" interleave="true" clone-max="2"
> is-managed="true" target-role="Started"
> clone cl_fencing p_stonith \
> meta target-role="Started"
> clone cl_fs_share p_fs_share \
> meta interleave="true" notify="true" globally-unique="false"
> target-role="Started"
> clone cl_o2cb p_o2cb \
> meta interleave="true" globally-unique="false"
> order o_ocfs2 inf: ms_drbd_share:promote cl_o2cb
> order o_share inf: cl_o2cb cl_fs_share
> property $id="cib-bootstrap-options" \
> stonith-enabled="true" \
> stonith-timeout="180s" \
> dc-version="1.1.6-9971ebba4494012a93c03b40a2c58ec0eb60f50c" \
> cluster-infrastructure="cman" \
> last-lrm-refresh="1354808774"
>
>
> Thoughts? Ideas? Suggestions?
>
> Thank you,
> -- Art Z.
>
> --
> Art Zemon, President
> [http://www.hens-teeth.net/] Hen's Teeth Network for reliable web
> hosting and programming
> (866)HENS-NET / (636)447-3030 ext. 200 / www.hens-teeth.net
>
>
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started:
> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>
More information about the Pacemaker
mailing list