[Pacemaker] Nodes will not promote DRBD resources to master on failover
Andreas Kurz
andreas at hastexo.com
Tue Apr 10 10:28:15 UTC 2012
On 04/10/2012 06:17 AM, Andrew Martin wrote:
> Hi Andreas,
>
> Yes, I attempted to generalize hostnames and usernames/passwords in the
> archive. Sorry for making it more confusing :(
>
> I completely purged pacemaker from all 3 nodes and reinstalled
> everything. I then completely rebuild the CIB by manually adding in each
> primitive/constraint one at a time and testing along the way. After
> doing this DRBD appears to be working at least somewhat better - the
> ocf:linbit:drbd devices are started and managed by pacemaker. However,
> if for example a node is STONITHed when it comes back up it will not
> restart the ocf:linbit:drbd resources until I manually load the DRBD
> kernel module, bring the DRBD devices up (drbdadm up all), and cleanup
> the resources (e.g. crm resource cleanup ms_drbd_vmstore). Is it
> possible that the DRBD kernel module needs to be loaded at boot time,
> independent of pacemaker?
No, this is done by the drbd OCF script on start.
>
> Here's the new CIB (mostly the same as before):
> http://pastebin.com/MxrqBXMp
>
> Typically quorumnode stays in the OFFLINE (standby) state, though
> occasionally it changes to pending. I have just tried
> cleaning /var/lib/heartbeat/crm on quorumnode again so we will see if
> that helps keep it in the OFFLINE (standby) state. I have it explicitly
> set to standby in the CIB configuration and also created a rule to
> prevent some of the resources from running on it?
> node $id="c4bf25d7-a6b7-4863-984d-aafd937c0da4" quorumnode \
> attributes standby="on"
> ...
The node should be in "ONLINE (standby)" state if you start heartbeat
and pacemaker is enabled with "crm yes" or "crm respawn"in ha.cf
> location loc_not_on_quorumnode g_vm -inf: quorumnode
>
> Would it be wise to create additional constraints to prevent all
> resources (including each ms_drbd resource) from running on it, even
> though this should be implied by standby?
There is no need for that. A node in standby will never run resources
and if there is no DRBD and installed on that node your resources won't
start anyways.
>
> Below is a portion of the log from when I started a node yet DRBD failed
> to start. As you can see it thinks the DRBD device is operating
> correctly as it proceeds to starting subsequent resources, e.g.
> Apr 9 20:22:55 node1 Filesystem[2939]: [2956]: WARNING: Couldn't find
> device [/dev/drbd0]. Expected /dev/??? to exist
> http://pastebin.com/zTCHPtWy
The only thing i can read from that log fragments is, that probes are
running ... not enough information. Really interesting would be logs
from the DC.
>
> After seeing these messages in the log I run
> # service drbd start
> # drbdadm up all
> # crm resource cleanup ms_drbd_vmstore
> # crm resource cleanup ms_drbd_mount1
> # crm resource clenaup ms_drbd_mount2
That should all not be needed ... what is the output of "crm_mon -1frA"
before you do all that cleanups?
> After this sequence of commands the DRBD resources appear to be
> functioning normally and the subsequent resources start. Any ideas on
> why DRBD is not being started as expected, or why the cluster is
> continuing with starting resources that according to the o_drbd-fs-vm
> constraint should not start until DRBD is master?
No idea, maybe creating a crm_report archive and sending it to the list
can shed some light on that problem.
Regards,
Andreas
--
Need help with Pacemaker?
http://www.hastexo.com/now
>
> Thanks,
>
> Andrew
> ------------------------------------------------------------------------
> *From: *"Andreas Kurz" <andreas at hastexo.com>
> *To: *pacemaker at oss.clusterlabs.org
> *Sent: *Monday, April 2, 2012 6:33:44 PM
> *Subject: *Re: [Pacemaker] Nodes will not promote DRBD resources to
> master on failover
>
> On 04/02/2012 05:47 PM, Andrew Martin wrote:
>> Hi Andreas,
>>
>> Here is the crm_report:
>> http://dl.dropbox.com/u/2177298/pcmk-Mon-02-Apr-2012.bz2
>
> You tried to do some obfuscation on parts of that archive? ... doesn't
> really make it easier to debug ....
>
> Does the third node ever change its state?
>
> Node quorumnode (c4bf25d7-a6b7-4863-984d-aafd937c0da4): pending
>
> Looking at the logs and the transition graph says it aborts due to
> un-runable operations on that node which seems to be related to it's
> pending state.
>
> Try to get that node up (or down) completely ... maybe a fresh
> start-over with a clean /var/lib/heartbeat/crm directory is sufficient.
>
> Regards,
> Andreas
>
>>
>> Hi Emmanuel,
>>
>> Here is the configuration:
>> node $id="6553a515-273e-42fe-ab9e-00f74bd582c3" node1 \
>> attributes standby="off"
>> node $id="9100538b-7a1f-41fd-9c1a-c6b4b1c32b18" node2 \
>> attributes standby="off"
>> node $id="c4bf25d7-a6b7-4863-984d-aafd937c0da4" quorumnode \
>> attributes standby="on"
>> primitive p_drbd_mount2 ocf:linbit:drbd \
>> params drbd_resource="mount2" \
>> op start interval="0" timeout="240" \
>> op stop interval="0" timeout="100" \
>> op monitor interval="10" role="Master" timeout="20" start-delay="1m" \
>> op monitor interval="20" role="Slave" timeout="20" start-delay="1m"
>> primitive p_drbd_mount1 ocf:linbit:drbd \
>> params drbd_resource="mount1" \
>> op start interval="0" timeout="240" \
>> op stop interval="0" timeout="100" \
>> op monitor interval="10" role="Master" timeout="20" start-delay="1m" \
>> op monitor interval="20" role="Slave" timeout="20" start-delay="1m"
>> primitive p_drbd_vmstore ocf:linbit:drbd \
>> params drbd_resource="vmstore" \
>> op start interval="0" timeout="240" \
>> op stop interval="0" timeout="100" \
>> op monitor interval="10" role="Master" timeout="20" start-delay="1m" \
>> op monitor interval="20" role="Slave" timeout="20" start-delay="1m"
>> primitive p_fs_vmstore ocf:heartbeat:Filesystem \
>> params device="/dev/drbd0" directory="/mnt/storage/vmstore"
> fstype="ext4" \
>> op start interval="0" timeout="60s" \
>> op stop interval="0" timeout="60s" \
>> op monitor interval="20s" timeout="40s"
>> primitive p_libvirt-bin upstart:libvirt-bin \
>> op monitor interval="30"
>> primitive p_ping ocf:pacemaker:ping \
>> params name="p_ping" host_list="192.168.3.1 192.168.3.2"
> multiplier="1000" \
>> op monitor interval="20s"
>> primitive p_sysadmin_notify ocf:heartbeat:MailTo \
>> params email="me at example.com" \
>> params subject="Pacemaker Change" \
>> op start interval="0" timeout="10" \
>> op stop interval="0" timeout="10" \
>> op monitor interval="10" timeout="10"
>> primitive p_vm ocf:heartbeat:VirtualDomain \
>> params config="/mnt/storage/vmstore/config/vm.xml" \
>> meta allow-migrate="false" \
>> op start interval="0" timeout="180" \
>> op stop interval="0" timeout="180" \
>> op monitor interval="10" timeout="30"
>> primitive stonith-node1 stonith:external/tripplitepdu \
>> params pdu_ipaddr="192.168.3.100" pdu_port="1" pdu_username="xxx"
>> pdu_password="xxx" hostname_to_stonith="node1"
>> primitive stonith-node2 stonith:external/tripplitepdu \
>> params pdu_ipaddr="192.168.3.100" pdu_port="2" pdu_username="xxx"
>> pdu_password="xxx" hostname_to_stonith="node2"
>> group g_daemons p_libvirt-bin
>> group g_vm p_fs_vmstore p_vm
>> ms ms_drbd_mount2 p_drbd_mount2 \
>> meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1"
>> notify="true"
>> ms ms_drbd_mount1 p_drbd_mount1 \
>> meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1"
>> notify="true"
>> ms ms_drbd_vmstore p_drbd_vmstore \
>> meta master-max="1" master-node-max="1" clone-max="2" clone-node-max="1"
>> notify="true"
>> clone cl_daemons g_daemons
>> clone cl_ping p_ping \
>> meta interleave="true"
>> clone cl_sysadmin_notify p_sysadmin_notify \
>> meta target-role="Started"
>> location l-st-node1 stonith-node1 -inf: node1
>> location l-st-node2 stonith-node2 -inf: node2
>> location l_run_on_most_connected p_vm \
>> rule $id="l_run_on_most_connected-rule" p_ping: defined p_ping
>> colocation c_drbd_libvirt_vm inf: ms_drbd_vmstore:Master
>> ms_drbd_mount1:Master ms_drbd_mount2:Master g_vm
>> order o_drbd-fs-vm inf: ms_drbd_vmstore:promote ms_drbd_mount1:promote
>> ms_drbd_mount2:promote cl_daemons:start g_vm:start
>> property $id="cib-bootstrap-options" \
>> dc-version="1.1.6-9971ebba4494012a93c03b40a2c58ec0eb60f50c" \
>> cluster-infrastructure="Heartbeat" \
>> stonith-enabled="true" \
>> no-quorum-policy="freeze" \
>> last-lrm-refresh="1333041002" \
>> cluster-recheck-interval="5m" \
>> crmd-integration-timeout="3m" \
>> shutdown-escalation="5m"
>>
>> Thanks,
>>
>> Andrew
>>
>>
>> ------------------------------------------------------------------------
>> *From: *"emmanuel segura" <emi2fast at gmail.com>
>> *To: *"The Pacemaker cluster resource manager"
>> <pacemaker at oss.clusterlabs.org>
>> *Sent: *Monday, April 2, 2012 9:43:20 AM
>> *Subject: *Re: [Pacemaker] Nodes will not promote DRBD resources to
>> master on failover
>>
>> Sorry Andrew
>>
>> Can you post me your crm configure show again?
>>
>> Thanks
>>
>> Il giorno 30 marzo 2012 18:53, Andrew Martin <amartin at xes-inc.com
>> <mailto:amartin at xes-inc.com>> ha scritto:
>>
>> Hi Emmanuel,
>>
>> Thanks, that is a good idea. I updated the colocation contraint as
>> you described. After, the cluster remains in this state (with the
>> filesystem not mounted and the VM not started):
>> Online: [ node2 node1 ]
>>
>> Master/Slave Set: ms_drbd_vmstore [p_drbd_vmstore]
>> Masters: [ node1 ]
>> Slaves: [ node2 ]
>> Master/Slave Set: ms_drbd_tools [p_drbd_mount1]
>> Masters: [ node1 ]
>> Slaves: [ node2 ]
>> Master/Slave Set: ms_drbd_crm [p_drbd_mount2]
>> Masters: [ node1 ]
>> Slaves: [ node2 ]
>> Clone Set: cl_daemons [g_daemons]
>> Started: [ node2 node1 ]
>> Stopped: [ g_daemons:2 ]
>> stonith-node1 (stonith:external/tripplitepdu): Started node2
>> stonith-node2 (stonith:external/tripplitepdu): Started node1
>>
>> I noticed that Pacemaker had not issued "drbdadm connect" for any of
>> the DRBD resources on node2
>> # service drbd status
>> drbd driver loaded OK; device status:
>> version: 8.3.7 (api:88/proto:86-91)
>> GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by
>> root at node2, 2012-02-02 12:29:26
>> m:res cs ro ds p
>> mounted fstype
>> 0:vmstore StandAlone Secondary/Unknown Outdated/DUnknown r----
>> 1:mount1 StandAlone Secondary/Unknown Outdated/DUnknown r----
>> 2:mount2 StandAlone Secondary/Unknown Outdated/DUnknown r----
>> # drbdadm cstate all
>> StandAlone
>> StandAlone
>> StandAlone
>>
>> After manually issuing "drbdadm connect all" on node2 the rest of
>> the resources eventually started (several minutes later) on node1:
>> Online: [ node2 node1 ]
>>
>> Master/Slave Set: ms_drbd_vmstore [p_drbd_vmstore]
>> Masters: [ node1 ]
>> Slaves: [ node2 ]
>> Master/Slave Set: ms_drbd_mount1 [p_drbd_mount1]
>> Masters: [ node1 ]
>> Slaves: [ node2 ]
>> Master/Slave Set: ms_drbd_mount2 [p_drbd_mount2]
>> Masters: [ node1 ]
>> Slaves: [ node2 ]
>> Resource Group: g_vm
>> p_fs_vmstore (ocf::heartbeat:Filesystem): Started node1
>> p_vm (ocf::heartbeat:VirtualDomain): Started node1
>> Clone Set: cl_daemons [g_daemons]
>> Started: [ node2 node1 ]
>> Stopped: [ g_daemons:2 ]
>> Clone Set: cl_sysadmin_notify [p_sysadmin_notify]
>> Started: [ node2 node1 ]
>> Stopped: [ p_sysadmin_notify:2 ]
>> stonith-node1 (stonith:external/tripplitepdu): Started node2
>> stonith-node2 (stonith:external/tripplitepdu): Started node1
>> Clone Set: cl_ping [p_ping]
>> Started: [ node2 node1 ]
>> Stopped: [ p_ping:2 ]
>>
>> The DRBD devices on node1 were all UpToDate, so it doesn't seem
>> right that it would need to wait for node2 to be connected before it
>> could continue promoting additional resources. I then restarted
>> heartbeat on node2 to see if it would automatically connect the DRBD
>> devices this time. After restarting it, the DRBD devices are not
>> even configured:
>> # service drbd status
>> drbd driver loaded OK; device status:
>> version: 8.3.7 (api:88/proto:86-91)
>> GIT-hash: ea9e28dbff98e331a62bcbcc63a6135808fe2917 build by
>> root at webapps2host, 2012-02-02 12:29:26
>> m:res cs ro ds p mounted fstype
>> 0:vmstore Unconfigured
>> 1:mount1 Unconfigured
>> 2:mount2 Unconfigured
>>
>> Looking at the log I found this part about the drbd primitives:
>> Mar 30 11:10:32 node2 lrmd: [10702]: info: operation monitor[2] on
>> p_drbd_vmstore:1 for client 10705: pid 11065 exited with return code 7
>> Mar 30 11:10:32 node2 crmd: [10705]: info: process_lrm_event: LRM
>> operation p_drbd_vmstore:1_monitor_0 (call=2, rc=7, cib-update=11,
>> confirmed=true) not running
>> Mar 30 11:10:32 node2 lrmd: [10702]: info: operation monitor[4] on
>> p_drbd_mount2:1 for client 10705: pid 11069 exited with return code 7
>> Mar 30 11:10:32 node2 crmd: [10705]: info: process_lrm_event: LRM
>> operation p_drbd_mount2:1_monitor_0 (call=4, rc=7, cib-update=12,
>> confirmed=true) not running
>> Mar 30 11:10:32 node2 lrmd: [10702]: info: operation monitor[3] on
>> p_drbd_mount1:1 for client 10705: pid 11066 exited with return code 7
>> Mar 30 11:10:32 node2 crmd: [10705]: info: process_lrm_event: LRM
>> operation p_drbd_mount1:1_monitor_0 (call=3, rc=7, cib-update=13,
>> confirmed=true) not running
>>
>> I am not sure what exit code 7 is - is it possible to manually run
>> the monitor code or somehow obtain more debug about this? Here is
>> the complete log after restarting heartbeat on node2:
>> http://pastebin.com/KsHKi3GW
>>
>> Thanks,
>>
>> Andrew
>>
>>
> ------------------------------------------------------------------------
>> *From: *"emmanuel segura" <emi2fast at gmail.com
>> <mailto:emi2fast at gmail.com>>
>> *To: *"The Pacemaker cluster resource manager"
>> <pacemaker at oss.clusterlabs.org <mailto:pacemaker at oss.clusterlabs.org>>
>> *Sent: *Friday, March 30, 2012 10:26:48 AM
>> *Subject: *Re: [Pacemaker] Nodes will not promote DRBD resources to
>> master on failover
>>
>> I think this constrain it's wrong
>> ==================================================
>> colocation c_drbd_libvirt_vm inf: ms_drbd_vmstore:Master
>> ms_drbd_mount1:Master ms_drbd_mount2:Master g_vm
>> ===================================================
>>
>> change to
>> ======================================================
>> colocation c_drbd_libvirt_vm inf: g_vm ms_drbd_vmstore:Master
>> ms_drbd_mount1:Master ms_drbd_mount2:Master
>> =======================================================
>>
>> Il giorno 30 marzo 2012 17:16, Andrew Martin <amartin at xes-inc.com
>> <mailto:amartin at xes-inc.com>> ha scritto:
>>
>> Hi Emmanuel,
>>
>> Here is the output of crm configure show:
>> http://pastebin.com/NA1fZ8dL
>>
>> Thanks,
>>
>> Andrew
>>
>>
> ------------------------------------------------------------------------
>> *From: *"emmanuel segura" <emi2fast at gmail.com
>> <mailto:emi2fast at gmail.com>>
>> *To: *"The Pacemaker cluster resource manager"
>> <pacemaker at oss.clusterlabs.org
>> <mailto:pacemaker at oss.clusterlabs.org>>
>> *Sent: *Friday, March 30, 2012 9:43:45 AM
>> *Subject: *Re: [Pacemaker] Nodes will not promote DRBD resources
>> to master on failover
>>
>> can you show me?
>>
>> crm configure show
>>
>> Il giorno 30 marzo 2012 16:10, Andrew Martin
>> <amartin at xes-inc.com <mailto:amartin at xes-inc.com>> ha scritto:
>>
>> Hi Andreas,
>>
>> Here is a copy of my complete CIB:
>> http://pastebin.com/v5wHVFuy
>>
>> I'll work on generating a report using crm_report as well.
>>
>> Thanks,
>>
>> Andrew
>>
>>
> ------------------------------------------------------------------------
>> *From: *"Andreas Kurz" <andreas at hastexo.com
>> <mailto:andreas at hastexo.com>>
>> *To: *pacemaker at oss.clusterlabs.org
>> <mailto:pacemaker at oss.clusterlabs.org>
>> *Sent: *Friday, March 30, 2012 4:41:16 AM
>> *Subject: *Re: [Pacemaker] Nodes will not promote DRBD
>> resources to master on failover
>>
>> On 03/28/2012 04:56 PM, Andrew Martin wrote:
>> > Hi Andreas,
>> >
>> > I disabled the DRBD init script and then restarted the
>> slave node
>> > (node2). After it came back up, DRBD did not start:
>> > Node quorumnode (c4bf25d7-a6b7-4863-984d-aafd937c0da4):
>> pending
>> > Online: [ node2 node1 ]
>> >
>> > Master/Slave Set: ms_drbd_vmstore [p_drbd_vmstore]
>> > Masters: [ node1 ]
>> > Stopped: [ p_drbd_vmstore:1 ]
>> > Master/Slave Set: ms_drbd_mount1 [p_drbd_tools]
>> > Masters: [ node1 ]
>> > Stopped: [ p_drbd_mount1:1 ]
>> > Master/Slave Set: ms_drbd_mount2 [p_drbdmount2]
>> > Masters: [ node1 ]
>> > Stopped: [ p_drbd_mount2:1 ]
>> > ...
>> >
>> > root at node2:~# service drbd status
>> > drbd not loaded
>>
>> Yes, expected unless Pacemaker starts DRBD
>>
>> >
>> > Is there something else I need to change in the CIB to
>> ensure that DRBD
>> > is started? All of my DRBD devices are configured like this:
>> > primitive p_drbd_mount2 ocf:linbit:drbd \
>> > params drbd_resource="mount2" \
>> > op monitor interval="15" role="Master" \
>> > op monitor interval="30" role="Slave"
>> > ms ms_drbd_mount2 p_drbd_mount2 \
>> > meta master-max="1" master-node-max="1"
> clone-max="2"
>> > clone-node-max="1" notify="true"
>>
>> That should be enough ... unable to say more without seeing
>> the complete
>> configuration ... too much fragments of information ;-)
>>
>> Please provide (e.g. pastebin) your complete cib (cibadmin
>> -Q) when
>> cluster is in that state ... or even better create a
>> crm_report archive
>>
>> >
>> > Here is the output from the syslog (grep -i drbd
>> /var/log/syslog):
>> > Mar 28 09:24:47 node2 crmd: [3213]: info: do_lrm_rsc_op:
>> Performing
>> > key=12:315:7:24416169-73ba-469b-a2e3-56a22b437cbc
>> > op=p_drbd_vmstore:1_monitor_0 )
>> > Mar 28 09:24:47 node2 lrmd: [3210]: info:
>> rsc:p_drbd_vmstore:1 probe[2]
>> > (pid 3455)
>> > Mar 28 09:24:47 node2 crmd: [3213]: info: do_lrm_rsc_op:
>> Performing
>> > key=13:315:7:24416169-73ba-469b-a2e3-56a22b437cbc
>> > op=p_drbd_mount1:1_monitor_0 )
>> > Mar 28 09:24:48 node2 lrmd: [3210]: info:
>> rsc:p_drbd_mount1:1 probe[3]
>> > (pid 3456)
>> > Mar 28 09:24:48 node2 crmd: [3213]: info: do_lrm_rsc_op:
>> Performing
>> > key=14:315:7:24416169-73ba-469b-a2e3-56a22b437cbc
>> > op=p_drbd_mount2:1_monitor_0 )
>> > Mar 28 09:24:48 node2 lrmd: [3210]: info:
>> rsc:p_drbd_mount2:1 probe[4]
>> > (pid 3457)
>> > Mar 28 09:24:48 node2 Filesystem[3458]: [3517]: WARNING:
>> Couldn't find
>> > device [/dev/drbd0]. Expected /dev/??? to exist
>> > Mar 28 09:24:48 node2 crm_attribute: [3563]: info: Invoked:
>> > crm_attribute -N node2 -n master-p_drbd_mount2:1 -l
> reboot -D
>> > Mar 28 09:24:48 node2 crm_attribute: [3557]: info: Invoked:
>> > crm_attribute -N node2 -n master-p_drbd_vmstore:1 -l
> reboot -D
>> > Mar 28 09:24:48 node2 crm_attribute: [3562]: info: Invoked:
>> > crm_attribute -N node2 -n master-p_drbd_mount1:1 -l
> reboot -D
>> > Mar 28 09:24:48 node2 lrmd: [3210]: info: operation
>> monitor[4] on
>> > p_drbd_mount2:1 for client 3213: pid 3457 exited with
>> return code 7
>> > Mar 28 09:24:48 node2 lrmd: [3210]: info: operation
>> monitor[2] on
>> > p_drbd_vmstore:1 for client 3213: pid 3455 exited with
>> return code 7
>> > Mar 28 09:24:48 node2 crmd: [3213]: info:
>> process_lrm_event: LRM
>> > operation p_drbd_mount2:1_monitor_0 (call=4, rc=7,
>> cib-update=10,
>> > confirmed=true) not running
>> > Mar 28 09:24:48 node2 lrmd: [3210]: info: operation
>> monitor[3] on
>> > p_drbd_mount1:1 for client 3213: pid 3456 exited with
>> return code 7
>> > Mar 28 09:24:48 node2 crmd: [3213]: info:
>> process_lrm_event: LRM
>> > operation p_drbd_vmstore:1_monitor_0 (call=2, rc=7,
>> cib-update=11,
>> > confirmed=true) not running
>> > Mar 28 09:24:48 node2 crmd: [3213]: info:
>> process_lrm_event: LRM
>> > operation p_drbd_mount1:1_monitor_0 (call=3, rc=7,
>> cib-update=12,
>> > confirmed=true) not running
>>
>> No errors, just probing ... so for any reason Pacemaker does
>> not like to
>> start it ... use crm_simulate to find out why ... or provide
>> information
>> as requested above.
>>
>> Regards,
>> Andreas
>>
>> --
>> Need help with Pacemaker?
>> http://www.hastexo.com/now
>>
>> >
>> > Thanks,
>> >
>> > Andrew
>> >
>> >
>>
> ------------------------------------------------------------------------
>> > *From: *"Andreas Kurz" <andreas at hastexo.com
>> <mailto:andreas at hastexo.com>>
>> > *To: *pacemaker at oss.clusterlabs.org
>> <mailto:pacemaker at oss.clusterlabs.org>
>> > *Sent: *Wednesday, March 28, 2012 9:03:06 AM
>> > *Subject: *Re: [Pacemaker] Nodes will not promote DRBD
>> resources to
>> > master on failover
>> >
>> > On 03/28/2012 03:47 PM, Andrew Martin wrote:
>> >> Hi Andreas,
>> >>
>> >>> hmm ... what is that fence-peer script doing? If you
>> want to use
>> >>> resource-level fencing with the help of dopd, activate the
>> >>> drbd-peer-outdater script in the line above ... and
>> double check if the
>> >>> path is correct
>> >> fence-peer is just a wrapper for drbd-peer-outdater that
>> does some
>> >> additional logging. In my testing dopd has been working
> well.
>> >
>> > I see
>> >
>> >>
>> >>>> I am thinking of making the following changes to the
>> CIB (as per the
>> >>>> official DRBD
>> >>>> guide
>> >>
>> >
>>
> http://www.drbd.org/users-guide/s-pacemaker-crm-drbd-backed-service.html)
>> in
>> >>>> order to add the DRBD lsb service and require that it
>> start before the
>> >>>> ocf:linbit:drbd resources. Does this look correct?
>> >>>
>> >>> Where did you read that? No, deactivate the startup of
>> DRBD on system
>> >>> boot and let Pacemaker manage it completely.
>> >>>
>> >>>> primitive p_drbd-init lsb:drbd op monitor interval="30"
>> >>>> colocation c_drbd_together inf:
>> >>>> p_drbd-init ms_drbd_vmstore:Master ms_drbd_mount1:Master
>> >>>> ms_drbd_mount2:Master
>> >>>> order drbd_init_first inf: ms_drbd_vmstore:promote
>> >>>> ms_drbd_mount1:promote ms_drbd_mount2:promote
>> p_drbd-init:start
>> >>>>
>> >>>> This doesn't seem to require that drbd be also running
>> on the node where
>> >>>> the ocf:linbit:drbd resources are slave (which it would
>> need to do to be
>> >>>> a DRBD SyncTarget) - how can I ensure that drbd is
>> running everywhere?
>> >>>> (clone cl_drbd p_drbd-init ?)
>> >>>
>> >>> This is really not needed.
>> >> I was following the official DRBD Users Guide:
>> >>
>>
> http://www.drbd.org/users-guide/s-pacemaker-crm-drbd-backed-service.html
>> >>
>> >> If I am understanding your previous message correctly, I
>> do not need to
>> >> add a lsb primitive for the drbd daemon? It will be
>> >> started/stopped/managed automatically by my
>> ocf:linbit:drbd resources
>> >> (and I can remove the /etc/rc* symlinks)?
>> >
>> > Yes, you don't need that LSB script when using Pacemaker
>> and should not
>> > let init start it.
>> >
>> > Regards,
>> > Andreas
>> >
>> > --
>> > Need help with Pacemaker?
>> > http://www.hastexo.com/now
>> >
>> >>
>> >> Thanks,
>> >>
>> >> Andrew
>> >>
>> >>
>>
> ------------------------------------------------------------------------
>> >> *From: *"Andreas Kurz" <andreas at hastexo.com
>> <mailto:andreas at hastexo.com> <mailto:andreas at hastexo.com
>> <mailto:andreas at hastexo.com>>>
>> >> *To: *pacemaker at oss.clusterlabs.org
>> <mailto:pacemaker at oss.clusterlabs.org>
>> <mailto:pacemaker at oss.clusterlabs.org
>> <mailto:pacemaker at oss.clusterlabs.org>>
>> >> *Sent: *Wednesday, March 28, 2012 7:27:34 AM
>> >> *Subject: *Re: [Pacemaker] Nodes will not promote DRBD
>> resources to
>> >> master on failover
>> >>
>> >> On 03/28/2012 12:13 AM, Andrew Martin wrote:
>> >>> Hi Andreas,
>> >>>
>> >>> Thanks, I've updated the colocation rule to be in the
>> correct order. I
>> >>> also enabled the STONITH resource (this was temporarily
>> disabled before
>> >>> for some additional testing). DRBD has its own network
>> connection over
>> >>> the br1 interface (192.168.5.0/24
>> <http://192.168.5.0/24> network), a direct crossover cable
>> >>> between node1 and node2:
>> >>> global { usage-count no; }
>> >>> common {
>> >>> syncer { rate 110M; }
>> >>> }
>> >>> resource vmstore {
>> >>> protocol C;
>> >>> startup {
>> >>> wfc-timeout 15;
>> >>> degr-wfc-timeout 60;
>> >>> }
>> >>> handlers {
>> >>> #fence-peer
>> "/usr/lib/heartbeat/drbd-peer-outdater -t 5";
>> >>> fence-peer "/usr/local/bin/fence-peer";
>> >>
>> >> hmm ... what is that fence-peer script doing? If you want
>> to use
>> >> resource-level fencing with the help of dopd, activate the
>> >> drbd-peer-outdater script in the line above ... and
>> double check if the
>> >> path is correct
>> >>
>> >>> split-brain
>> "/usr/lib/drbd/notify-split-brain.sh
>> >>> me at example.com <mailto:me at example.com>
>> <mailto:me at example.com <mailto:me at example.com>>";
>> >>> }
>> >>> net {
>> >>> after-sb-0pri discard-zero-changes;
>> >>> after-sb-1pri discard-secondary;
>> >>> after-sb-2pri disconnect;
>> >>> cram-hmac-alg md5;
>> >>> shared-secret "xxxxx";
>> >>> }
>> >>> disk {
>> >>> fencing resource-only;
>> >>> }
>> >>> on node1 {
>> >>> device /dev/drbd0;
>> >>> disk /dev/sdb1;
>> >>> address 192.168.5.10:7787
>> <http://192.168.5.10:7787>;
>> >>> meta-disk internal;
>> >>> }
>> >>> on node2 {
>> >>> device /dev/drbd0;
>> >>> disk /dev/sdf1;
>> >>> address 192.168.5.11:7787
>> <http://192.168.5.11:7787>;
>> >>> meta-disk internal;
>> >>> }
>> >>> }
>> >>> # and similar for mount1 and mount2
>> >>>
>> >>> Also, here is my ha.cf <http://ha.cf>. It uses both the
>> direct link between the nodes
>> >>> (br1) and the shared LAN network on br0 for communicating:
>> >>> autojoin none
>> >>> mcast br0 239.0.0.43 694 1 0
>> >>> bcast br1
>> >>> warntime 5
>> >>> deadtime 15
>> >>> initdead 60
>> >>> keepalive 2
>> >>> node node1
>> >>> node node2
>> >>> node quorumnode
>> >>> crm respawn
>> >>> respawn hacluster /usr/lib/heartbeat/dopd
>> >>> apiauth dopd gid=haclient uid=hacluster
>> >>>
>> >>> I am thinking of making the following changes to the CIB
>> (as per the
>> >>> official DRBD
>> >>> guide
>> >>
>> >
>>
> http://www.drbd.org/users-guide/s-pacemaker-crm-drbd-backed-service.html)
>> in
>> >>> order to add the DRBD lsb service and require that it
>> start before the
>> >>> ocf:linbit:drbd resources. Does this look correct?
>> >>
>> >> Where did you read that? No, deactivate the startup of
>> DRBD on system
>> >> boot and let Pacemaker manage it completely.
>> >>
>> >>> primitive p_drbd-init lsb:drbd op monitor interval="30"
>> >>> colocation c_drbd_together inf:
>> >>> p_drbd-init ms_drbd_vmstore:Master ms_drbd_mount1:Master
>> >>> ms_drbd_mount2:Master
>> >>> order drbd_init_first inf: ms_drbd_vmstore:promote
>> >>> ms_drbd_mount1:promote ms_drbd_mount2:promote
>> p_drbd-init:start
>> >>>
>> >>> This doesn't seem to require that drbd be also running
>> on the node where
>> >>> the ocf:linbit:drbd resources are slave (which it would
>> need to do to be
>> >>> a DRBD SyncTarget) - how can I ensure that drbd is
>> running everywhere?
>> >>> (clone cl_drbd p_drbd-init ?)
>> >>
>> >> This is really not needed.
>> >>
>> >> Regards,
>> >> Andreas
>> >>
>> >> --
>> >> Need help with Pacemaker?
>> >> http://www.hastexo.com/now
>> >>
>> >>>
>> >>> Thanks,
>> >>>
>> >>> Andrew
>> >>>
>>
> ------------------------------------------------------------------------
>> >>> *From: *"Andreas Kurz" <andreas at hastexo.com
>> <mailto:andreas at hastexo.com> <mailto:andreas at hastexo.com
>> <mailto:andreas at hastexo.com>>>
>> >>> *To: *pacemaker at oss.clusterlabs.org
>> <mailto:pacemaker at oss.clusterlabs.org>
>> > <mailto:*pacemaker at oss.clusterlabs.org
>> <mailto:pacemaker at oss.clusterlabs.org>>
>> >>> *Sent: *Monday, March 26, 2012 5:56:22 PM
>> >>> *Subject: *Re: [Pacemaker] Nodes will not promote DRBD
>> resources to
>> >>> master on failover
>> >>>
>> >>> On 03/24/2012 08:15 PM, Andrew Martin wrote:
>> >>>> Hi Andreas,
>> >>>>
>> >>>> My complete cluster configuration is as follows:
>> >>>> ============
>> >>>> Last updated: Sat Mar 24 13:51:55 2012
>> >>>> Last change: Sat Mar 24 13:41:55 2012
>> >>>> Stack: Heartbeat
>> >>>> Current DC: node2
>> (9100538b-7a1f-41fd-9c1a-c6b4b1c32b18) - partition
>> >>>> with quorum
>> >>>> Version: 1.1.6-9971ebba4494012a93c03b40a2c58ec0eb60f50c
>> >>>> 3 Nodes configured, unknown expected votes
>> >>>> 19 Resources configured.
>> >>>> ============
>> >>>>
>> >>>> Node quorumnode (c4bf25d7-a6b7-4863-984d-aafd937c0da4):
>> OFFLINE
>> > (standby)
>> >>>> Online: [ node2 node1 ]
>> >>>>
>> >>>> Master/Slave Set: ms_drbd_vmstore [p_drbd_vmstore]
>> >>>> Masters: [ node2 ]
>> >>>> Slaves: [ node1 ]
>> >>>> Master/Slave Set: ms_drbd_mount1 [p_drbd_mount1]
>> >>>> Masters: [ node2 ]
>> >>>> Slaves: [ node1 ]
>> >>>> Master/Slave Set: ms_drbd_mount2 [p_drbd_mount2]
>> >>>> Masters: [ node2 ]
>> >>>> Slaves: [ node1 ]
>> >>>> Resource Group: g_vm
>> >>>> p_fs_vmstore(ocf::heartbeat:Filesystem):Started
> node2
>> >>>> p_vm(ocf::heartbeat:VirtualDomain):Started node2
>> >>>> Clone Set: cl_daemons [g_daemons]
>> >>>> Started: [ node2 node1 ]
>> >>>> Stopped: [ g_daemons:2 ]
>> >>>> Clone Set: cl_sysadmin_notify [p_sysadmin_notify]
>> >>>> Started: [ node2 node1 ]
>> >>>> Stopped: [ p_sysadmin_notify:2 ]
>> >>>> stonith-node1(stonith:external/tripplitepdu):Started
> node2
>> >>>> stonith-node2(stonith:external/tripplitepdu):Started
> node1
>> >>>> Clone Set: cl_ping [p_ping]
>> >>>> Started: [ node2 node1 ]
>> >>>> Stopped: [ p_ping:2 ]
>> >>>>
>> >>>> node $id="6553a515-273e-42fe-ab9e-00f74bd582c3" node1 \
>> >>>> attributes standby="off"
>> >>>> node $id="9100538b-7a1f-41fd-9c1a-c6b4b1c32b18" node2 \
>> >>>> attributes standby="off"
>> >>>> node $id="c4bf25d7-a6b7-4863-984d-aafd937c0da4"
>> quorumnode \
>> >>>> attributes standby="on"
>> >>>> primitive p_drbd_mount2 ocf:linbit:drbd \
>> >>>> params drbd_resource="mount2" \
>> >>>> op monitor interval="15" role="Master" \
>> >>>> op monitor interval="30" role="Slave"
>> >>>> primitive p_drbd_mount1 ocf:linbit:drbd \
>> >>>> params drbd_resource="mount1" \
>> >>>> op monitor interval="15" role="Master" \
>> >>>> op monitor interval="30" role="Slave"
>> >>>> primitive p_drbd_vmstore ocf:linbit:drbd \
>> >>>> params drbd_resource="vmstore" \
>> >>>> op monitor interval="15" role="Master" \
>> >>>> op monitor interval="30" role="Slave"
>> >>>> primitive p_fs_vmstore ocf:heartbeat:Filesystem \
>> >>>> params device="/dev/drbd0" directory="/vmstore"
>> fstype="ext4" \
>> >>>> op start interval="0" timeout="60s" \
>> >>>> op stop interval="0" timeout="60s" \
>> >>>> op monitor interval="20s" timeout="40s"
>> >>>> primitive p_libvirt-bin upstart:libvirt-bin \
>> >>>> op monitor interval="30"
>> >>>> primitive p_ping ocf:pacemaker:ping \
>> >>>> params name="p_ping" host_list="192.168.1.10
>> 192.168.1.11"
>> >>>> multiplier="1000" \
>> >>>> op monitor interval="20s"
>> >>>> primitive p_sysadmin_notify ocf:heartbeat:MailTo \
>> >>>> params email="me at example.com
>> <mailto:me at example.com> <mailto:me at example.com
>> <mailto:me at example.com>>" \
>> >>>> params subject="Pacemaker Change" \
>> >>>> op start interval="0" timeout="10" \
>> >>>> op stop interval="0" timeout="10" \
>> >>>> op monitor interval="10" timeout="10"
>> >>>> primitive p_vm ocf:heartbeat:VirtualDomain \
>> >>>> params config="/vmstore/config/vm.xml" \
>> >>>> meta allow-migrate="false" \
>> >>>> op start interval="0" timeout="120s" \
>> >>>> op stop interval="0" timeout="120s" \
>> >>>> op monitor interval="10" timeout="30"
>> >>>> primitive stonith-node1 stonith:external/tripplitepdu \
>> >>>> params pdu_ipaddr="192.168.1.12" pdu_port="1"
>> pdu_username="xxx"
>> >>>> pdu_password="xxx" hostname_to_stonith="node1"
>> >>>> primitive stonith-node2 stonith:external/tripplitepdu \
>> >>>> params pdu_ipaddr="192.168.1.12" pdu_port="2"
>> pdu_username="xxx"
>> >>>> pdu_password="xxx" hostname_to_stonith="node2"
>> >>>> group g_daemons p_libvirt-bin
>> >>>> group g_vm p_fs_vmstore p_vm
>> >>>> ms ms_drbd_mount2 p_drbd_mount2 \
>> >>>> meta master-max="1" master-node-max="1"
>> clone-max="2"
>> >>>> clone-node-max="1" notify="true"
>> >>>> ms ms_drbd_mount1 p_drbd_mount1 \
>> >>>> meta master-max="1" master-node-max="1"
>> clone-max="2"
>> >>>> clone-node-max="1" notify="true"
>> >>>> ms ms_drbd_vmstore p_drbd_vmstore \
>> >>>> meta master-max="1" master-node-max="1"
>> clone-max="2"
>> >>>> clone-node-max="1" notify="true"
>> >>>> clone cl_daemons g_daemons
>> >>>> clone cl_ping p_ping \
>> >>>> meta interleave="true"
>> >>>> clone cl_sysadmin_notify p_sysadmin_notify
>> >>>> location l-st-node1 stonith-node1 -inf: node1
>> >>>> location l-st-node2 stonith-node2 -inf: node2
>> >>>> location l_run_on_most_connected p_vm \
>> >>>> rule $id="l_run_on_most_connected-rule" p_ping:
>> defined p_ping
>> >>>> colocation c_drbd_libvirt_vm inf: ms_drbd_vmstore:Master
>> >>>> ms_drbd_mount1:Master ms_drbd_mount2:Master g_vm
>> >>>
>> >>> As Emmanuel already said, g_vm has to be in the first
>> place in this
>> >>> collocation constraint .... g_vm must be colocated with
>> the drbd masters.
>> >>>
>> >>>> order o_drbd-fs-vm inf: ms_drbd_vmstore:promote
>> ms_drbd_mount1:promote
>> >>>> ms_drbd_mount2:promote cl_daemons:start g_vm:start
>> >>>> property $id="cib-bootstrap-options" \
>> >>>>
>> dc-version="1.1.6-9971ebba4494012a93c03b40a2c58ec0eb60f50c" \
>> >>>> cluster-infrastructure="Heartbeat" \
>> >>>> stonith-enabled="false" \
>> >>>> no-quorum-policy="stop" \
>> >>>> last-lrm-refresh="1332539900" \
>> >>>> cluster-recheck-interval="5m" \
>> >>>> crmd-integration-timeout="3m" \
>> >>>> shutdown-escalation="5m"
>> >>>>
>> >>>> The STONITH plugin is a custom plugin I wrote for the
>> Tripp-Lite
>> >>>> PDUMH20ATNET that I'm using as the STONITH device:
>> >>>>
>>
> http://www.tripplite.com/shared/product-pages/en/PDUMH20ATNET.pdf
>> >>>
>> >>> And why don't using it? .... stonith-enabled="false"
>> >>>
>> >>>>
>> >>>> As you can see, I left the DRBD service to be started
>> by the operating
>> >>>> system (as an lsb script at boot time) however
>> Pacemaker controls
>> >>>> actually bringing up/taking down the individual DRBD
>> devices.
>> >>>
>> >>> Don't start drbd on system boot, give Pacemaker the full
>> control.
>> >>>
>> >>> The
>> >>>> behavior I observe is as follows: I issue "crm resource
>> migrate p_vm" on
>> >>>> node1 and failover successfully to node2. During this
>> time, node2 fences
>> >>>> node1's DRBD devices (using dopd) and marks them as
>> Outdated. Meanwhile
>> >>>> node2's DRBD devices are UpToDate. I then shutdown both
>> nodes and then
>> >>>> bring them back up. They reconnect to the cluster (with
>> quorum), and
>> >>>> node1's DRBD devices are still Outdated as expected and
>> node2's DRBD
>> >>>> devices are still UpToDate, as expected. At this point,
>> DRBD starts on
>> >>>> both nodes, however node2 will not set DRBD as master:
>> >>>> Node quorumnode (c4bf25d7-a6b7-4863-984d-aafd937c0da4):
>> OFFLINE
>> > (standby)
>> >>>> Online: [ node2 node1 ]
>> >>>>
>> >>>> Master/Slave Set: ms_drbd_vmstore [p_drbd_vmstore]
>> >>>> Slaves: [ node1 node2 ]
>> >>>> Master/Slave Set: ms_drbd_mount1 [p_drbd_mount1]
>> >>>> Slaves: [ node1 node 2 ]
>> >>>> Master/Slave Set: ms_drbd_mount2 [p_drbd_mount2]
>> >>>> Slaves: [ node1 node2 ]
>> >>>
>> >>> There should really be no interruption of the drbd
>> replication on vm
>> >>> migration that activates the dopd ... drbd has its own
>> direct network
>> >>> connection?
>> >>>
>> >>> Please share your ha.cf <http://ha.cf> file and your
>> drbd configuration. Watch out for
>> >>> drbd messages in your kernel log file, that should give
>> you additional
>> >>> information when/why the drbd connection was lost.
>> >>>
>> >>> Regards,
>> >>> Andreas
>> >>>
>> >>> --
>> >>> Need help with Pacemaker?
>> >>> http://www.hastexo.com/now
>> >>>
>> >>>>
>> >>>> I am having trouble sorting through the logging
>> information because
>> >>>> there is so much of it in /var/log/daemon.log, but I
>> can't find an
>> >>>> error message printed about why it will not promote
>> node2. At this point
>> >>>> the DRBD devices are as follows:
>> >>>> node2: cstate = WFConnection dstate=UpToDate
>> >>>> node1: cstate = StandAlone dstate=Outdated
>> >>>>
>> >>>> I don't see any reason why node2 can't become DRBD
>> master, or am I
>> >>>> missing something? If I do "drbdadm connect all" on
>> node1, then the
>> >>>> cstate on both nodes changes to "Connected" and node2
>> immediately
>> >>>> promotes the DRBD resources to master. Any ideas on why
>> I'm observing
>> >>>> this incorrect behavior?
>> >>>>
>> >>>> Any tips on how I can better filter through the
>> pacemaker/heartbeat logs
>> >>>> or how to get additional useful debug information?
>> >>>>
>> >>>> Thanks,
>> >>>>
>> >>>> Andrew
>> >>>>
>> >>>>
>>
> ------------------------------------------------------------------------
>> >>>> *From: *"Andreas Kurz" <andreas at hastexo.com
>> <mailto:andreas at hastexo.com>
>> > <mailto:andreas at hastexo.com <mailto:andreas at hastexo.com>>>
>> >>>> *To: *pacemaker at oss.clusterlabs.org
>> <mailto:pacemaker at oss.clusterlabs.org>
>> >> <mailto:*pacemaker at oss.clusterlabs.org
>> <mailto:pacemaker at oss.clusterlabs.org>>
>> >>>> *Sent: *Wednesday, 1 February, 2012 4:19:25 PM
>> >>>> *Subject: *Re: [Pacemaker] Nodes will not promote DRBD
>> resources to
>> >>>> master on failover
>> >>>>
>> >>>> On 01/25/2012 08:58 PM, Andrew Martin wrote:
>> >>>>> Hello,
>> >>>>>
>> >>>>> Recently I finished configuring a two-node cluster
>> with pacemaker 1.1.6
>> >>>>> and heartbeat 3.0.5 on nodes running Ubuntu 10.04.
>> This cluster
>> > includes
>> >>>>> the following resources:
>> >>>>> - primitives for DRBD storage devices
>> >>>>> - primitives for mounting the filesystem on the DRBD
>> storage
>> >>>>> - primitives for some mount binds
>> >>>>> - primitive for starting apache
>> >>>>> - primitives for starting samba and nfs servers
>> (following instructions
>> >>>>> here
>> <http://www.linbit.com/fileadmin/tech-guides/ha-nfs.pdf>)
>> >>>>> - primitives for exporting nfs shares
>> (ocf:heartbeat:exportfs)
>> >>>>
>> >>>> not enough information ... please share at least your
>> complete cluster
>> >>>> configuration
>> >>>>
>> >>>> Regards,
>> >>>> Andreas
>> >>>>
>> >>>> --
>> >>>> Need help with Pacemaker?
>> >>>> http://www.hastexo.com/now
>> >>>>
>> >>>>>
>> >>>>> Perhaps this is best described through the output of
>> crm_mon:
>> >>>>> Online: [ node1 node2 ]
>> >>>>>
>> >>>>> Master/Slave Set: ms_drbd_mount1 [p_drbd_mount1]
>> (unmanaged)
>> >>>>> p_drbd_mount1:0 (ocf::linbit:drbd):
>> Started node2
>> >>> (unmanaged)
>> >>>>> p_drbd_mount1:1 (ocf::linbit:drbd):
>> Started node1
>> >>>>> (unmanaged) FAILED
>> >>>>> Master/Slave Set: ms_drbd_mount2 [p_drbd_mount2]
>> >>>>> p_drbd_mount2:0 (ocf::linbit:drbd):
>> Master node1
>> >>>>> (unmanaged) FAILED
>> >>>>> Slaves: [ node2 ]
>> >>>>> Resource Group: g_core
>> >>>>> p_fs_mount1 (ocf::heartbeat:Filesystem):
>> Started node1
>> >>>>> p_fs_mount2 (ocf::heartbeat:Filesystem):
>> Started node1
>> >>>>> p_ip_nfs (ocf::heartbeat:IPaddr2):
>> Started node1
>> >>>>> Resource Group: g_apache
>> >>>>> p_fs_mountbind1 (ocf::heartbeat:Filesystem):
>> Started node1
>> >>>>> p_fs_mountbind2 (ocf::heartbeat:Filesystem):
>> Started node1
>> >>>>> p_fs_mountbind3 (ocf::heartbeat:Filesystem):
>> Started node1
>> >>>>> p_fs_varwww (ocf::heartbeat:Filesystem):
>> Started node1
>> >>>>> p_apache (ocf::heartbeat:apache):
>> Started node1
>> >>>>> Resource Group: g_fileservers
>> >>>>> p_lsb_smb (lsb:smbd): Started node1
>> >>>>> p_lsb_nmb (lsb:nmbd): Started node1
>> >>>>> p_lsb_nfsserver (lsb:nfs-kernel-server):
>> Started node1
>> >>>>> p_exportfs_mount1 (ocf::heartbeat:exportfs):
>> Started node1
>> >>>>> p_exportfs_mount2 (ocf::heartbeat:exportfs):
>> Started
>> > node1
>> >>>>>
>> >>>>> I have read through the Pacemaker Explained
>> >>>>>
>> >>>>
>> >>>
>> >
>>
> <http://www.clusterlabs.org/doc/en-US/Pacemaker/1.1/html-single/Pacemaker_Explained>
>> >>>>> documentation, however could not find a way to further
>> debug these
>> >>>>> problems. First, I put node1 into standby mode to
>> attempt failover to
>> >>>>> the other node (node2). Node2 appeared to start the
>> transition to
>> >>>>> master, however it failed to promote the DRBD
>> resources to master (the
>> >>>>> first step). I have attached a copy of this session in
>> commands.log and
>> >>>>> additional excerpts from /var/log/syslog during
>> important steps. I have
>> >>>>> attempted everything I can think of to try and start
>> the DRBD resource
>> >>>>> (e.g. start/stop/promote/manage/cleanup under crm
>> resource, restarting
>> >>>>> heartbeat) but cannot bring it out of the slave state.
>> However, if
>> > I set
>> >>>>> it to unmanaged and then run drbdadm primary all in
>> the terminal,
>> >>>>> pacemaker is satisfied and continues starting the rest
>> of the
>> > resources.
>> >>>>> It then failed when attempting to mount the filesystem
>> for mount2, the
>> >>>>> p_fs_mount2 resource. I attempted to mount the
>> filesystem myself
>> > and was
>> >>>>> successful. I then unmounted it and ran cleanup on
>> p_fs_mount2 and then
>> >>>>> it mounted. The rest of the resources started as
>> expected until the
>> >>>>> p_exportfs_mount2 resource, which failed as follows:
>> >>>>> p_exportfs_mount2 (ocf::heartbeat:exportfs):
>> started node2
>> >>>>> (unmanaged) FAILED
>> >>>>>
>> >>>>> I ran cleanup on this and it started, however when
>> running this test
>> >>>>> earlier today no command could successfully start this
>> exportfs
>> >> resource.
>> >>>>>
>> >>>>> How can I configure pacemaker to better resolve these
>> problems and be
>> >>>>> able to bring the node up successfully on its own?
>> What can I check to
>> >>>>> determine why these failures are occuring?
>> /var/log/syslog did not seem
>> >>>>> to contain very much useful information regarding why
>> the failures
>> >>>> occurred.
>> >>>>>
>> >>>>> Thanks,
>> >>>>>
>> >>>>> Andrew
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> This body part will be downloaded on demand.
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>>
>> >>>> _______________________________________________
>> >>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> <mailto:Pacemaker at oss.clusterlabs.org>
>> >> <mailto:Pacemaker at oss.clusterlabs.org
>> <mailto:Pacemaker at oss.clusterlabs.org>>
>> >>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>> >>>>
>> >>>> Project Home: http://www.clusterlabs.org
>> >>>> Getting started:
>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> >>>> Bugs: http://bugs.clusterlabs.org
>> >>>>
>> >>>>
>> >>>>
>> >>>> _______________________________________________
>> >>>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> <mailto:Pacemaker at oss.clusterlabs.org>
>> >> <mailto:Pacemaker at oss.clusterlabs.org
>> <mailto:Pacemaker at oss.clusterlabs.org>>
>> >>>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>> >>>>
>> >>>> Project Home: http://www.clusterlabs.org
>> >>>> Getting started:
>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> >>>> Bugs: http://bugs.clusterlabs.org
>> >>>
>> >>>
>> >>>
>> >>> _______________________________________________
>> >>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> <mailto:Pacemaker at oss.clusterlabs.org>
>> >> <mailto:Pacemaker at oss.clusterlabs.org
>> <mailto:Pacemaker at oss.clusterlabs.org>>
>> >>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>> >>>
>> >>> Project Home: http://www.clusterlabs.org
>> >>> Getting started:
>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> >>> Bugs: http://bugs.clusterlabs.org
>> >>>
>> >>>
>> >>>
>> >>> _______________________________________________
>> >>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> <mailto:Pacemaker at oss.clusterlabs.org>
>> >> <mailto:Pacemaker at oss.clusterlabs.org
>> <mailto:Pacemaker at oss.clusterlabs.org>>
>> >>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>> >>>
>> >>> Project Home: http://www.clusterlabs.org
>> >>> Getting started:
>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> >>> Bugs: http://bugs.clusterlabs.org
>> >>
>> >>
>> >>
>> >> _______________________________________________
>> >> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> <mailto:Pacemaker at oss.clusterlabs.org>
>> >> <mailto:Pacemaker at oss.clusterlabs.org
>> <mailto:Pacemaker at oss.clusterlabs.org>>
>> >> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>> >>
>> >> Project Home: http://www.clusterlabs.org
>> >> Getting started:
>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> >> Bugs: http://bugs.clusterlabs.org
>> >>
>> >>
>> >>
>> >> _______________________________________________
>> >> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> <mailto:Pacemaker at oss.clusterlabs.org>
>> >> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>> >>
>> >> Project Home: http://www.clusterlabs.org
>> >> Getting started:
>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> >> Bugs: http://bugs.clusterlabs.org
>> >
>> >
>> >
>> >
>> > _______________________________________________
>> > Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> <mailto:Pacemaker at oss.clusterlabs.org>
>> > http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>> >
>> > Project Home: http://www.clusterlabs.org
>> > Getting started:
>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> > Bugs: http://bugs.clusterlabs.org
>> >
>> >
>> >
>> > _______________________________________________
>> > Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> <mailto:Pacemaker at oss.clusterlabs.org>
>> > http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>> >
>> > Project Home: http://www.clusterlabs.org
>> > Getting started:
>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> > Bugs: http://bugs.clusterlabs.org
>>
>>
>>
>> _______________________________________________
>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> <mailto:Pacemaker at oss.clusterlabs.org>
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started:
>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>>
>>
>> _______________________________________________
>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> <mailto:Pacemaker at oss.clusterlabs.org>
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started:
>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>>
>>
>>
>>
>> --
>> esta es mi vida e me la vivo hasta que dios quiera
>>
>> _______________________________________________
>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> <mailto:Pacemaker at oss.clusterlabs.org>
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started:
>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>>
>>
>> _______________________________________________
>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> <mailto:Pacemaker at oss.clusterlabs.org>
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started:
>> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>>
>>
>>
>>
>> --
>> esta es mi vida e me la vivo hasta que dios quiera
>>
>> _______________________________________________
>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> <mailto:Pacemaker at oss.clusterlabs.org>
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started:
> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>>
>>
>> _______________________________________________
>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> <mailto:Pacemaker at oss.clusterlabs.org>
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started:
> http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>>
>>
>>
>>
>> --
>> esta es mi vida e me la vivo hasta que dios quiera
>>
>> _______________________________________________
>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>>
>>
>>
>> _______________________________________________
>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>
> --
> Need help with Pacemaker?
> http://www.hastexo.com/now
>
>
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
>
>
>
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 222 bytes
Desc: OpenPGP digital signature
URL: <https://lists.clusterlabs.org/pipermail/pacemaker/attachments/20120410/6c9d76a7/attachment-0004.sig>
More information about the Pacemaker
mailing list