[Pacemaker] Resource ordering/colocating question (DRBD + LVM + FS)

Heikki Manninen hma at iki.fi
Tue Sep 10 12:51:32 UTC 2013


Not sure whether I'm doing this the right way but here goes..

With resources started on node #1:

# crm_simulate -L -s -d pgdbsrv01.cl1.local

Current cluster status:
Online: [ pgdbsrv01.cl1.local pgdbsrv02.cl1.local ]

 Master/Slave Set: DRBD_ms_data01 [DRBD_data01]
     Masters: [ pgdbsrv01.cl1.local ]
     Slaves: [ pgdbsrv02.cl1.local ]
 Master/Slave Set: DRBD_ms_data02 [DRBD_data02]
     Masters: [ pgdbsrv01.cl1.local ]
     Slaves: [ pgdbsrv02.cl1.local ]
 Resource Group: GRP_data01
     LVM_vgdata01	(ocf::heartbeat:LVM):	Started pgdbsrv01.cl1.local
     FS_data01	(ocf::heartbeat:Filesystem):	Started pgdbsrv01.cl1.local
 Resource Group: GRP_data02
     LVM_vgdata02	(ocf::heartbeat:LVM):	Started pgdbsrv01.cl1.local
     FS_data02	(ocf::heartbeat:Filesystem):	Started pgdbsrv01.cl1.local
 fusion-fencing	(stonith:fence_fusion):	Started pgdbsrv02.cl1.local

Performing requested modifications
 + Taking node pgdbsrv01.cl1.local offline
Allocation scores:
clone_color: DRBD_ms_data01 allocation score on pgdbsrv01.cl1.local: 0
clone_color: DRBD_ms_data01 allocation score on pgdbsrv02.cl1.local: 0
clone_color: DRBD_data01:0 allocation score on pgdbsrv01.cl1.local: 0
clone_color: DRBD_data01:0 allocation score on pgdbsrv02.cl1.local: 10000
clone_color: DRBD_data01:1 allocation score on pgdbsrv01.cl1.local: 0
clone_color: DRBD_data01:1 allocation score on pgdbsrv02.cl1.local: 10000
native_color: DRBD_data01:0 allocation score on pgdbsrv01.cl1.local: -INFINITY
native_color: DRBD_data01:0 allocation score on pgdbsrv02.cl1.local: 10000
native_color: DRBD_data01:1 allocation score on pgdbsrv01.cl1.local: -INFINITY
native_color: DRBD_data01:1 allocation score on pgdbsrv02.cl1.local: -INFINITY
DRBD_data01:0 promotion score on pgdbsrv02.cl1.local: 10000
DRBD_data01:1 promotion score on none: 0
clone_color: DRBD_ms_data02 allocation score on pgdbsrv01.cl1.local: 0
clone_color: DRBD_ms_data02 allocation score on pgdbsrv02.cl1.local: 0
clone_color: DRBD_data02:0 allocation score on pgdbsrv01.cl1.local: 0
clone_color: DRBD_data02:0 allocation score on pgdbsrv02.cl1.local: 10000
clone_color: DRBD_data02:1 allocation score on pgdbsrv01.cl1.local: 0
clone_color: DRBD_data02:1 allocation score on pgdbsrv02.cl1.local: 10000
native_color: DRBD_data02:0 allocation score on pgdbsrv01.cl1.local: -INFINITY
native_color: DRBD_data02:0 allocation score on pgdbsrv02.cl1.local: 10000
native_color: DRBD_data02:1 allocation score on pgdbsrv01.cl1.local: -INFINITY
native_color: DRBD_data02:1 allocation score on pgdbsrv02.cl1.local: -INFINITY
DRBD_data02:0 promotion score on pgdbsrv02.cl1.local: 10000
DRBD_data02:1 promotion score on none: 0
group_color: GRP_data01 allocation score on pgdbsrv01.cl1.local: 0
group_color: GRP_data01 allocation score on pgdbsrv02.cl1.local: 0
group_color: LVM_vgdata01 allocation score on pgdbsrv01.cl1.local: 0
group_color: LVM_vgdata01 allocation score on pgdbsrv02.cl1.local: 0
group_color: FS_data01 allocation score on pgdbsrv01.cl1.local: 0
group_color: FS_data01 allocation score on pgdbsrv02.cl1.local: 0
native_color: LVM_vgdata01 allocation score on pgdbsrv01.cl1.local: -INFINITY
native_color: LVM_vgdata01 allocation score on pgdbsrv02.cl1.local: 10000
native_color: FS_data01 allocation score on pgdbsrv01.cl1.local: -INFINITY
native_color: FS_data01 allocation score on pgdbsrv02.cl1.local: 0
group_color: GRP_data02 allocation score on pgdbsrv01.cl1.local: 0
group_color: GRP_data02 allocation score on pgdbsrv02.cl1.local: 0
group_color: LVM_vgdata02 allocation score on pgdbsrv01.cl1.local: 0
group_color: LVM_vgdata02 allocation score on pgdbsrv02.cl1.local: 0
group_color: FS_data02 allocation score on pgdbsrv01.cl1.local: 0
group_color: FS_data02 allocation score on pgdbsrv02.cl1.local: 0
native_color: LVM_vgdata02 allocation score on pgdbsrv01.cl1.local: -INFINITY
native_color: LVM_vgdata02 allocation score on pgdbsrv02.cl1.local: 10000
native_color: FS_data02 allocation score on pgdbsrv01.cl1.local: -INFINITY
native_color: FS_data02 allocation score on pgdbsrv02.cl1.local: 0
native_color: fusion-fencing allocation score on pgdbsrv01.cl1.local: 0
native_color: fusion-fencing allocation score on pgdbsrv02.cl1.local: 0

Transition Summary:
 * Promote DRBD_data01:0	(Slave -> Master pgdbsrv02.cl1.local)
 * Demote  DRBD_data01:1	(Master -> Stopped pgdbsrv01.cl1.local)
 * Promote DRBD_data02:0	(Slave -> Master pgdbsrv02.cl1.local)
 * Demote  DRBD_data02:1	(Master -> Stopped pgdbsrv01.cl1.local)
 * Move    LVM_vgdata01	(Started pgdbsrv01.cl1.local -> pgdbsrv02.cl1.local)
 * Move    FS_data01	(Started pgdbsrv01.cl1.local -> pgdbsrv02.cl1.local)
 * Move    LVM_vgdata02	(Started pgdbsrv01.cl1.local -> pgdbsrv02.cl1.local)
 * Move    FS_data02	(Started pgdbsrv01.cl1.local -> pgdbsrv02.cl1.local)


..taking node #1 offline (standby) for real, resources running on node #2, then:

# crm_simulate -L -s -u pgdbsrv01.cl1.local

Current cluster status:
Node pgdbsrv01.cl1.local: standby
Online: [ pgdbsrv02.cl1.local ]

 Master/Slave Set: DRBD_ms_data01 [DRBD_data01]
     Masters: [ pgdbsrv02.cl1.local ]
     Stopped: [ DRBD_data01:1 ]
 Master/Slave Set: DRBD_ms_data02 [DRBD_data02]
     Masters: [ pgdbsrv02.cl1.local ]
     Stopped: [ DRBD_data02:1 ]
 Resource Group: GRP_data01
     LVM_vgdata01	(ocf::heartbeat:LVM):	Started pgdbsrv02.cl1.local
     FS_data01	(ocf::heartbeat:Filesystem):	Started pgdbsrv02.cl1.local
 Resource Group: GRP_data02
     LVM_vgdata02	(ocf::heartbeat:LVM):	Started pgdbsrv02.cl1.local
     FS_data02	(ocf::heartbeat:Filesystem):	Started pgdbsrv02.cl1.local
 fusion-fencing	(stonith:fence_fusion):	Started pgdbsrv02.cl1.local

Performing requested modifications
 + Bringing node pgdbsrv01.cl1.local online
Allocation scores:
clone_color: DRBD_ms_data01 allocation score on pgdbsrv01.cl1.local: 0
clone_color: DRBD_ms_data01 allocation score on pgdbsrv02.cl1.local: 0
clone_color: DRBD_data01:0 allocation score on pgdbsrv01.cl1.local: 0
clone_color: DRBD_data01:0 allocation score on pgdbsrv02.cl1.local: 10000
clone_color: DRBD_data01:1 allocation score on pgdbsrv01.cl1.local: 0
clone_color: DRBD_data01:1 allocation score on pgdbsrv02.cl1.local: 0
native_color: DRBD_data01:0 allocation score on pgdbsrv01.cl1.local: -INFINITY
native_color: DRBD_data01:0 allocation score on pgdbsrv02.cl1.local: 10000
native_color: DRBD_data01:1 allocation score on pgdbsrv01.cl1.local: -INFINITY
native_color: DRBD_data01:1 allocation score on pgdbsrv02.cl1.local: -INFINITY
DRBD_data01:0 promotion score on pgdbsrv02.cl1.local: 10000
DRBD_data01:1 promotion score on none: 0
clone_color: DRBD_ms_data02 allocation score on pgdbsrv01.cl1.local: 0
clone_color: DRBD_ms_data02 allocation score on pgdbsrv02.cl1.local: 0
clone_color: DRBD_data02:0 allocation score on pgdbsrv01.cl1.local: 0
clone_color: DRBD_data02:0 allocation score on pgdbsrv02.cl1.local: 10000
clone_color: DRBD_data02:1 allocation score on pgdbsrv01.cl1.local: 0
clone_color: DRBD_data02:1 allocation score on pgdbsrv02.cl1.local: 0
native_color: DRBD_data02:0 allocation score on pgdbsrv01.cl1.local: -INFINITY
native_color: DRBD_data02:0 allocation score on pgdbsrv02.cl1.local: 10000
native_color: DRBD_data02:1 allocation score on pgdbsrv01.cl1.local: -INFINITY
native_color: DRBD_data02:1 allocation score on pgdbsrv02.cl1.local: -INFINITY
DRBD_data02:0 promotion score on pgdbsrv02.cl1.local: 10000
DRBD_data02:1 promotion score on none: 0
group_color: GRP_data01 allocation score on pgdbsrv01.cl1.local: 0
group_color: GRP_data01 allocation score on pgdbsrv02.cl1.local: 0
group_color: LVM_vgdata01 allocation score on pgdbsrv01.cl1.local: 0
group_color: LVM_vgdata01 allocation score on pgdbsrv02.cl1.local: 0
group_color: FS_data01 allocation score on pgdbsrv01.cl1.local: 0
group_color: FS_data01 allocation score on pgdbsrv02.cl1.local: 0
native_color: LVM_vgdata01 allocation score on pgdbsrv01.cl1.local: -INFINITY
native_color: LVM_vgdata01 allocation score on pgdbsrv02.cl1.local: 10000
native_color: FS_data01 allocation score on pgdbsrv01.cl1.local: -INFINITY
native_color: FS_data01 allocation score on pgdbsrv02.cl1.local: 0
group_color: GRP_data02 allocation score on pgdbsrv01.cl1.local: 0
group_color: GRP_data02 allocation score on pgdbsrv02.cl1.local: 0
group_color: LVM_vgdata02 allocation score on pgdbsrv01.cl1.local: 0
group_color: LVM_vgdata02 allocation score on pgdbsrv02.cl1.local: 0
group_color: FS_data02 allocation score on pgdbsrv01.cl1.local: 0
group_color: FS_data02 allocation score on pgdbsrv02.cl1.local: 0
native_color: LVM_vgdata02 allocation score on pgdbsrv01.cl1.local: -INFINITY
native_color: LVM_vgdata02 allocation score on pgdbsrv02.cl1.local: 10000
native_color: FS_data02 allocation score on pgdbsrv01.cl1.local: -INFINITY
native_color: FS_data02 allocation score on pgdbsrv02.cl1.local: 0
native_color: fusion-fencing allocation score on pgdbsrv01.cl1.local: 0
native_color: fusion-fencing allocation score on pgdbsrv02.cl1.local: 0

Transition Summary:


And that's it. Once I unstandby node #1 and rerun the same simulate:

# crm_simulate -L -s -u pgdbsrv01.cl1.local

Current cluster status:
Online: [ pgdbsrv01.cl1.local pgdbsrv02.cl1.local ]

 Master/Slave Set: DRBD_ms_data01 [DRBD_data01]
     Masters: [ pgdbsrv02.cl1.local ]
     Slaves: [ pgdbsrv01.cl1.local ]
 Master/Slave Set: DRBD_ms_data02 [DRBD_data02]
     Masters: [ pgdbsrv02.cl1.local ]
     Slaves: [ pgdbsrv01.cl1.local ]
 Resource Group: GRP_data01
     LVM_vgdata01	(ocf::heartbeat:LVM):	Stopped
     FS_data01	(ocf::heartbeat:Filesystem):	Stopped
 Resource Group: GRP_data02
     LVM_vgdata02	(ocf::heartbeat:LVM):	Stopped
     FS_data02	(ocf::heartbeat:Filesystem):	Stopped
 fusion-fencing	(stonith:fence_fusion):	Started pgdbsrv01.cl1.local

Performing requested modifications
 + Bringing node pgdbsrv01.cl1.local online
Allocation scores:
clone_color: DRBD_ms_data01 allocation score on pgdbsrv01.cl1.local: 0
clone_color: DRBD_ms_data01 allocation score on pgdbsrv02.cl1.local: 0
clone_color: DRBD_data01:0 allocation score on pgdbsrv01.cl1.local: 10000
clone_color: DRBD_data01:0 allocation score on pgdbsrv02.cl1.local: 10000
clone_color: DRBD_data01:1 allocation score on pgdbsrv01.cl1.local: 10000
clone_color: DRBD_data01:1 allocation score on pgdbsrv02.cl1.local: 10000
native_color: DRBD_data01:1 allocation score on pgdbsrv01.cl1.local: 10000
native_color: DRBD_data01:1 allocation score on pgdbsrv02.cl1.local: 10000
native_color: DRBD_data01:0 allocation score on pgdbsrv01.cl1.local: 10000
native_color: DRBD_data01:0 allocation score on pgdbsrv02.cl1.local: -INFINITY
DRBD_data01:1 promotion score on pgdbsrv02.cl1.local: 10000
DRBD_data01:0 promotion score on pgdbsrv01.cl1.local: 10000
clone_color: DRBD_ms_data02 allocation score on pgdbsrv01.cl1.local: 0
clone_color: DRBD_ms_data02 allocation score on pgdbsrv02.cl1.local: 0
clone_color: DRBD_data02:0 allocation score on pgdbsrv01.cl1.local: 10000
clone_color: DRBD_data02:0 allocation score on pgdbsrv02.cl1.local: 10000
clone_color: DRBD_data02:1 allocation score on pgdbsrv01.cl1.local: 10000
clone_color: DRBD_data02:1 allocation score on pgdbsrv02.cl1.local: 10000
native_color: DRBD_data02:1 allocation score on pgdbsrv01.cl1.local: 10000
native_color: DRBD_data02:1 allocation score on pgdbsrv02.cl1.local: 10000
native_color: DRBD_data02:0 allocation score on pgdbsrv01.cl1.local: 10000
native_color: DRBD_data02:0 allocation score on pgdbsrv02.cl1.local: -INFINITY
DRBD_data02:1 promotion score on pgdbsrv02.cl1.local: 10000
DRBD_data02:0 promotion score on pgdbsrv01.cl1.local: 10000
group_color: GRP_data01 allocation score on pgdbsrv01.cl1.local: 0
group_color: GRP_data01 allocation score on pgdbsrv02.cl1.local: 0
group_color: LVM_vgdata01 allocation score on pgdbsrv01.cl1.local: 0
group_color: LVM_vgdata01 allocation score on pgdbsrv02.cl1.local: 0
group_color: FS_data01 allocation score on pgdbsrv01.cl1.local: 0
group_color: FS_data01 allocation score on pgdbsrv02.cl1.local: 0
native_color: LVM_vgdata01 allocation score on pgdbsrv01.cl1.local: -INFINITY
native_color: LVM_vgdata01 allocation score on pgdbsrv02.cl1.local: 10000
native_color: FS_data01 allocation score on pgdbsrv01.cl1.local: -INFINITY
native_color: FS_data01 allocation score on pgdbsrv02.cl1.local: 0
group_color: GRP_data02 allocation score on pgdbsrv01.cl1.local: 0
group_color: GRP_data02 allocation score on pgdbsrv02.cl1.local: 0
group_color: LVM_vgdata02 allocation score on pgdbsrv01.cl1.local: 0
group_color: LVM_vgdata02 allocation score on pgdbsrv02.cl1.local: 0
group_color: FS_data02 allocation score on pgdbsrv01.cl1.local: 0
group_color: FS_data02 allocation score on pgdbsrv02.cl1.local: 0
native_color: LVM_vgdata02 allocation score on pgdbsrv01.cl1.local: -INFINITY
native_color: LVM_vgdata02 allocation score on pgdbsrv02.cl1.local: 10000
native_color: FS_data02 allocation score on pgdbsrv01.cl1.local: -INFINITY
native_color: FS_data02 allocation score on pgdbsrv02.cl1.local: 0
native_color: fusion-fencing allocation score on pgdbsrv01.cl1.local: 0
native_color: fusion-fencing allocation score on pgdbsrv02.cl1.local: 0

Transition Summary:
 * Start   LVM_vgdata01	(pgdbsrv02.cl1.local - blocked)
 * Start   FS_data01	(pgdbsrv02.cl1.local - blocked)
 * Start   LVM_vgdata02	(pgdbsrv02.cl1.local - blocked)
 * Start   FS_data02	(pgdbsrv02.cl1.local - blocked)




-- 
Heikki M



On 9.9.2013, at 12.02, Andreas Mock <andreas.mock at web.de> wrote:

> Hi Heikki,
> 
> it has to be crm_simulate -L -s. Sorry for the wrong command line
> parameters.
> 
> Best regards
> Andreas
> 
> 
> -----Ursprüngliche Nachricht-----
> Von: Heikki Manninen [mailto:hma at iki.fi] 
> Gesendet: Montag, 9. September 2013 10:46
> An: The Pacemaker cluster resource manager
> Betreff: Re: [Pacemaker] Resource ordering/colocating question (DRBD + LVM +
> FS)
> 
> Hello Andreas, thanks for your input, much appreciated.
> 
> On 5.9.2013, at 16.39, "Andreas Mock" <andreas.mock at web.de> wrote:
> 
>> 1) The second output of crm_mon show a resource IP_database
>> which is not shown in the initial crm_mon output and also
>> not in the config. => Reduce your problem/config to the
>> minimum being reproducible.
> 
> True. I edited out the resource from the e-mail that did not have anything
> to do with the problem as such (works ok all the time). Just forgot to
> remove it from the second copy-paste also. And yes, no more IP resource in
> the configuration.
> 
>> 2) Enable logging and look out which node is the DC.
>> There in the logs you find many many informations showing
>> what is going on. Hint: Open a terminal session with an
>> opened tail -f logfile. Watch it while inserting commands.
>> You'll get used to it.
> 
> Seems that node #2 was the DC (also visible in the pcs status output). I
> have looked at the logs all the time, just not yet too familiar with the
> contents of pacemaker logging. Here's the thing that keeps repeating
> everytime those LVM and FS resources stay in stopped state:
> 
> Sep  3 20:01:23 pgdbsrv02 pengine[1667]:   notice: LogActions: Start
> LVM_vgdata01#011(pgdbsrv01.cl1.local - blocked)
> Sep  3 20:01:23 pgdbsrv02 pengine[1667]:   notice: LogActions: Start
> FS_data01#011(pgdbsrv01.cl1.local - blocked)
> Sep  3 20:01:23 pgdbsrv02 pengine[1667]:   notice: LogActions: Start
> LVM_vgdata02#011(pgdbsrv01.cl1.local - blocked)
> Sep  3 20:01:23 pgdbsrv02 pengine[1667]:   notice: LogActions: Start
> FS_data02#011(pgdbsrv01.cl1.local - blocked)
> 
> So what does blocked mean here? Is it that the node #1 in this case is in
> need of fencing/stonithing and thus being blocked or something else (I have
> a backgroud in the RHCS/HACMP/LifeKeeper etc. world). No quorum policy is
> set to ignore.
> 
>> 3) The shown status of a drbd resource (crm_mon) doesn't show
>> you all informations of the drbd devices. Have a look at
>> drbd-overview on both nodes. (e.g. syncing status).
> 
> True, DRBD is working fine on these occations. Connected, Synced etc.
> 
>> 4) This setup CRIES for stonithing. Even in a test environment.
>> When stonith happens (this is what you see immediately) you
>> know something went wrong. This is a good indicator for
>> errors in agents or in the config. Believe me, as tedious stonithing
>> is the valuable it is for getting hints for bad cluster state.
>> On virtual machines stonithing is not as painful as on real
>> servers.
> 
> Very much true. I have implemented some custom fencing/stonithing agents
> before on physical and virtual cluster environments. Problem being here is
> that I'm not aware of reasonably simple ways to implement stonith with
> VMware Fusion that I'm bound to use for this test setup. Have to dig more
> into this though. So fencing from cman cluster.conf is chained to pacemaker
> fencing and pacemaker stonithing is disabled, no quorum policy is ignore.
> 
>> 5) Is the drbd fencing script enabled? If yes, in certain circumstances
>> -INF rules are inserted to deny promoting of "wrong" nodes.
>> You should grep for them 'cibadmin -Q | grep <resname>'
> 
> No, DRBD fencing is not enabled and split-brain recovery is done manually.
> 
>> 6) crm_simulate -L -v gives you an output of the scores of
>> the resources on each node. I really don't know how to read it
>> exactly (Is there a documentation of that anywhere?), but it
>> gives you a hint where to look at, when resources don't start.
>> Especially the aggregation of stickiness values in groups are
>> sometimes misleading.
> 
> Could be that I have some different version maybe, because -v is unknown
> option and:
> 
> # crm_simulate -L -V
> 
> Current cluster status:
> Online: [ pgdbsrv01.cl1.local pgdbsrv02.cl1.local ]
> 
> Master/Slave Set: DRBD_ms_data01 [DRBD_data01]
>    Masters: [ pgdbsrv01.cl1.local ]
>    Slaves: [ pgdbsrv02.cl1.local ]
> Master/Slave Set: DRBD_ms_data02 [DRBD_data02]
>    Masters: [ pgdbsrv01.cl1.local ]
>    Slaves: [ pgdbsrv02.cl1.local ]
> Resource Group: GRP_data01
>    LVM_vgdata01	(ocf::heartbeat:LVM):	Stopped
>    FS_data01	(ocf::heartbeat:Filesystem):	Stopped
> Resource Group: GRP_data02
>    LVM_vgdata02	(ocf::heartbeat:LVM):	Stopped
>    FS_data02	(ocf::heartbeat:Filesystem):	Stopped
> 
> 
> Only shows that much.
> 
> Original problem description left quoted below.
> 
> 
> -- 
> Heikki M
> 
> 
>> -----Ursprüngliche Nachricht-----
>> Von: Heikki Manninen [mailto:hma at iki.fi] 
>> Gesendet: Donnerstag, 5. September 2013 14:08
>> An: pacemaker at oss.clusterlabs.org
>> Betreff: [Pacemaker] Resource ordering/colocating question (DRBD + LVM +
> FS)
>> 
>> Hello,
>> 
>> I'm having a bit of a problem understanding what's going on with my simple
>> two-node demo cluster here. My resources come up correctly after
> restarting
>> the whole cluster but the LVM and Filesystem resources fail to start after
> a
>> single node restart or standby/unstandby (after node comes back online -
> why
>> do they even stop/start after the second node comes back?).
>> 
>> OS: CentOS 6.4 (cman stack)
>> Pacemaker: pacemaker-1.1.8-7.el6.x86_64
>> DRBD: drbd84-utils-8.4.3-1.el6.elrepo.x86_64
>> 
>> Everything is configured using: pcs-0.9.26-10.el6_4.1.noarch
>> 
>> Two DRBD resources configured and working: data01 & data02
>> Two nodes: pgdbsrv01.cl1.local & pgdbsrv02.cl1.local
>> 
>> Configuration:
>> 
>> node pgdbsrv01.cl1.local
>> node pgdbsrv02.cl1.local
>> primitive DRBD_data01 ocf:linbit:drbd \
>>   params drbd_resource="data01" \
>>   op monitor interval="30s"
>> primitive DRBD_data02 ocf:linbit:drbd \
>>   params drbd_resource="data02" \
>>   op monitor interval="30s"
>> primitive FS_data01 ocf:heartbeat:Filesystem \
>>   params device="/dev/mapper/vgdata01-lvdata01" directory="/data01"
>> fstype="ext4" \
>>   op monitor interval="30s"
>> primitive FS_data02 ocf:heartbeat:Filesystem \
>>   params device="/dev/mapper/vgdata02-lvdata02" directory="/data02"
>> fstype="ext4" \
>>   op monitor interval="30s"
>> primitive LVM_vgdata01 ocf:heartbeat:LVM \
>>   params volgrpname="vgdata01" exclusive="true" \
>>   op monitor interval="30s"
>> primitive LVM_vgdata02 ocf:heartbeat:LVM \
>>   params volgrpname="vgdata02" exclusive="true" \
>>   op monitor interval="30s"
>> group GRP_data01 LVM_vgdata01 FS_data01
>> group GRP_data02 LVM_vgdata02 FS_data02
>> ms DRBD_ms_data01 DRBD_data01 \
>>   meta master-max="1" master-node-max="1" clone-max="2"
>> clone-node-max="1" notify="true"
>> ms DRBD_ms_data02 DRBD_data02 \
>>   meta master-max="1" master-node-max="1" clone-max="2"
>> clone-node-max="1" notify="true"
>> colocation colocation-GRP_data01-DRBD_ms_data01-INFINITY inf: GRP_data01
>> DRBD_ms_data01:Master
>> colocation colocation-GRP_data02-DRBD_ms_data02-INFINITY inf: GRP_data02
>> DRBD_ms_data02:Master
>> order order-DRBD_data01-GRP_data01-mandatory : DRBD_data01:promote
>> GRP_data01:start
>> order order-DRBD_data02-GRP_data02-mandatory : DRBD_data02:promote
>> GRP_data02:start
>> property $id="cib-bootstrap-options" \
>>   dc-version="1.1.8-7.el6-394e906" \
>>   cluster-infrastructure="cman" \
>>   stonith-enabled="false" \
>>   no-quorum-policy="ignore" \
>>   migration-threshold="1"
>> rsc_defaults $id="rsc_defaults-options" \
>>   resource-stickiness="100"
>> 
>> 
>> 1) After starting the cluster, everything runs happily:
>> 
>> Last updated: Tue Sep  3 00:11:13 2013
>> Last change: Tue Sep  3 00:05:15 2013 via cibadmin on pgdbsrv01.cl1.local
>> Stack: cman
>> Current DC: pgdbsrv02.cl1.local - partition with quorum
>> Version: 1.1.8-7.el6-394e906
>> 2 Nodes configured, unknown expected votes
>> 9 Resources configured.
>> 
>> Online: [ pgdbsrv01.cl1.local pgdbsrv02.cl1.local ]
>> 
>> Full list of resources:
>> 
>> Master/Slave Set: DRBD_ms_data01 [DRBD_data01]
>>   Masters: [ pgdbsrv01.cl1.local ]
>>   Slaves: [ pgdbsrv02.cl1.local ]
>> Master/Slave Set: DRBD_ms_data02 [DRBD_data02]
>>   Masters: [ pgdbsrv01.cl1.local ]
>>   Slaves: [ pgdbsrv02.cl1.local ]
>> Resource Group: GRP_data01
>>   LVM_vgdata01 (ocf::heartbeat:LVM): Started pgdbsrv01.cl1.local
>>   FS_data01 (ocf::heartbeat:Filesystem): Started pgdbsrv01.cl1.local
>> Resource Group: GRP_data02
>>   LVM_vgdata02 (ocf::heartbeat:LVM): Started pgdbsrv01.cl1.local
>>   FS_data02 (ocf::heartbeat:Filesystem): Started pgdbsrv01.cl1.local
>> 
>> 2) Putting node #1 to standby mode - after which everything runs happily
> on
>> node pgdbsrv02.cl1.local
>> 
>> # pcs cluster standby pgdbsrv01.cl1.local
>> # pcs status
>> Last updated: Tue Sep  3 00:16:01 2013
>> Last change: Tue Sep  3 00:15:55 2013 via crm_attribute on
>> pgdbsrv02.cl1.local
>> Stack: cman
>> Current DC: pgdbsrv02.cl1.local - partition with quorum
>> Version: 1.1.8-7.el6-394e906
>> 2 Nodes configured, unknown expected votes
>> 9 Resources configured.
>> 
>> 
>> Node pgdbsrv01.cl1.local: standby
>> Online: [ pgdbsrv02.cl1.local ]
>> 
>> Full list of resources:
>> 
>> IP_database     (ocf::heartbeat:IPaddr2):     Started pgdbsrv02.cl1.local
>> Master/Slave Set: DRBD_ms_data01 [DRBD_data01]
>>   Masters: [ pgdbsrv02.cl1.local ]
>>   Stopped: [ DRBD_data01:1 ]
>> Master/Slave Set: DRBD_ms_data02 [DRBD_data02]
>>   Masters: [ pgdbsrv02.cl1.local ]
>>   Stopped: [ DRBD_data02:1 ]
>> Resource Group: GRP_data01
>>   LVM_vgdata01     (ocf::heartbeat:LVM):     Started pgdbsrv02.cl1.local
>>   FS_data01     (ocf::heartbeat:Filesystem):     Started
>> pgdbsrv02.cl1.local
>> Resource Group: GRP_data02
>>   LVM_vgdata02     (ocf::heartbeat:LVM):     Started pgdbsrv02.cl1.local
>>   FS_data02     (ocf::heartbeat:Filesystem):     Started
>> pgdbsrv02.cl1.local
>> 
>> 3) Putting node #1 back online - it seems that all the resources stop (?)
>> and then DRBD gets promoted successfully on node #2 but LVM and FS
> resources
>> never start
>> 
>> # pcs cluster unstandby pgdbsrv01.cl1.local
>> # pcs status
>> Last updated: Tue Sep  3 00:17:00 2013
>> Last change: Tue Sep  3 00:16:56 2013 via crm_attribute on
>> pgdbsrv02.cl1.local
>> Stack: cman
>> Current DC: pgdbsrv02.cl1.local - partition with quorum
>> Version: 1.1.8-7.el6-394e906
>> 2 Nodes configured, unknown expected votes
>> 9 Resources configured.
>> 
>> 
>> Online: [ pgdbsrv01.cl1.local pgdbsrv02.cl1.local ]
>> 
>> Full list of resources:
>> 
>> IP_database     (ocf::heartbeat:IPaddr2):     Started pgdbsrv02.cl1.local
>> Master/Slave Set: DRBD_ms_data01 [DRBD_data01]
>>   Masters: [ pgdbsrv02.cl1.local ]
>>   Slaves: [ pgdbsrv01.cl1.local ]
>> Master/Slave Set: DRBD_ms_data02 [DRBD_data02]
>>   Masters: [ pgdbsrv02.cl1.local ]
>>   Slaves: [ pgdbsrv01.cl1.local ]
>> Resource Group: GRP_data01
>>   LVM_vgdata01     (ocf::heartbeat:LVM):     Stopped
>>   FS_data01     (ocf::heartbeat:Filesystem):     Stopped
>> Resource Group: GRP_data02
>>   LVM_vgdata02     (ocf::heartbeat:LVM):     Stopped
>>   FS_data02     (ocf::heartbeat:Filesystem):     Stopped
>> 
>> 
>> 
>> Any ideas why this is happening/what could be wrong in the resource
>> configuration? The same thing happens when testing the situation with the
>> resources located vice-versa in the beginning. Also, if I stop & start one
>> of the nodes, same thing happens once the node gets back online.
>> 
>> 
>> -- 
>> Heikki Manninen <hma at iki.fi>
>> _______________________________________________
>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>> 
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>> 
>> 
>> _______________________________________________
>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>> 
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
> 
> 
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
> 
> 
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org





More information about the Pacemaker mailing list