[Pacemaker] [Linux-HA] Probably a regression of the linbit drbd agent between pacemaker 1.1.8 and 1.1.10

Andrew Beekhof andrew at beekhof.net
Mon Sep 9 04:42:45 UTC 2013


On 06/09/2013, at 5:51 PM, Lars Ellenberg <Lars.Ellenberg at linbit.com> wrote:

> On Tue, Aug 27, 2013 at 06:51:45AM +0200, Andreas Mock wrote:
>> Hi Andrew,
>> 
>> as this is a real showstopper at the moment I invested some other
>> hours to be sure (as far as possible) not having made an error.
>> 
>> Some additions:
>> 1) I mirrored the whole mini drbd config to another pacemaker cluster.
>> Same result: pacemaker 1.1.8 works, pacemaker 1.1.10 not 
>> 2) When I remove the target role Stopped from the drbd ms resource
>> and insert the config snippet related to the drbd device via crm -f <file>
>> to a lean running pacemaker config (pacemaker cluster options, stonith
>> resources),
>> it seems to work. That means one of the nodes gets promoted.
>> 
>> Then after stopping 'crm resource stop ms_drbd_xxx' and starting again
>> I see the same promotion error as described.
>> 
>> The drbd resource agent is using /usr/sbin/crm_master.
>> Is there a possibility that feedback given through this client tool
>> is changing the timing behaviour of pacemaker? Or the way
>> transitions are scheduled?
>> Any idea that may be related to a change in pacemaker?
> 
> I think that recent pacemaker allows for "start" and "promote" in the
> same transition.

At least in the one case I saw logs of, this wasn't the case.
The PE computed:

Current cluster status:
Online: [ db05 db06 ]

r_stonith-db05	(stonith:fence_imm):	Started db06 
r_stonith-db06	(stonith:fence_imm):	Started db05 
Master/Slave Set: ms_drbd_fodb [r_drbd_fodb]
    Slaves: [ db05 db06 ]
Master/Slave Set: ms_drbd_fodblog [r_drbd_fodblog]
    Slaves: [ db05 db06 ]

Transition Summary:
* Promote r_drbd_fodb:0	(Slave -> Master db05)
* Promote r_drbd_fodblog:0	(Slave -> Master db05)

and it was the promotion of r_drbd_fodb:0 that failed.

> Or at least has promote follow start much faster than before.
> 
> To be able to really tell why DRBD has problems to promote,
> I'd need the drbd (kernel!) logs, the agent logs are not good enough.
> 
> Maybe you hit some interesting race between DRBD establishing the
> connection, and pacemaker trying to promote one of the nodes.
> Correlating DRBD kernel logs with pacemaker logs should tell.
> 
> Do you have DRBD resource level fencing enabled?
> 
> I guess right now you can reproduce it easily by:
>  crm resource stop ms_drbd
>  crm resource start ms_drbd
> 
> I suspect you would not be able to reproduce by:
>  crm resource stop ms_drbd
>  crm resource demote ms_drbd (will only make drbd Secondary stuff)
>    ... meanwhile, DRBD will establish the connection ...
>  crm resource promote ms_drbd (will then promote one node)
> 
> Hth,
> 
>        Lars
> 
> 
>> -----Ursprüngliche Nachricht-----
>> Von: Andrew Beekhof [mailto:andrew at beekhof.net] 
>> Gesendet: Dienstag, 27. August 2013 05:02
>> An: General Linux-HA mailing list
>> Cc: pacemaker at oss.clusterlabs.org
>> Betreff: Re: [Pacemaker] [Linux-HA] Probably a regression of the linbit drbd
>> agent between pacemaker 1.1.8 and 1.1.10
>> 
>> 
>> On 27/08/2013, at 3:31 AM, Andreas Mock <Andreas.Mock at web.de> wrote:
>> 
>>> Hi all,
>>> 
>>> while the linbit drbd resource agent seems to work perfectly on 
>>> pacemaker 1.1.8 (standard software repository) we have problems with 
>>> the last release 1.1.10 and also with the newest head 1.1.11.xxx.
>>> 
>>> As using drbd is not so uncommon I really hope to find interested 
>>> people helping me out. I can provide as much debug information as you 
>>> want.
>>> 
>>> 
>>> Environment:
>>> RHEL 6.4 clone (Scientific Linux 6.4) cman based cluster.
>>> DRBD 8.4.3 compiled from sources.
>>> 64bit
>>> 
>>> - A drbd resource configured following the linbit documentation.
>>> - Manual start and stop (up/down) and setting primary of drbd resource 
>>> working smoothly.
>>> - 2 nodes dis03-test/dis04-test
>>> 
>>> 
>>> 
>>> - Following simple config on pacemaker 1.1.8 configure
>>>   property no-quorum-policy=stop
>>>   property stonith-enabled=true
>>>   rsc_defaults resource-stickiness=2
>>>   primitive r_stonith-dis03-test stonith:fence_mock \
>>>       meta resource-stickiness="INFINITY" target-role="Started" \
>>>       op monitor interval="180" timeout="300" requires="nothing" \
>>>       op start interval="0" timeout="300" \
>>>       op stop interval="0" timeout="300" \
>>>       params vmname=dis03-test pcmk_host_list="dis03-test"
>>>   primitive r_stonith-dis04-test stonith:fence_mock \
>>>       meta resource-stickiness="INFINITY" target-role="Started" \
>>>       op monitor interval="180" timeout="300" requires="nothing" \
>>>       op start interval="0" timeout="300" \
>>>       op stop interval="0" timeout="300" \
>>>       params vmname=dis04-test pcmk_host_list="dis04-test"
>>>   location r_stonith-dis03_hates_dis03 r_stonith-dis03-test \
>>>       rule $id="r_stonith-dis03_hates_dis03-test_rule" -inf: #uname 
>>> eq dis03-test
>>>   location r_stonith-dis04_hates_dis04 r_stonith-dis04-test \
>>>       rule $id="r_stonith-dis04_hates_dis04-test_rule" -inf: #uname 
>>> eq dis04-test
>>>   primitive r_drbd_postfix ocf:linbit:drbd \
>>>       params drbd_resource="postfix" drbdconf="/usr/local/etc/drbd.conf"
>> \
>>>       op monitor interval="15s"  timeout="60s" role="Master" \
>>>       op monitor interval="45s"  timeout="60s" role="Slave" \
>>>       op start timeout="240" \
>>>       op stop timeout="240" \
>>>       meta target-role="Stopped" migration-threshold="2"
>>>   ms ms_drbd_postfix r_drbd_postfix \
>>>       meta master-max="1" master-node-max="1" \
>>>       clone-max="2" clone-node-max="1" \
>>>       notify="true" \
>>>       meta target-role="Stopped"
>>> commit
>>> 
>>> - Pacemaker is started from scratch
>>> - Config above is applied by crm -f <file> where <file> has the above 
>>> config snippet.
>>> 
>>> - After that crm_mon shows the following status
>>> ----------------------8<-------------------------
>>> Last updated: Mon Aug 26 18:42:47 2013 Last change: Mon Aug 26 
>>> 18:42:42 2013 via cibadmin on dis03-test
>>> Stack: cman
>>> Current DC: dis03-test - partition with quorum
>>> Version: 1.1.10-1.el6-9abe687
>>> 2 Nodes configured
>>> 4 Resources configured
>>> 
>>> 
>>> Online: [ dis03-test dis04-test ]
>>> 
>>> Full list of resources:
>>> 
>>> r_stonith-dis03-test   (stonith:fence_mock):   Started dis04-test
>>> r_stonith-dis04-test   (stonith:fence_mock):   Started dis03-test
>>> Master/Slave Set: ms_drbd_postfix [r_drbd_postfix]
>>>    Stopped: [ dis03-test dis04-test ]
>>> 
>>> Migration summary:
>>> * Node dis04-test:
>>> * Node dis03-test:
>>> ----------------------8<-------------------------
>>> 
>>> cat /proc/drbd
>>> version: 8.4.3 (api:1/proto:86-101)
>>> GIT-hash: 89a294209144b68adb3ee85a73221f964d3ee515 build by 
>>> root at dis03-test,
>>> 2013-07-24 17:19:24
>>> 
>>> on both nodes. The drbd resource was shutdown previously in a clean 
>>> state, so that any node can be the primary.
>>> 
>>> - Now the weird behaviour when trying to start the drbd with
>>>  crm resource start ms_drbd_postfix
>>> 
>>> 
>>> Output of crm_mon -1rf
>>> ----------------------8<-------------------------
>>> Last updated: Mon Aug 26 18:46:33 2013 Last change: Mon Aug 26 
>>> 18:46:30 2013 via cibadmin on dis04-test
>>> Stack: cman
>>> Current DC: dis03-test - partition with quorum
>>> Version: 1.1.10-1.el6-9abe687
>>> 2 Nodes configured
>>> 4 Resources configured
>>> 
>>> 
>>> Online: [ dis03-test dis04-test ]
>>> 
>>> Full list of resources:
>>> 
>>> r_stonith-dis03-test   (stonith:fence_mock):   Started dis04-test
>>> r_stonith-dis04-test   (stonith:fence_mock):   Started dis03-test
>>> Master/Slave Set: ms_drbd_postfix [r_drbd_postfix]
>>>    Slaves: [ dis03-test ]
>>>    Stopped: [ dis04-test ]
>>> 
>>> Migration summary:
>>> * Node dis04-test:
>>>  r_drbd_postfix: migration-threshold=2 fail-count=2 last-failure='Mon 
>>> Aug
>>> 26 18:46:30 2013'
>>> * Node dis03-test:
>>> 
>> 
>> Its hard to imagine how pacemaker could cause drbdadm to fail, short of
>> leaving the other side promoted while trying to promote another.
>> Perhaps the drbd folks could comment on what the error means.
>> 
>>> Failed actions:
>>>   r_drbd_postfix_promote_0 (node=dis04-test, call=34, rc=1, 
>>> status=complete, last-rc-change=Mon Aug 26 18:46:29 2013 , 
>>> queued=1212ms, exec=0ms
>>> ): unknown error
>>> ----------------------8<-------------------------
>>> 
>>> In the log of the drbd agent I can find the following when the 
>>> promoting request is handled on dis03-test
>>> 
>>> ----------------------8<-------------------------
>>> ++ drbdadm -c /usr/local/etc/drbd.conf primary postfix
>>> 0: State change failed: (-2) Need access to UpToDate data Command 
>>> 'drbdsetup primary 0' terminated with exit code 17
>>> + cmd_out=
>>> + ret=17
>>> + '[' 17 '!=' 0 ']'
>>> + ocf_log err 'postfix: Called drbdadm -c /usr/local/etc/drbd.conf 
>>> + primary
>>> postfix'
>>> + '[' 2 -lt 2 ']'
>>> + __OCF_PRIO=err
>>> + shift
>>> ----------------------8<-------------------------
>>> 
>>> While working without problems on pacemaker 1.1.8 it doesn't work here.
>>> The error message let me assume that there is a kind of race condition 
>>> where pacemaker is firing the promotion too early.
>>> Probably it has something to do with applying attributes from the drbd 
>>> resource agent.
>>> But this is just a guess and I really don't know.
>>> 
>>> ONE ADDITIONAL information: As soon as I do a resource cleanup on the 
>>> "defective" node the master is promoted as expected. That means a:
>>>  crm resource cleanup r_drbd_postfix dis03-test results in the 
>>> following:
>>> 
>>> ----------------------8<-------------------------
>>> Last updated: Mon Aug 26 19:29:38 2013 Last change: Mon Aug 26 
>>> 19:29:28 2013 via cibadmin on dis04-test
>>> Stack: cman
>>> Current DC: dis03-test - partition with quorum
>>> Version: 1.1.10-1.el6-9abe687
>>> 2 Nodes configured
>>> 4 Resources configured
>>> 
>>> 
>>> Online: [ dis03-test dis04-test ]
>>> 
>>> Full list of resources:
>>> 
>>> r_stonith-dis03-test   (stonith:fence_mock):   Started dis04-test
>>> r_stonith-dis04-test   (stonith:fence_mock):   Started dis03-test
>>> Master/Slave Set: ms_drbd_postfix [r_drbd_postfix]
>>>    Masters: [ dis03-test ]
>>>    Slaves: [ dis04-test ]
>>> 
>>> Migration summary:
>>> * Node dis03-test:
>>> * Node dis04-test:
>>> ----------------------8<-------------------------
>>> 
>>> 
>>> 
>>> I really hope I can get some attention as pacemaker 1.1.10 is a 
>>> milestone for Andrew and drbd from linbit is pretty sure a building 
>>> block of many pacemaker based clusters.
>>> 
>>> Cluster log of DC dis03-test at http://pastebin.com/2S9Y6V3P DRBD 
>>> agent log at http://pastebin.com/ceYNEAhH
>>> 
>>> 
>>> So, any help welcome.
>>> 
>>> Best regards
>>> Andreas Mock
>>> 
>>> 
>>> _______________________________________________
>>> Linux-HA mailing list
>>> Linux-HA at lists.linux-ha.org
>>> http://lists.linux-ha.org/mailman/listinfo/linux-ha
>>> See also: http://linux-ha.org/ReportingProblems
>> 
>> 
>> 
>> _______________________________________________
>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>> 
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
> 
> -- 
> : Lars Ellenberg
> : LINBIT | Your Way to High Availability
> : DRBD/HA support and consulting http://www.linbit.com
> 
> DRBD® and LINBIT® are registered trademarks of LINBIT, Austria.
> _______________________________________________
> Linux-HA mailing list
> Linux-HA at lists.linux-ha.org
> http://lists.linux-ha.org/mailman/listinfo/linux-ha
> See also: http://linux-ha.org/ReportingProblems

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 841 bytes
Desc: Message signed with OpenPGP using GPGMail
URL: <https://lists.clusterlabs.org/pipermail/pacemaker/attachments/20130909/1e49d1cc/attachment-0004.sig>


More information about the Pacemaker mailing list