[ClusterLabs] DRBD demote/promote not called - Why? How to fix?

CART Andreas andreas.cart at sonorys.at
Wed Nov 9 19:27:39 CET 2016


Hi again



Sorry for missing the omission of the master role within the colocation constraint.

I  added it  - but unfortunately still no success.



(In the meantime I added 2 additional filesystem resources on top of the NFSServer, but that should not change anything regarding the root problem that I miss the demote of DRBDClone.)



I again started with all resources located at ventsi-clst1 and issued a 'pcs resource move DRBD_global_clst' (the resource next collocated next to the DRBDClone).



With that I end up with all primitive resources stopped and the DRBDClone resource still being master at ventsi-clst1.

Here is what pacemaker pretends has to be done:
==================================================================

[root at ventsi-clst2 ~]# crm_simulate -Ls



Current cluster status:

Online: [ ventsi-clst1-sync ventsi-clst2-sync ]



ipmi-fence-clst1       (stonith:fence_ipmilan):        Started ventsi-clst2-sync

ipmi-fence-clst2       (stonith:fence_ipmilan):        Started ventsi-clst1-sync

IPaddrNFS      (ocf::heartbeat:IPaddr2):       Stopped

NFSServer      (ocf::heartbeat:nfsserver):     Stopped

Master/Slave Set: DRBDClone [DRBD]

     Masters: [ ventsi-clst1-sync ]    <=== still not demoted

     Slaves: [ ventsi-clst2-sync ]

DRBD_global_clst       (ocf::heartbeat:Filesystem):    Stopped

NFS_global_clst        (ocf::heartbeat:Filesystem):    Stopped

BIND_global_clst       (ocf::heartbeat:Filesystem):    Stopped



Allocation scores:

native_color: ipmi-fence-clst1 allocation score on ventsi-clst1-sync: -INFINITY

native_color: ipmi-fence-clst1 allocation score on ventsi-clst2-sync: INFINITY

native_color: ipmi-fence-clst2 allocation score on ventsi-clst1-sync: INFINITY

native_color: ipmi-fence-clst2 allocation score on ventsi-clst2-sync: -INFINITY

clone_color: DRBDClone allocation score on ventsi-clst1-sync: 0

clone_color: DRBDClone allocation score on ventsi-clst2-sync: 0

clone_color: DRBD:0 allocation score on ventsi-clst1-sync: INFINITY

clone_color: DRBD:0 allocation score on ventsi-clst2-sync: 0

clone_color: DRBD:1 allocation score on ventsi-clst1-sync: 0

clone_color: DRBD:1 allocation score on ventsi-clst2-sync: INFINITY

native_color: DRBD:0 allocation score on ventsi-clst1-sync: INFINITY

native_color: DRBD:0 allocation score on ventsi-clst2-sync: 0

native_color: DRBD:1 allocation score on ventsi-clst1-sync: -INFINITY

native_color: DRBD:1 allocation score on ventsi-clst2-sync: INFINITY

DRBD:1 promotion score on ventsi-clst2-sync: 10000

DRBD:0 promotion score on ventsi-clst1-sync: 1

native_color: DRBD_global_clst allocation score on ventsi-clst1-sync: -INFINITY

native_color: DRBD_global_clst allocation score on ventsi-clst2-sync: INFINITY

native_color: IPaddrNFS allocation score on ventsi-clst1-sync: -INFINITY

native_color: IPaddrNFS allocation score on ventsi-clst2-sync: 0

native_color: NFSServer allocation score on ventsi-clst1-sync: -INFINITY

native_color: NFSServer allocation score on ventsi-clst2-sync: 0

native_color: NFS_global_clst allocation score on ventsi-clst1-sync: 0

native_color: NFS_global_clst allocation score on ventsi-clst2-sync: -INFINITY

native_color: BIND_global_clst allocation score on ventsi-clst1-sync: -INFINITY

native_color: BIND_global_clst allocation score on ventsi-clst2-sync: 0



Transition Summary:

* Start   IPaddrNFS    (ventsi-clst2-sync)

* Start   NFSServer    (ventsi-clst2-sync)

* Demote  DRBD:0       (Master -> Slave ventsi-clst1-sync)    <=== this demote never happens

* Promote DRBD:1       (Slave -> Master ventsi-clst2-sync)

* Start   DRBD_global_clst     (ventsi-clst2-sync)

* Start   NFS_global_clst      (ventsi-clst1-sync)

* Start   BIND_global_clst     (ventsi-clst2-sync)



And this is the executed transaction:
==================================================================

[root at ventsi-clst2 ~]# crm_simulate --xml-file /var/lib/pacemaker/pengine/pe-input-1157.bz2 --save-graph problem5.graph --save-dotfile problem5.dot -V --simulate

Using the original execution date of: 2016-11-09 17:54:10Z



Current cluster status:

Online: [ ventsi-clst1-sync ventsi-clst2-sync ]



ipmi-fence-clst1       (stonith:fence_ipmilan):        Started ventsi-clst2-sync

ipmi-fence-clst2       (stonith:fence_ipmilan):        Started ventsi-clst1-sync

IPaddrNFS      (ocf::heartbeat:IPaddr2):       Started ventsi-clst1-sync

NFSServer      (ocf::heartbeat:nfsserver):     Started ventsi-clst1-sync

Master/Slave Set: DRBDClone [DRBD]

     Masters: [ ventsi-clst1-sync ]

     Slaves: [ ventsi-clst2-sync ]

DRBD_global_clst       (ocf::heartbeat:Filesystem):    Started ventsi-clst1-sync

NFS_global_clst        (ocf::heartbeat:Filesystem):    Started ventsi-clst2-sync

BIND_global_clst       (ocf::heartbeat:Filesystem):    Started ventsi-clst1-sync



Transition Summary:

* Stop    IPaddrNFS    (ventsi-clst1-sync)

* Stop    NFSServer    (ventsi-clst1-sync)

* Stop    DRBD_global_clst     (ventsi-clst1-sync)

* Stop    NFS_global_clst      (Started ventsi-clst2-sync)

* Stop    BIND_global_clst     (ventsi-clst1-sync)



Executing cluster transition:

* Resource action: NFS_global_clst stop on ventsi-clst2-sync

* Resource action: BIND_global_clst stop on ventsi-clst1-sync

* Resource action: NFSServer       stop on ventsi-clst1-sync

* Resource action: IPaddrNFS       stop on ventsi-clst1-sync

* Resource action: DRBD_global_clst stop on ventsi-clst1-sync

* Pseudo action:   all_stopped    <=== no demote

Using the original execution date of: 2016-11-09 17:54:10Z



Revised cluster status:

Online: [ ventsi-clst1-sync ventsi-clst2-sync ]



ipmi-fence-clst1       (stonith:fence_ipmilan):        Started ventsi-clst2-sync

ipmi-fence-clst2       (stonith:fence_ipmilan):        Started ventsi-clst1-sync

IPaddrNFS      (ocf::heartbeat:IPaddr2):       Stopped

NFSServer      (ocf::heartbeat:nfsserver):     Stopped

Master/Slave Set: DRBDClone [DRBD]

     Masters: [ ventsi-clst1-sync ]

     Slaves: [ ventsi-clst2-sync ]

DRBD_global_clst       (ocf::heartbeat:Filesystem):    Stopped

NFS_global_clst        (ocf::heartbeat:Filesystem):    Stopped

BIND_global_clst       (ocf::heartbeat:Filesystem):    Stopped



And finally here the updated config:
==================================================================

[root at ventsi-clst1 ~]# pcs config

Cluster Name: clst1

Corosync Nodes:

ventsi-clst1-sync ventsi-clst2-sync

Pacemaker Nodes:

ventsi-clst1-sync ventsi-clst2-sync



Resources:

Resource: IPaddrNFS (class=ocf provider=heartbeat type=IPaddr2)

  Attributes: ip=xxx.xxx.xxx.xxx cidr_netmask=24

  Operations: start interval=0 timeout=20 (IPaddrNFS-start-interval-0)

              stop interval=0 timeout=20 (IPaddrNFS-stop-interval-0)

              monitor interval=10 timeout=20 (IPaddrNFS-monitor-interval-10)

Resource: NFSServer (class=ocf provider=heartbeat type=nfsserver)

  Attributes: nfs_shared_infodir=/drbdmnts/global_clst/nfsserversettings/ nfs_ip=xxx.xxx.xxx.xxx nfsd_args="-H xxx.xxx.xxx.xxx"

  Operations: start interval=0 timeout=40 (NFSServer-start-interval-0)

              stop interval=0 timeout=20 (NFSServer-stop-interval-0)

              monitor interval=10 timeout=20 (NFSServer-monitor-interval-10)

Master: DRBDClone

  Meta Attrs: master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true

  Resource: DRBD (class=ocf provider=linbit type=drbd)

   Attributes: drbd_resource=nfsdata

   Operations: start interval=0 timeout=240 (DRBD-start-interval-0)

               promote interval=0 timeout=90 (DRBD-promote-interval-0)

               demote interval=0 timeout=90 (DRBD-demote-interval-0)

               stop interval=0 timeout=100 (DRBD-stop-interval-0)

               monitor interval=9 role=Master timeout=5 (DRBD-monitor-interval-9)

               monitor interval=10 role=Slave timeout=5 (DRBD-monitor-interval-10)

Resource: DRBD_global_clst (class=ocf provider=heartbeat type=Filesystem)

  Attributes: device=/dev/drbd1 directory=/drbdmnts/global_clst fstype=ext4

  Operations: start interval=0 timeout=60 (DRBD_global_clst-start-interval-0)

              stop interval=0 timeout=60 (DRBD_global_clst-stop-interval-0)

              monitor interval=20 timeout=40 (DRBD_global_clst-monitor-interval-20)

Resource: NFS_global_clst (class=ocf provider=heartbeat type=Filesystem)

  Attributes: device=xxx.xxx.xxx.xxx:/drbdmnts/global_clst/nfs directory=/global/nfs fstype=nfs

  Operations: start interval=0 timeout=60 (NFS_global_clst-start-interval-0)

              stop interval=0 timeout=60 (NFS_global_clst-stop-interval-0)

              monitor interval=20 timeout=40 (NFS_global_clst-monitor-interval-20)

Resource: BIND_global_clst (class=ocf provider=heartbeat type=Filesystem)

  Attributes: device=/drbdmnts/global_clst/nfs directory=/global/nfs fstype=none options=bind

  Operations: start interval=0 timeout=60 (BIND_global_clst-start-interval-0)

              stop interval=0 timeout=60 (BIND_global_clst-stop-interval-0)

              monitor interval=20 timeout=40 (BIND_global_clst-monitor-interval-20)



Stonith Devices:

Resource: ipmi-fence-clst1 (class=stonith type=fence_ipmilan)

  Attributes: lanplus=1 login=foo passwd=bar action=reboot ipaddr=yyy.yyy.yyy.yyy pcmk_host_check=static-list pcmk_host_list=ventsi-clst1-sync auth=password timeout=30 cipher=1

  Operations: monitor interval=60 (ipmi-fence-clst1-monitor-interval-60)

Resource: ipmi-fence-clst2 (class=stonith type=fence_ipmilan)

  Attributes: lanplus=1 login=foo passwd=bar action=reboot ipaddr=zzz.zzz.zzz.zzz pcmk_host_check=static-list pcmk_host_list=ventsi-clst2-sync auth=password timeout=30 cipher=1

  Operations: monitor interval=60 (ipmi-fence-clst2-monitor-interval-60)

Fencing Levels:



Location Constraints:

  Resource: DRBD_global_clst

    Disabled on: ventsi-clst1-sync (score:-INFINITY) (role: Started) (id:cli-ban-DRBD_global_clst-on-ventsi-clst1-sync)

  Resource: ipmi-fence-clst1

    Disabled on: ventsi-clst1-sync (score:-INFINITY) (id:location-ipmi-fence-clst1-ventsi-clst1-sync--INFINITY)

  Resource: ipmi-fence-clst2

    Disabled on: ventsi-clst2-sync (score:-INFINITY) (id:location-ipmi-fence-clst2-ventsi-clst2-sync--INFINITY)

Ordering Constraints:

  start IPaddrNFS then start NFSServer (kind:Mandatory) (id:order-IPaddrNFS-NFSServer-mandatory)

  promote DRBDClone then start DRBD_global_clst (kind:Mandatory) (id:order-DRBDClone-DRBD_global_clst-mandatory)

  start DRBD_global_clst then start IPaddrNFS (kind:Mandatory) (id:order-DRBD_global_clst-IPaddrNFS-mandatory)

  start NFSServer then start NFS_global_clst (kind:Mandatory) (id:order-NFSServer-NFS_global_clst-mandatory)

  start NFSServer then start BIND_global_clst (kind:Mandatory) (id:order-NFSServer-BIND_global_clst-mandatory)

Colocation Constraints:

  NFSServer with IPaddrNFS (score:INFINITY) (id:colocation-NFSServer-IPaddrNFS-INFINITY)

  IPaddrNFS with DRBD_global_clst (score:INFINITY) (id:colocation-IPaddrNFS-DRBD_global_clst-INFINITY)

  NFS_global_clst with NFSServer (score:-INFINITY) (id:colocation-NFS_global_clst-NFSServer--INFINITY)

  BIND_global_clst with NFSServer (score:INFINITY) (id:colocation-BIND_global_clst-NFSServer-INFINITY)

  DRBD_global_clst with DRBDClone (score:INFINITY) (rsc-role:Started) (with-rsc-role:Master) (id:colocation-DRBD_global_clst-DRBDClone-INFINITY)



Resources Defaults:

resource-stickiness: INFINITY

Operations Defaults:

timeout: 10s



Cluster Properties:

cluster-infrastructure: cman

dc-version: 1.1.14-8.el6-70404b0

have-watchdog: false

last-lrm-refresh: 1478703150

no-quorum-policy: ignore

stonith-enabled: true

symmetric-cluster: true

Node Attributes:

ventsi-clst1-sync: PostgresSon-data-status=DISCONNECT

ventsi-clst2-sync: PostgresSon-data-status=DISCONNECT





Kind regards

Andi



-----Original Message-----
From: Ken Gaillot [mailto:kgaillot at redhat.com]
Sent: Dienstag, 8. November 2016 22:29
To: users at clusterlabs.org
Subject: Re: [ClusterLabs] DRBD demote/promote not called - Why? How to fix?



On 11/04/2016 01:57 PM, CART Andreas wrote:

> Hi

>

> I have a basic 2 node active/passive cluster with Pacemaker (1.1.14 ,

> pcs: 0.9.148) / CMAN (3.0.12.1) / Corosync (1.4.7) on RHEL 6.8.

> This cluster runs NFS on top of DRBD (8.4.4).

>

> Basically the system is working on both nodes and I can switch the

> resources from one node to the other.

> But switching resources to the other node does not work, if I try to

> move just one resource and have the others follow due to the location

> constraints.

>

> From the logged messages I see that in this "failure case" there is NO

> attempt to demote/promote the DRBD clone resource.

>

> Here is my setup:

> ==================================================================

> Cluster Name: clst1

> Corosync Nodes:

> ventsi-clst1-sync ventsi-clst2-sync

> Pacemaker Nodes:

> ventsi-clst1-sync ventsi-clst2-sync

>

> Resources:

> Resource: IPaddrNFS (class=ocf provider=heartbeat type=IPaddr2)

>   Attributes: ip=xxx.xxx.xxx.xxx cidr_netmask=24

>   Operations: start interval=0s timeout=20s (IPaddrNFS-start-interval-0s)

>               stop interval=0s timeout=20s (IPaddrNFS-stop-interval-0s)

>               monitor interval=5s (IPaddrNFS-monitor-interval-5s)

> Resource: NFSServer (class=ocf provider=heartbeat type=nfsserver)

>   Attributes: nfs_shared_infodir=/var/lib/nfsserversettings/

> nfs_ip=xxx.xxx.xxx.xxx nfsd_args="-H xxx.xxx.xxx.xxx"

>   Operations: start interval=0s timeout=40 (NFSServer-start-interval-0s)

>               stop interval=0s timeout=20s (NFSServer-stop-interval-0s)

>               monitor interval=10s timeout=20s

> (NFSServer-monitor-interval-10s)

> Master: DRBDClone

>   Meta Attrs: master-max=1 master-node-max=1 clone-max=2

> clone-node-max=1 notify=true

>   Resource: DRBD (class=ocf provider=linbit type=drbd)

>    Attributes: drbd_resource=nfsdata

>    Operations: start interval=0s timeout=240 (DRBD-start-interval-0s)

>                promote interval=0s timeout=90 (DRBD-promote-interval-0s)

>                demote interval=0s timeout=90 (DRBD-demote-interval-0s)

>                stop interval=0s timeout=100 (DRBD-stop-interval-0s)

>                monitor interval=1s timeout=5 (DRBD-monitor-interval-1s)

> Resource: DRBD_global_clst (class=ocf provider=heartbeat type=Filesystem)

>   Attributes: device=/dev/drbd1 directory=/drbdmnts/global_clst fstype=ext4

>   Operations: start interval=0s timeout=60

> (DRBD_global_clst-start-interval-0s)

>               stop interval=0s timeout=60

> (DRBD_global_clst-stop-interval-0s)

>               monitor interval=20 timeout=40

> (DRBD_global_clst-monitor-interval-20)

>

> Stonith Devices:

> Resource: ipmi-fence-clst1 (class=stonith type=fence_ipmilan)

>   Attributes: lanplus=1 login=foo passwd=bar action=reboot

> ipaddr=yyy.yyy.yyy.yyy pcmk_host_check=static-list

> pcmk_host_list=ventsi-clst1-sync auth=password timeout=30 cipher=1

>   Operations: monitor interval=60s (ipmi-fence-clst1-monitor-interval-60s)

> Resource: ipmi-fence-clst2 (class=stonith type=fence_ipmilan)

>   Attributes: lanplus=1 login=foo passwd=bar action=reboot

> ipaddr=zzz.zzz.zzz.zzz pcmk_host_check=static-list

> pcmk_host_list=ventsi-clst2-sync auth=password timeout=30 cipher=1

>   Operations: monitor interval=60s (ipmi-fence-clst2-monitor-interval-60s)

> Fencing Levels:

>

> Location Constraints:

>   Resource: ipmi-fence-clst1

>     Disabled on: ventsi-clst1-sync (score:-INFINITY)

> (id:location-ipmi-fence-clst1-ventsi-clst1-sync--INFINITY)

>   Resource: ipmi-fence-clst2

>     Disabled on: ventsi-clst2-sync (score:-INFINITY)

> (id:location-ipmi-fence-clst2-ventsi-clst2-sync--INFINITY)

> Ordering Constraints:

>   start IPaddrNFS then start NFSServer (kind:Mandatory)

> (id:order-IPaddrNFS-NFSServer-mandatory)

>   promote DRBDClone then start DRBD_global_clst (kind:Mandatory)

> (id:order-DRBDClone-DRBD_global_clst-mandatory)

>   start DRBD_global_clst then start IPaddrNFS (kind:Mandatory)

> (id:order-DRBD_global_clst-IPaddrNFS-mandatory)

> Colocation Constraints:

>   NFSServer with IPaddrNFS (score:INFINITY)

> (id:colocation-NFSServer-IPaddrNFS-INFINITY)

>   DRBD_global_clst with DRBDClone (score:INFINITY)

> (id:colocation-DRBD_global_clst-DRBDClone-INFINITY)



It took me a while to notice it, it's easily overlooked, but the above

constraint is the problem. It says DRBD_global_clst must be located

where DRBDClone is running ... not necessarily where DRBDClone is

master. This constraint should be created like this:



pcs constraint colocation add DRBD_global_clst with master DBRDClone



>   IPaddrNFS with DRBD_global_clst (score:INFINITY)

> (id:colocation-IPaddrNFS-DRBD_global_clst-INFINITY)

>

> Resources Defaults:

> resource-stickiness: INFINITY

> Operations Defaults:

> timeout: 10s

>

> Cluster Properties:

> cluster-infrastructure: cman

> dc-version: 1.1.14-8.el6-70404b0

> have-watchdog: false

> last-lrm-refresh: 1478277432

> no-quorum-policy: ignore

> stonith-enabled: true

> symmetric-cluster: true

> ==================================================================

>

> Initial state is e.g. this (all resources at node1):

>

> Online: [ ventsi-clst1-sync ventsi-clst2-sync ]

>

> Full list of resources:

>

> ipmi-fence-clst1       (stonith:fence_ipmilan):        Started

> ventsi-clst2-sync

> ipmi-fence-clst2       (stonith:fence_ipmilan):        Started

> ventsi-clst1-sync

> IPaddrNFS      (ocf::heartbeat:IPaddr2):       Started ventsi-clst1-sync

> NFSServer      (ocf::heartbeat:nfsserver):     Started ventsi-clst1-sync

> Master/Slave Set: DRBDClone [DRBD]

>      Masters: [ ventsi-clst1-sync ]

>      Slaves: [ ventsi-clst2-sync ]

> DRBD_global_clst       (ocf::heartbeat:Filesystem):    Started

> ventsi-clst1-sync

> ==================================================================

>

> If I shutdown the cluster at node 1 ('pcs cluster stop') or if I move

> the DRBD clone resource ('pcs resource move DRBDClone') all resources

> switch successfully to node2.

> I.e. the demote/promote of the DRBD clone resource is working in these

> cases.

>

> But if I try to move any other resource (e.g. 'pcs resource move

> NFSServer') the resources NFSServer, IPaddrNFS and DRBD_global_clst are

> stopped at node 1, but then already follows starting of the

> DRBD_global_clst resource at node2, which fails due to the missing

> demote/promote.

> As far as I can see there is some follow-up attempt to repair things

> partially as the resources are started again at node1 exclusive the

> resource which I moved due to my move command.

>

> Final state is like this:

>

> Online: [ ventsi-clst1-sync ventsi-clst2-sync ]

>

> Full list of resources:

>

> ipmi-fence-clst1       (stonith:fence_ipmilan):        Started

> ventsi-clst2-sync

> ipmi-fence-clst2       (stonith:fence_ipmilan):        Started

> ventsi-clst1-sync

> IPaddrNFS      (ocf::heartbeat:IPaddr2):       Started ventsi-clst1-sync

> NFSServer      (ocf::heartbeat:nfsserver):     Stopped

> Master/Slave Set: DRBDClone [DRBD]

>      Masters: [ ventsi-clst1-sync ]

>      Slaves: [ ventsi-clst2-sync ]

> DRBD_global_clst       (ocf::heartbeat:Filesystem):    Started

> ventsi-clst1-sync

>

> Failed Actions:

> * DRBD_global_clst_start_0 on ventsi-clst2-sync 'unknown error' (1):

> call=778, status=complete, exitreason='none',

>     last-rc-change='Fri Nov  4 19:32:56 2016', queued=0ms, exec=43ms

> ==================================================================

>

> Here are the logged messages for this "failure case":

>

> 2016-11-04T19:32:55.163982+01:00 ventsi-clst1 crmd[6116]:   notice:

> State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC

> cause=C_FSA_INTERNAL origin=abort_transition_graph ]

> 2016-11-04T19:32:55.168100+01:00 ventsi-clst1 pengine[6115]:   notice:

> On loss of CCM Quorum: Ignore

> 2016-11-04T19:32:55.181252+01:00 ventsi-clst1 pengine[6115]:   notice:

> Move    IPaddrNFS#011(Started ventsi-clst1-sync -> ventsi-clst2-sync)

> 2016-11-04T19:32:55.181260+01:00 ventsi-clst1 pengine[6115]:   notice:

> Move    NFSServer#011(Started ventsi-clst1-sync -> ventsi-clst2-sync)

> 2016-11-04T19:32:55.181278+01:00 ventsi-clst1 pengine[6115]:   notice:

> Move    DRBD_global_clst#011(Started ventsi-clst1-sync ->

> ventsi-clst2-sync)  <=== here no demote/promote is listed

> 2016-11-04T19:32:55.182385+01:00 ventsi-clst1 pengine[6115]:   notice:

> Calculated Transition 202: /var/lib/pacemaker/pengine/pe-input-766.bz2

> 2016-11-04T19:32:55.182998+01:00 ventsi-clst1 crmd[6116]:   notice:

> Initiating action 15: stop NFSServer_stop_0 on ventsi-clst1-sync (local)

> 2016-11-04T19:32:55.196265+01:00 ventsi-clst1

> nfsserver(NFSServer)[15978]: INFO: Stopping NFS server ...

> 2016-11-04T19:32:55.249137+01:00 ventsi-clst1 kernel: nfsd: last server

> has exited, flushing export cache

> 2016-11-04T19:32:55.252241+01:00 ventsi-clst1 rpc.mountd[15282]: Caught

> signal 15, un-registering and exiting.

> 2016-11-04T19:32:55.632708+01:00 ventsi-clst1

> nfsserver(NFSServer)[15978]: INFO: Stopping sm-notify

> 2016-11-04T19:32:55.650552+01:00 ventsi-clst1

> nfsserver(NFSServer)[15978]: INFO: Stopping rpc.statd

> 2016-11-04T19:32:55.666777+01:00 ventsi-clst1 rpc.statd[15243]: Caught

> signal 15, un-registering and exiting

> 2016-11-04T19:32:56.692819+01:00 ventsi-clst1

> nfsserver(NFSServer)[15978]: INFO: NFS server stopped

> 2016-11-04T19:32:56.695523+01:00 ventsi-clst1 crmd[6116]:   notice:

> Operation NFSServer_stop_0: ok (node=ventsi-clst1-sync, call=1220, rc=0,

> cib-update=1695, confirmed=true)

> 2016-11-04T19:32:56.696243+01:00 ventsi-clst1 crmd[6116]:   notice:

> Initiating action 12: stop IPaddrNFS_stop_0 on ventsi-clst1-sync (local)

> 2016-11-04T19:32:56.727882+01:00 ventsi-clst1 IPaddr2(IPaddrNFS)[16108]:

> INFO: IP status = ok, IP_CIP=

> 2016-11-04T19:32:56.733383+01:00 ventsi-clst1 crmd[6116]:   notice:

> Operation IPaddrNFS_stop_0: ok (node=ventsi-clst1-sync, call=1222, rc=0,

> cib-update=1696, confirmed=true)

> 2016-11-04T19:32:56.733917+01:00 ventsi-clst1 crmd[6116]:   notice:

> Initiating action 48: stop DRBD_global_clst_stop_0 on ventsi-clst1-sync

> (local)

> 2016-11-04T19:32:56.757181+01:00 ventsi-clst1

> Filesystem(DRBD_global_clst)[16163]: INFO: Running stop for /dev/drbd1

> on /drbdmnts/global_clst

> 2016-11-04T19:32:56.764684+01:00 ventsi-clst1

> Filesystem(DRBD_global_clst)[16163]: INFO: Trying to unmount

> /drbdmnts/global_clst

> 2016-11-04T19:32:56.771260+01:00 ventsi-clst1

> Filesystem(DRBD_global_clst)[16163]: INFO: unmounted

> /drbdmnts/global_clst successfully

> 2016-11-04T19:32:56.776640+01:00 ventsi-clst1 crmd[6116]:   notice:

> Operation DRBD_global_clst_stop_0: ok (node=ventsi-clst1-sync,

> call=1224, rc=0, cib-update=1697, confirmed=true)

> 2016-11-04T19:32:56.777140+01:00 ventsi-clst1 crmd[6116]:   notice:

> Initiating action 49: start DRBD_global_clst_start_0 on

> ventsi-clst2-sync   <=== hereis the attempt to start the filesystem at

> the other node, although DRBD has not yet been promoted

> 2016-11-04T19:32:56.840137+01:00 ventsi-clst1 crmd[6116]:  warning:

> Action 49 (DRBD_global_clst_start_0) on ventsi-clst2-sync failed

> (target: 0 vs. rc: 1): Error

> 2016-11-04T19:32:56.840158+01:00 ventsi-clst1 crmd[6116]:   notice:

> Transition aborted by DRBD_global_clst_start_0 'modify' on

> ventsi-clst2-sync: Event failed

> (magic=0:1;49:202:0:b7941532-c74b-40cc-a8ad-27b5502b8fba, cib=0.649.4,

> source=match_graph_event:381, 0)

> 2016-11-04T19:32:56.840232+01:00 ventsi-clst1 crmd[6116]:  warning:

> Action 49 (DRBD_global_clst_start_0) on ventsi-clst2-sync failed

> (target: 0 vs. rc: 1): Error

> 2016-11-04T19:32:56.840328+01:00 ventsi-clst1 crmd[6116]:   notice:

> Transition 202 (Complete=5, Pending=0, Fired=0, Skipped=0, Incomplete=5,

> Source=/var/lib/pacemaker/pengine/pe-input-766.bz2): Complete

> 2016-11-04T19:32:56.843693+01:00 ventsi-clst1 pengine[6115]:   notice:

> On loss of CCM Quorum: Ignore

> 2016-11-04T19:32:56.844072+01:00 ventsi-clst1 pengine[6115]:  warning:

> Processing failed op start for DRBD_global_clst on ventsi-clst2-sync:

> unknown error (1)

> 2016-11-04T19:32:56.844102+01:00 ventsi-clst1 pengine[6115]:  warning:

> Processing failed op start for DRBD_global_clst on ventsi-clst2-sync:

> unknown error (1)

> 2016-11-04T19:32:56.845071+01:00 ventsi-clst1 pengine[6115]:   notice:

> Start   IPaddrNFS#011(ventsi-clst2-sync)

> 2016-11-04T19:32:56.845078+01:00 ventsi-clst1 pengine[6115]:   notice:

> Start   NFSServer#011(ventsi-clst2-sync)

> 2016-11-04T19:32:56.845081+01:00 ventsi-clst1 pengine[6115]:   notice:

> Demote  DRBD:0#011(Master -> Slave ventsi-clst1-sync)   <=== here there

> would be the necessarydemote/promote ... but it's too late; the start of

> the filesystem already failed...

> 2016-11-04T19:32:56.845083+01:00 ventsi-clst1 pengine[6115]:   notice:

> Promote DRBD:1#011(Slave -> Master ventsi-clst2-sync)

> 2016-11-04T19:32:56.845084+01:00 ventsi-clst1 pengine[6115]:   notice:

> Recover DRBD_global_clst#011(Started ventsi-clst2-sync)

> 2016-11-04T19:32:56.847986+01:00 ventsi-clst1 pengine[6115]:   notice:

> Calculated Transition 203: /var/lib/pacemaker/pengine/pe-input-767.bz2

> <=== ... so the above transition gets caught by thefollowing attempt to

> repair things partially

> 2016-11-04T19:32:56.867679+01:00 ventsi-clst1 pengine[6115]:   notice:

> On loss of CCM Quorum: Ignore

> 2016-11-04T19:32:56.868074+01:00 ventsi-clst1 pengine[6115]:  warning:

> Processing failed op start for DRBD_global_clst on ventsi-clst2-sync:

> unknown error (1)

> 2016-11-04T19:32:56.868101+01:00 ventsi-clst1 pengine[6115]:  warning:

> Processing failed op start for DRBD_global_clst on ventsi-clst2-sync:

> unknown error (1)

> 2016-11-04T19:32:56.868287+01:00 ventsi-clst1 pengine[6115]:  warning:

> Forcing DRBD_global_clst away from ventsi-clst2-sync after 1000000

> failures (max=1000000)

> 2016-11-04T19:32:56.869011+01:00 ventsi-clst1 pengine[6115]:   notice:

> Start   IPaddrNFS#011(ventsi-clst1-sync)

> 2016-11-04T19:32:56.869023+01:00 ventsi-clst1 pengine[6115]:   notice:

> Recover DRBD_global_clst#011(Started ventsi-clst2-sync -> ventsi-clst1-sync)

> 2016-11-04T19:32:56.869770+01:00 ventsi-clst1 pengine[6115]:   notice:

> Calculated Transition 204: /var/lib/pacemaker/pengine/pe-input-768.bz2

> 2016-11-04T19:32:56.870065+01:00 ventsi-clst1 crmd[6116]:   notice:

> Initiating action 3: stop DRBD_global_clst_stop_0 on ventsi-clst2-sync

> 2016-11-04T19:32:56.908075+01:00 ventsi-clst1 crmd[6116]:   notice:

> Initiating action 42: start DRBD_global_clst_start_0 on

> ventsi-clst1-sync (local)

> 2016-11-04T19:32:56.931072+01:00 ventsi-clst1

> Filesystem(DRBD_global_clst)[16242]: INFO: Running start for /dev/drbd1

> on /drbdmnts/global_clst

> 2016-11-04T19:32:56.943250+01:00 ventsi-clst1 kernel: EXT4-fs (drbd1):

> warning: maximal mount count reached, running e2fsck is recommended

> 2016-11-04T19:32:56.953253+01:00 ventsi-clst1 kernel: EXT4-fs (drbd1):

> mounted filesystem with ordered data mode. Opts:

> 2016-11-04T19:32:56.964284+01:00 ventsi-clst1 crmd[6116]:   notice:

> Operation DRBD_global_clst_start_0: ok (node=ventsi-clst1-sync,

> call=1225, rc=0, cib-update=1701, confirmed=true)

> 2016-11-04T19:32:56.965104+01:00 ventsi-clst1 crmd[6116]:   notice:

> Initiating action 10: start IPaddrNFS_start_0 on ventsi-clst1-sync (local)

> 2016-11-04T19:32:56.965325+01:00 ventsi-clst1 crmd[6116]:   notice:

> Initiating action 43: monitor DRBD_global_clst_monitor_20000 on

> ventsi-clst1-sync (local)

> 2016-11-04T19:32:56.996235+01:00 ventsi-clst1 IPaddr2(IPaddrNFS)[16308]:

> INFO: Adding inet address xxx.xxx.xxx.xxx/24 with broadcast address

> xxx.xxx.xxx.255 to device bond0

> 2016-11-04T19:32:57.002059+01:00 ventsi-clst1 IPaddr2(IPaddrNFS)[16308]:

> INFO: Bringing device bond0 up

> 2016-11-04T19:32:57.008128+01:00 ventsi-clst1 IPaddr2(IPaddrNFS)[16308]:

> INFO: /usr/libexec/heartbeat/send_arp -i 200 -r 5 -p

> /var/run/resource-agents/send_arp-xxx.xxx.xxx.xxx bond0 xxx.xxx.xxx.xxx

> auto not_used not_used

> 2016-11-04T19:32:57.020159+01:00 ventsi-clst1 crmd[6116]:   notice:

> Operation IPaddrNFS_start_0: ok (node=ventsi-clst1-sync, call=1226,

> rc=0, cib-update=1703, confirmed=true)

> 2016-11-04T19:32:57.020901+01:00 ventsi-clst1 crmd[6116]:   notice:

> Initiating action 11: monitor IPaddrNFS_monitor_5000 on

> ventsi-clst1-sync (local)

> 2016-11-04T19:32:57.052231+01:00 ventsi-clst1 crmd[6116]:   notice:

> Transition 204 (Complete=6, Pending=0, Fired=0, Skipped=0, Incomplete=0,

> Source=/var/lib/pacemaker/pengine/pe-input-768.bz2): Complete

> 2016-11-04T19:32:57.052251+01:00 ventsi-clst1 crmd[6116]:   notice:

> State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS

> cause=C_FSA_INTERNAL origin=notify_crmd ]

> ==================================================================

>

> Any ideas what could be the reason for this behavior?

> And how could this be fixed?

>

>

> (I already found several articles on the internet with the

> recommendation to have two separately configured monitor operations for

> the DRBD resource configured one for the master role and another one for

> the slave role.

> Already tried this to no avail.)

>

> Regards

> Andi



_______________________________________________

Users mailing list: Users at clusterlabs.org

http://clusterlabs.org/mailman/listinfo/users



Project Home: http://www.clusterlabs.org

Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf

Bugs: http://bugs.clusterlabs.org
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://clusterlabs.org/pipermail/users/attachments/20161109/b32404f4/attachment-0001.html>


More information about the Users mailing list