[ClusterLabs] DRBD: both nodes stuck in secondary mode

Wed Jun 15 07:18:55 UTC 2016

Hello !

I wasn't expecting such great explanations, thanks a lot Ken ! Also 
thank you for your example Dimitri ! It solved the issue I had !

I'm still having trouble though: when I make the primary node fail (I 
unplug it), the secondary node starts everything as expected but the 
httpd service. However when I plug back the primary node, all resources 
go back to it and everything works fine, even the httpd service.

Starting the httpd server manually on the second node works fine (using 
systemctl). I suspect an issue with the allocation scores but I don't 
know how to solve it.

################################################
# grep httpd /var/log/cluster/corosync.log
################################################

Jun 15 11:58:27 [27192] laa-billing-backup    pengine:  warning: 
unpack_rsc_op_failure:    Processing failed op start for httpd on 
billing-backup-sync: unknown error (1)
Jun 15 11:58:27 [27192] laa-billing-backup    pengine:     info: 
native_print:    httpd    (ocf::heartbeat:apache):    Started 
billing-primary-sync
Jun 15 11:58:27 [27192] laa-billing-backup    pengine:     info: 
get_failcount_full:    httpd has failed INFINITY times on 
billing-backup-sync
Jun 15 11:58:27 [27192] laa-billing-backup    pengine:  warning: 
common_apply_stickiness:    Forcing httpd away from billing-backup-sync 
after 1000000 failures (max=1000000)
Jun 15 11:58:27 [27192] laa-billing-backup    pengine:     info: 
rsc_merge_weights:    vip: Rolling back scores from httpd
Jun 15 11:58:27 [27192] laa-billing-backup    pengine:     info: 
rsc_merge_weights:    drbd:1: Rolling back scores from httpd
Jun 15 11:58:27 [27192] laa-billing-backup    pengine:     info: 
rsc_merge_weights:    drbd-master: Rolling back scores from httpd
Jun 15 11:58:27 [27192] laa-billing-backup    pengine:     info: 
rsc_merge_weights:    fs: Rolling back scores from httpd
Jun 15 11:58:27 [27192] laa-billing-backup    pengine:     info: 
rsc_merge_weights:    pgsql: Rolling back scores from httpd
Jun 15 11:58:27 [27192] laa-billing-backup    pengine:     info: 
native_color:    Resource http cannot run anywhere
Jun 15 11:58:27 [27192] laa-billing-backup    pengine:   notice: 
LogActions:    Stop    httpd(billing-primary-sync)
Jun 15 11:58:27 [27193] laa-billing-backup       crmd:   notice: 
te_rsc_command:    Initiating action 54: stop httpd_stop_0 on 
billing-primary-sync
Jun 15 11:58:29 [27188] laa-billing-backup        cib:     info: 
cib_perform_op:    + 
/cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='httpd']/lrm_rsc_op[@id='httpd_last_0']: 
@operation_key=httpd_stop_0, @operation=stop, 
@transition-key=54:75:0:4fdfd97f-42dd-4f04-bf86-294d300df414, 
@transition-magic=0:0;54:75:0:4fdfd97f-42dd-4f04-bf86-294d300df414, 
@call-id=166, @last-run=1465966707, @last-rc-change=1465966707, 
@exec-time=2118
Jun 15 11:58:29 [27193] laa-billing-backup       crmd:     info: 
match_graph_event:    Action httpd_stop_0 (54) confirmed on 
billing-primary-sync (rc=0)
On 06/14/2016 09:30 PM, Ken Gaillot wrote:

################################################
# crm_simulate -sL
################################################

Current cluster status:
Online: [ billing-backup-sync billing-primary-sync ]

  vip    (ocf::heartbeat:IPaddr2):    Started billing-primary-sync
  Master/Slave Set: drbd-master [drbd]
      Masters: [ billing-primary-sync ]
      Slaves: [ billing-backup-sync ]
  fs    (ocf::heartbeat:Filesystem):    Started billing-primary-sync
  pgsql    (ocf::heartbeat:pgsql):    Started billing-primary-sync
  httpd    (ocf::heartbeat:apache):    Started billing-primary-sync
  Clone Set: ping-clone [ping]
      Started: [ billing-backup-sync billing-primary-sync ]

Allocation scores:
native_color: vip allocation score on billing-backup-sync: -INFINITY
native_color: vip allocation score on billing-primary-sync: 400
clone_color: drbd-master allocation score on billing-backup-sync: 0
clone_color: drbd-master allocation score on billing-primary-sync: 300
clone_color: drbd:0 allocation score on billing-backup-sync: 0
clone_color: drbd:0 allocation score on billing-primary-sync: 10100
clone_color: drbd:1 allocation score on billing-backup-sync: 10100
clone_color: drbd:1 allocation score on billing-primary-sync: 0
native_color: drbd:0 allocation score on billing-backup-sync: 0
native_color: drbd:0 allocation score on billing-primary-sync: 10100
native_color: drbd:1 allocation score on billing-backup-sync: 10100
native_color: drbd:1 allocation score on billing-primary-sync: -INFINITY
drbd:0 promotion score on billing-primary-sync: INFINITY
drbd:1 promotion score on billing-backup-sync: -INFINITY
native_color: fs allocation score on billing-backup-sync: -INFINITY
native_color: fs allocation score on billing-primary-sync: 10400
native_color: pgsql allocation score on billing-backup-sync: -INFINITY
native_color: pgsql allocation score on billing-primary-sync: 200
native_color: httpd allocation score on billing-backup-sync: -INFINITY
native_color: httpd allocation score on billing-primary-sync: 100
clone_color: ping-clone allocation score on billing-backup-sync: 0
clone_color: ping-clone allocation score on billing-primary-sync: 0
clone_color: ping:0 allocation score on billing-backup-sync: 0
clone_color: ping:0 allocation score on billing-primary-sync: 100
clone_color: ping:1 allocation score on billing-backup-sync: 100
clone_color: ping:1 allocation score on billing-primary-sync: 0
native_color: ping:1 allocation score on billing-backup-sync: 100
native_color: ping:1 allocation score on billing-primary-sync: 0
native_color: ping:0 allocation score on billing-backup-sync: -INFINITY
native_color: ping:0 allocation score on billing-primary-sync: 100

Transition Summary:

################################################
# Full config
################################################

# VIP
pcs cluster cib vip_cfg
pcs -f vip_cfg resource create vip ocf:heartbeat:IPaddr2 \
     ip=10.5.200.30 cidr_netmask=24 op monitor interval=30s
pcs -f vip_cfg constraint location vip prefers billing-primary=100
pcs cluster cib-push vip_cfg

# DRBD
pcs cluster cib drbd_cfg
pcs -f drbd_cfg resource create drbd ocf:linbit:drbd \
     drbd_resource=drbd0 \
     op monitor interval=29s role="Master" \
     op monitor interval=31s role="Slave"
pcs -f drbd_cfg resource master drbd-master drbd \
     master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true
pcs -f drbd_cfg constraint location drbd-master prefers billing-primary=100
pcs -f drbd_cfg constraint colocation add master drbd-master with vip
pcs -f drbd_cfg constraint order start vip then promote drbd-master
pcs cluster cib-push drbd_cfg

# FS
pcs cluster cib fs_cfg
pcs -f fs_cfg resource create fs Filesystem \
     device="/dev/drbd0" directory="/data" fstype="ext4"
pcs -f fs_cfg constraint location fs prefers billing-primary=100
pcs -f fs_cfg constraint colocation add fs with drbd-master INFINITY 
with-rsc-role=Master
pcs -f fs_cfg constraint order promote drbd-master then start fs
pcs cluster cib-push fs_cfg

# PGSQL
pcs cluster cib pgsql_cfg
pcs -f pgsql_cfg resource create pgsql pgsql \
     pgctl="/usr/bin/pg_ctl" \
     psql="/usr/bin/psql" \
     pgdata="/data/pgsql/data/" \
     node_list="billing-primary billing-backup" \
     restart_on_promote='true'
pcs -f pgsql_cfg constraint location pgsql prefers billing-primary=100
pcs -f pgsql_cfg constraint colocation add pgsql with fs INFINITY
pcs -f pgsql_cfg constraint order start fs then start pgsql
pcs cluster cib-push pgsql_cfg

# HTTPD
pcs cluster cib httpd_cfg
pcs -f httpd_cfg resource create httpd ocf:heartbeat:apache  \
     configfile=/etc/httpd/conf/httpd.conf \
     statusurl="http://localhost/welcome" \
     op monitor interval=1min
pcs -f httpd_cfg constraint location httpd prefers billing-primary=100
pcs -f httpd_cfg constraint colocation add httpd with pgsql INFINITY
pcs -f httpd_cfg constraint order start pgsql then start httpd
pcs cluster cib-push httpd_cfg

# PING
pcs cluster cib ping_cfg
pcs -f ping_cfg resource create ping ocf:pacemaker:ping dampen=5s 
multiplier=1000 host_list="10.5.200.254" --clone
pcs -f ping_cfg constraint location vip rule score=-INFINITY pingd lt 1 
or not_defined pingd
pcs cluster cib-push ping_cfg

-- 
Kevin THIERRY
IT System Engineer

CIT Lao Ltd. – A.T.M.
PO Box 10082
Vientiane Capital – Lao P.D.R.
Cell : +856 (0)20 2221 8623
kevin.thierry.citlao at gmail.com