[ClusterLabs] DRBD: both nodes stuck in secondary mode

Wed Jun 15 08:58:36 UTC 2016

I found out why it didn't work, fail-count for resource httpd was at 
INFINITY:

# crm_simulate -sL
[...]
<nvpair id="status-2-fail-count-httpd" name="fail-count-httpd" 
value="INFINITY"/>
[...]

To reset the fail-count:
# pcs resource failcount reset httpd

Now it works as expected :)

Next step: configure STONISH !

Thanks again !
Kevin

On 06/15/2016 02:18 PM, Kevin THIERRY wrote:
> Hello !
>
> I wasn't expecting such great explanations, thanks a lot Ken ! Also 
> thank you for your example Dimitri ! It solved the issue I had !
>
> I'm still having trouble though: when I make the primary node fail (I 
> unplug it), the secondary node starts everything as expected but the 
> httpd service. However when I plug back the primary node, all 
> resources go back to it and everything works fine, even the httpd 
> service.
>
> Starting the httpd server manually on the second node works fine 
> (using systemctl). I suspect an issue with the allocation scores but I 
> don't know how to solve it.
>
> ################################################
> # grep httpd /var/log/cluster/corosync.log
> ################################################
>
> Jun 15 11:58:27 [27192] laa-billing-backup    pengine:  warning: 
> unpack_rsc_op_failure:    Processing failed op start for httpd on 
> billing-backup-sync: unknown error (1)
> Jun 15 11:58:27 [27192] laa-billing-backup    pengine:     info: 
> native_print:    httpd    (ocf::heartbeat:apache):    Started 
> billing-primary-sync
> Jun 15 11:58:27 [27192] laa-billing-backup    pengine:     info: 
> get_failcount_full:    httpd has failed INFINITY times on 
> billing-backup-sync
> Jun 15 11:58:27 [27192] laa-billing-backup    pengine:  warning: 
> common_apply_stickiness:    Forcing httpd away from 
> billing-backup-sync after 1000000 failures (max=1000000)
> Jun 15 11:58:27 [27192] laa-billing-backup    pengine:     info: 
> rsc_merge_weights:    vip: Rolling back scores from httpd
> Jun 15 11:58:27 [27192] laa-billing-backup    pengine:     info: 
> rsc_merge_weights:    drbd:1: Rolling back scores from httpd
> Jun 15 11:58:27 [27192] laa-billing-backup    pengine:     info: 
> rsc_merge_weights:    drbd-master: Rolling back scores from httpd
> Jun 15 11:58:27 [27192] laa-billing-backup    pengine:     info: 
> rsc_merge_weights:    fs: Rolling back scores from httpd
> Jun 15 11:58:27 [27192] laa-billing-backup    pengine:     info: 
> rsc_merge_weights:    pgsql: Rolling back scores from httpd
> Jun 15 11:58:27 [27192] laa-billing-backup    pengine:     info: 
> native_color:    Resource http cannot run anywhere
> Jun 15 11:58:27 [27192] laa-billing-backup    pengine:   notice: 
> LogActions:    Stop    httpd(billing-primary-sync)
> Jun 15 11:58:27 [27193] laa-billing-backup       crmd:   notice: 
> te_rsc_command:    Initiating action 54: stop httpd_stop_0 on 
> billing-primary-sync
> Jun 15 11:58:29 [27188] laa-billing-backup        cib:     info: 
> cib_perform_op:    + 
> /cib/status/node_state[@id='1']/lrm[@id='1']/lrm_resources/lrm_resource[@id='httpd']/lrm_rsc_op[@id='httpd_last_0']: 
> @operation_key=httpd_stop_0, @operation=stop, 
> @transition-key=54:75:0:4fdfd97f-42dd-4f04-bf86-294d300df414, 
> @transition-magic=0:0;54:75:0:4fdfd97f-42dd-4f04-bf86-294d300df414, 
> @call-id=166, @last-run=1465966707, @last-rc-change=1465966707, 
> @exec-time=2118
> Jun 15 11:58:29 [27193] laa-billing-backup       crmd:     info: 
> match_graph_event:    Action httpd_stop_0 (54) confirmed on 
> billing-primary-sync (rc=0)
> On 06/14/2016 09:30 PM, Ken Gaillot wrote:
>
> ################################################
> # crm_simulate -sL
> ################################################
>
> Current cluster status:
> Online: [ billing-backup-sync billing-primary-sync ]
>
>  vip    (ocf::heartbeat:IPaddr2):    Started billing-primary-sync
>  Master/Slave Set: drbd-master [drbd]
>      Masters: [ billing-primary-sync ]
>      Slaves: [ billing-backup-sync ]
>  fs    (ocf::heartbeat:Filesystem):    Started billing-primary-sync
>  pgsql    (ocf::heartbeat:pgsql):    Started billing-primary-sync
>  httpd    (ocf::heartbeat:apache):    Started billing-primary-sync
>  Clone Set: ping-clone [ping]
>      Started: [ billing-backup-sync billing-primary-sync ]
>
> Allocation scores:
> native_color: vip allocation score on billing-backup-sync: -INFINITY
> native_color: vip allocation score on billing-primary-sync: 400
> clone_color: drbd-master allocation score on billing-backup-sync: 0
> clone_color: drbd-master allocation score on billing-primary-sync: 300
> clone_color: drbd:0 allocation score on billing-backup-sync: 0
> clone_color: drbd:0 allocation score on billing-primary-sync: 10100
> clone_color: drbd:1 allocation score on billing-backup-sync: 10100
> clone_color: drbd:1 allocation score on billing-primary-sync: 0
> native_color: drbd:0 allocation score on billing-backup-sync: 0
> native_color: drbd:0 allocation score on billing-primary-sync: 10100
> native_color: drbd:1 allocation score on billing-backup-sync: 10100
> native_color: drbd:1 allocation score on billing-primary-sync: -INFINITY
> drbd:0 promotion score on billing-primary-sync: INFINITY
> drbd:1 promotion score on billing-backup-sync: -INFINITY
> native_color: fs allocation score on billing-backup-sync: -INFINITY
> native_color: fs allocation score on billing-primary-sync: 10400
> native_color: pgsql allocation score on billing-backup-sync: -INFINITY
> native_color: pgsql allocation score on billing-primary-sync: 200
> native_color: httpd allocation score on billing-backup-sync: -INFINITY
> native_color: httpd allocation score on billing-primary-sync: 100
> clone_color: ping-clone allocation score on billing-backup-sync: 0
> clone_color: ping-clone allocation score on billing-primary-sync: 0
> clone_color: ping:0 allocation score on billing-backup-sync: 0
> clone_color: ping:0 allocation score on billing-primary-sync: 100
> clone_color: ping:1 allocation score on billing-backup-sync: 100
> clone_color: ping:1 allocation score on billing-primary-sync: 0
> native_color: ping:1 allocation score on billing-backup-sync: 100
> native_color: ping:1 allocation score on billing-primary-sync: 0
> native_color: ping:0 allocation score on billing-backup-sync: -INFINITY
> native_color: ping:0 allocation score on billing-primary-sync: 100
>
> Transition Summary:
>
> ################################################
> # Full config
> ################################################
>
> # VIP
> pcs cluster cib vip_cfg
> pcs -f vip_cfg resource create vip ocf:heartbeat:IPaddr2 \
>     ip=10.5.200.30 cidr_netmask=24 op monitor interval=30s
> pcs -f vip_cfg constraint location vip prefers billing-primary=100
> pcs cluster cib-push vip_cfg
>
> # DRBD
> pcs cluster cib drbd_cfg
> pcs -f drbd_cfg resource create drbd ocf:linbit:drbd \
>     drbd_resource=drbd0 \
>     op monitor interval=29s role="Master" \
>     op monitor interval=31s role="Slave"
> pcs -f drbd_cfg resource master drbd-master drbd \
>     master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 
> notify=true
> pcs -f drbd_cfg constraint location drbd-master prefers 
> billing-primary=100
> pcs -f drbd_cfg constraint colocation add master drbd-master with vip
> pcs -f drbd_cfg constraint order start vip then promote drbd-master
> pcs cluster cib-push drbd_cfg
>
> # FS
> pcs cluster cib fs_cfg
> pcs -f fs_cfg resource create fs Filesystem \
>     device="/dev/drbd0" directory="/data" fstype="ext4"
> pcs -f fs_cfg constraint location fs prefers billing-primary=100
> pcs -f fs_cfg constraint colocation add fs with drbd-master INFINITY 
> with-rsc-role=Master
> pcs -f fs_cfg constraint order promote drbd-master then start fs
> pcs cluster cib-push fs_cfg
>
> # PGSQL
> pcs cluster cib pgsql_cfg
> pcs -f pgsql_cfg resource create pgsql pgsql \
>     pgctl="/usr/bin/pg_ctl" \
>     psql="/usr/bin/psql" \
>     pgdata="/data/pgsql/data/" \
>     node_list="billing-primary billing-backup" \
>     restart_on_promote='true'
> pcs -f pgsql_cfg constraint location pgsql prefers billing-primary=100
> pcs -f pgsql_cfg constraint colocation add pgsql with fs INFINITY
> pcs -f pgsql_cfg constraint order start fs then start pgsql
> pcs cluster cib-push pgsql_cfg
>
> # HTTPD
> pcs cluster cib httpd_cfg
> pcs -f httpd_cfg resource create httpd ocf:heartbeat:apache  \
>     configfile=/etc/httpd/conf/httpd.conf \
>     statusurl="http://localhost/welcome" \
>     op monitor interval=1min
> pcs -f httpd_cfg constraint location httpd prefers billing-primary=100
> pcs -f httpd_cfg constraint colocation add httpd with pgsql INFINITY
> pcs -f httpd_cfg constraint order start pgsql then start httpd
> pcs cluster cib-push httpd_cfg
>
> # PING
> pcs cluster cib ping_cfg
> pcs -f ping_cfg resource create ping ocf:pacemaker:ping dampen=5s 
> multiplier=1000 host_list="10.5.200.254" --clone
> pcs -f ping_cfg constraint location vip rule score=-INFINITY pingd lt 
> 1 or not_defined pingd
> pcs cluster cib-push ping_cfg
>