<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class="">Summary: Two-node cluster setup with latest pgsql resource agent. Postgresql starts initially, but failover never happens.</div><div class=""><br class=""></div><div class="">Details:</div><div class=""><br class=""></div>I'm trying to get a cluster set up with Postgresql 9.6 in a streaming replication using named slots scenario. I'm using the latest pgsql Resource Agent, which does appear to support the named replication slot feature, and I've pulled in the various utility functions the RA uses that weren't available in my base install, so the RA itself no longer gives me errors.<div class=""><br class=""></div><div class="">Setup: Two machines, centtest1 and centtest2. Both are running CentOS 6.8. Centtest1 has an IP of 10.211.55.100, and centtest2 has an IP of 10.211.55.101. The cluster is set up and functioning, with a shared virtual IP resource at 10.211.55.200. Postgresql has been set up and tested functioning properly on both nodes with centtest1 as the master and centtest2 as the streaming replica slave. </div><div class=""><br class=""></div><div class="">I then set up the postgresql master/slave resource using the following commands:</div><div class=""><br class=""></div><div class=""><div class="">pcs resource create pgsql_96 pgsql \</div><div class="">pgctl="/usr/pgsql-9.6/bin/pg_ctl" \</div><div class="">logfile="/var/log/pgsql/test2.log" \</div><div class="">psql="/usr/pgsql-9.6/bin/psql" \</div><div class="">pgdata="/pgsql96/data" \</div><div class="">rep_mode="async" \</div><div class="">repuser="postgres" \</div><div class="">node_list="<a href="http://tcentest1.ravnalaska.net" class="">tcentest1.ravnalaska.net</a> <a href="http://centtest2.ravnalaska.net" class="">centtest2.ravnalaska.net</a>" \</div><div class="">master_ip="10.211.55.200" \</div><div class="">archive_cleanup_command="" \</div><div class="">restart_on_promote="true" \</div><div class="">replication_slot_name="centtest_2_slot" \</div><div class="">monitor_user="postgres" \</div><div class="">monitor_password="SuperSecret" \</div><div class="">op start timeout="60s" interval="0s" on-fail="restart" \</div><div class="">op monitor timeout="60s" interval="4s" on-fail="restart" \</div><div class="">op monitor timeout="60s" interval="3s" on-fail="restart" role="Master" \</div><div class="">op promote timeout="60s" interval="0s" on-fail="restart" \</div><div class="">op demote timeout="60s" interval="0s" on-fail=stop \</div><div class="">op stop timeout="60s" interval="0s" on-fail="block" \</div><div class="">op notify timeout="60s" interval="0s";</div><div class=""><br class=""></div><div class="">pcs resource master msPostgresql pgsql_96 master-max=1 master-node-max=1 clone-max=2 clone-node-max=1 notify=true</div><div class=""><br class=""></div><div class=""><div class="">pcs constraint colocation add virtual_ip with Master msPostgresql INFINITY</div><div class="">pcs constraint order promote msPostgresql then start virtual_ip symmetrical=false score=INFINITY</div><div class="">pcs constraint order demote msPostgresql then stop virtual_ip symmetrical=false score=0</div></div><div class=""><br class=""></div><div class="">My preference would be that the master runs on centtest1, so I add the following constraint as well:</div><div class=""><br class=""></div><div class="">pcs constraint location --master msPostgresql prefers <a href="http://centtest1.ravnalaska.net" class="">centtest1.ravnalaska.net</a>=50</div><div class=""><br class=""></div><div class="">When I then start the cluster, I first see *both* machines come up as "slave", which I feel is somewhat odd, however the cluster software quickly figures things out and promotes centtest2 to master. I've tried this a dozen different times, and it *always* promotes centtest2 to master - even if I put INFINITY in for the location constraint.</div><div class=""><br class=""></div><div class="">But whatever- this is a cluster, it doesn't really matter which node things are running on, as long as they are running. So the cluster is working - postgresql starts, the master process is on the same node as the IP, you can connect, etc, everything looks good. Obviously the next thing to try is failover - should the master node fail, the slave node should be promoted to master. So I try testing this by shutting down the cluster on the primary server: "pcs cluster stop"</div><div class="">...and nothing happens. The master shuts down (uncleanly, I might add - it leaves behind a lock file that prevents it from starting again until I manually remove said lock file), but the slave is never promoted to master. Neither pcs status or crm_mon show any errors, but centtest1 never becomes master.</div><div class=""><br class=""></div><div class="">If instead of stoping the cluster on centtest2, I try to simply move the master using the command "pcs resource move --master msPostgresql", I first run into the aforementioned unclean shutdown issue (lock file left behind that has to be manually removed), and after removing the lock file, I wind up with *both* nodes being slaves, and no master node. "pcs resource clear --master msPostgresql" re-promotes centtest2 to master.</div><div class=""><br class=""></div><div class="">What it looks like is that for some reason pacemaker/corosync is absolutely refusing to ever make centtest1 a master - even when I explicitly tell it to, or when it is the only node left.</div><div class=""><br class=""></div><div class="">Looking at the messages log when I do the node shutdown test I see this:</div><div class=""><br class=""></div><div class="">Oct 13 08:29:39 CentTest1 crmd[30096]: notice: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]<br class="">Oct 13 08:29:39 CentTest1 pengine[30095]: notice: On loss of CCM Quorum: Ignore<br class="">Oct 13 08:29:39 CentTest1 pengine[30095]: notice: Stop virtual_ip#011(<a href="http://centtest2.ravnalaska.net" class="">centtest2.ravnalaska.net</a>)<br class="">Oct 13 08:29:39 CentTest1 pengine[30095]: notice: Demote pgsql_96:0#011(Master -> Stopped <a href="http://centtest2.ravnalaska.net" class="">centtest2.ravnalaska.net</a>)<br class="">Oct 13 08:29:39 CentTest1 pengine[30095]: notice: Calculated Transition 193: /var/lib/pacemaker/pengine/pe-input-500.bz2<br class="">Oct 13 08:29:39 CentTest1 crmd[30096]: notice: Initiating action 43: notify pgsql_96_pre_notify_demote_0 on <a href="http://centtest2.ravnalaska.net" class="">centtest2.ravnalaska.net</a><br class="">Oct 13 08:29:39 CentTest1 crmd[30096]: notice: Initiating action 45: notify pgsql_96_pre_notify_demote_0 on <a href="http://centtest1.ravnalaska.net" class="">centtest1.ravnalaska.net</a> (local)<br class="">Oct 13 08:29:39 CentTest1 crmd[30096]: notice: Operation pgsql_96_notify_0: ok (node=<a href="http://centtest1.ravnalaska.net" class="">centtest1.ravnalaska.net</a>, call=230, rc=0, cib-update=0, confirmed=true)<br class="">Oct 13 08:29:39 CentTest1 crmd[30096]: notice: Initiating action 6: demote pgsql_96_demote_0 on <a href="http://centtest2.ravnalaska.net" class="">centtest2.ravnalaska.net</a><br class="">Oct 13 08:29:39 CentTest1 crmd[30096]: notice: Transition aborted by <a href="http://status-centtest2.ravnalaska.net" class="">status-centtest2.ravnalaska.net</a>-master-pgsql_96, master-pgsql_96=-INFINITY: Transient attribute change (modify cib=0.170.19, source=abort_unless_down:329, path=/cib/status/node_state[@id='<a href="http://centtest2.ravnalaska.net" class="">centtest2.ravnalaska.net</a>']/transient_attributes[@id='<a href="http://centtest2.ravnalaska.net" class="">centtest2.ravnalaska.net</a>']/instance_attributes[@id='<a href="http://status-centtest2.ravnalaska.net" class="">status-centtest2.ravnalaska.net</a>']/nvpair[@id='<a href="http://status-centtest2.ravnalaska.net" class="">status-centtest2.ravnalaska.net</a>-master-pgsql_96'], 0)<br class="">Oct 13 08:29:39 CentTest1 crmd[30096]: warning: No match for shutdown action on <a href="http://centtest2.ravnalaska.net" class="">centtest2.ravnalaska.net</a><br class="">Oct 13 08:29:40 CentTest1 crmd[30096]: notice: Initiating action 44: notify pgsql_96_post_notify_demote_0 on <a href="http://centtest2.ravnalaska.net" class="">centtest2.ravnalaska.net</a><br class="">Oct 13 08:29:40 CentTest1 crmd[30096]: notice: Initiating action 46: notify pgsql_96_post_notify_demote_0 on <a href="http://centtest1.ravnalaska.net" class="">centtest1.ravnalaska.net</a> (local)<br class="">Oct 13 08:29:41 CentTest1 pgsql(pgsql_96)[19150]: INFO: My master baseline : 000000000700B5C8.<br class="">Oct 13 08:29:41 CentTest1 attrd[30094]: notice: Sending flush op to all hosts for: pgsql_96-master-baseline (000000000700B5C8)<br class="">Oct 13 08:29:41 CentTest1 attrd[30094]: notice: Sent update 227: pgsql_96-master-baseline=000000000700B5C8<br class="">Oct 13 08:29:41 CentTest1 crmd[30096]: notice: Operation pgsql_96_notify_0: ok (node=<a href="http://centtest1.ravnalaska.net" class="">centtest1.ravnalaska.net</a>, call=231, rc=0, cib-update=0, confirmed=true)<br class="">Oct 13 08:29:41 CentTest1 crmd[30096]: notice: Transition 193 (Complete=12, Pending=0, Fired=0, Skipped=3, Incomplete=11, Source=/var/lib/pacemaker/pengine/pe-input-500.bz2): Stopped<br class="">Oct 13 08:29:41 CentTest1 pengine[30095]: notice: On loss of CCM Quorum: Ignore<br class="">Oct 13 08:29:41 CentTest1 pengine[30095]: notice: Stop virtual_ip#011(<a href="http://centtest2.ravnalaska.net" class="">centtest2.ravnalaska.net</a>)<br class="">Oct 13 08:29:41 CentTest1 pengine[30095]: notice: Stop pgsql_96:0#011(<a href="http://centtest2.ravnalaska.net" class="">centtest2.ravnalaska.net</a>)<br class="">Oct 13 08:29:41 CentTest1 pengine[30095]: notice: Calculated Transition 194: /var/lib/pacemaker/pengine/pe-input-501.bz2<br class="">Oct 13 08:29:41 CentTest1 crmd[30096]: notice: Initiating action 4: stop virtual_ip_stop_0 on <a href="http://centtest2.ravnalaska.net" class="">centtest2.ravnalaska.net</a><br class="">Oct 13 08:29:41 CentTest1 crmd[30096]: notice: Initiating action 38: notify pgsql_96_pre_notify_stop_0 on <a href="http://centtest2.ravnalaska.net" class="">centtest2.ravnalaska.net</a><br class="">Oct 13 08:29:41 CentTest1 crmd[30096]: notice: Initiating action 39: notify pgsql_96_pre_notify_stop_0 on <a href="http://centtest1.ravnalaska.net" class="">centtest1.ravnalaska.net</a> (local)<br class="">Oct 13 08:29:41 CentTest1 crmd[30096]: notice: Operation pgsql_96_notify_0: ok (node=<a href="http://centtest1.ravnalaska.net" class="">centtest1.ravnalaska.net</a>, call=232, rc=0, cib-update=0, confirmed=true)<br class="">Oct 13 08:29:41 CentTest1 crmd[30096]: notice: Initiating action 5: stop pgsql_96_stop_0 on <a href="http://centtest2.ravnalaska.net" class="">centtest2.ravnalaska.net</a><br class="">Oct 13 08:29:41 CentTest1 crmd[30096]: notice: Initiating action 40: notify pgsql_96_post_notify_stop_0 on <a href="http://centtest1.ravnalaska.net" class="">centtest1.ravnalaska.net</a> (local)<br class="">Oct 13 08:29:41 CentTest1 crmd[30096]: notice: Operation pgsql_96_notify_0: ok (node=<a href="http://centtest1.ravnalaska.net" class="">centtest1.ravnalaska.net</a>, call=233, rc=0, cib-update=0, confirmed=true)<br class="">Oct 13 08:29:41 CentTest1 crmd[30096]: notice: Transition 194 (Complete=12, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-501.bz2): Complete<br class="">Oct 13 08:29:41 CentTest1 crmd[30096]: notice: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]<br class="">Oct 13 08:29:41 CentTest1 crmd[30096]: notice: State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph ]<br class="">Oct 13 08:29:41 CentTest1 pengine[30095]: notice: On loss of CCM Quorum: Ignore<br class="">Oct 13 08:29:41 CentTest1 pengine[30095]: notice: Scheduling Node <a href="http://centtest2.ravnalaska.net" class="">centtest2.ravnalaska.net</a> for shutdown<br class="">Oct 13 08:29:41 CentTest1 pengine[30095]: notice: Calculated Transition 195: /var/lib/pacemaker/pengine/pe-input-502.bz2<br class="">Oct 13 08:29:41 CentTest1 crmd[30096]: notice: Transition 195 (Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-502.bz2): Complete<br class="">Oct 13 08:29:41 CentTest1 crmd[30096]: notice: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]<br class="">Oct 13 08:29:41 CentTest1 crmd[30096]: notice: do_shutdown of <a href="http://centtest2.ravnalaska.net" class="">centtest2.ravnalaska.net</a> (op 30) is complete<br class="">Oct 13 08:29:41 CentTest1 attrd[30094]: notice: crm_update_peer_proc: Node <a href="http://centtest2.ravnalaska.net" class="">centtest2.ravnalaska.net</a>[2] - state is now lost (was member)<br class="">Oct 13 08:29:41 CentTest1 attrd[30094]: notice: Removing <a href="http://centtest2.ravnalaska.net/2" class="">centtest2.ravnalaska.net/2</a> from the membership list<br class="">Oct 13 08:29:41 CentTest1 attrd[30094]: notice: Purged 1 peers with id=2 and/or uname=<a href="http://centtest2.ravnalaska.net" class="">centtest2.ravnalaska.net</a> from the membership cache<br class="">Oct 13 08:29:41 CentTest1 stonith-ng[30092]: notice: crm_update_peer_proc: Node <a href="http://centtest2.ravnalaska.net" class="">centtest2.ravnalaska.net</a>[2] - state is now lost (was member)<br class="">Oct 13 08:29:41 CentTest1 stonith-ng[30092]: notice: Removing <a href="http://centtest2.ravnalaska.net/2" class="">centtest2.ravnalaska.net/2</a> from the membership list<br class="">Oct 13 08:29:41 CentTest1 stonith-ng[30092]: notice: Purged 1 peers with id=2 and/or uname=<a href="http://centtest2.ravnalaska.net" class="">centtest2.ravnalaska.net</a> from the membership cache<br class="">Oct 13 08:29:41 CentTest1 cib[30091]: notice: crm_update_peer_proc: Node <a href="http://centtest2.ravnalaska.net" class="">centtest2.ravnalaska.net</a>[2] - state is now lost (was member)<br class="">Oct 13 08:29:41 CentTest1 cib[30091]: notice: Removing <a href="http://centtest2.ravnalaska.net/2" class="">centtest2.ravnalaska.net/2</a> from the membership list<br class="">Oct 13 08:29:41 CentTest1 cib[30091]: notice: Purged 1 peers with id=2 and/or uname=<a href="http://centtest2.ravnalaska.net" class="">centtest2.ravnalaska.net</a> from the membership cache<br class="">Oct 13 08:29:43 CentTest1 pgsql(pgsql_96)[19472]: INFO: Master does not exist.<br class="">Oct 13 08:29:43 CentTest1 pgsql(pgsql_96)[19472]: WARNING: My data is out-of-date. status=DISCONNECT<br class="">Oct 13 08:29:43 CentTest1 corosync[29835]: [QUORUM] Members[1]: 1<br class="">Oct 13 08:29:43 CentTest1 corosync[29835]: [TOTEM ] A processor joined or left the membership and a new membership was formed.<br class="">Oct 13 08:29:43 CentTest1 corosync[29835]: [CPG ] chosen downlist: sender r(0) ip(10.211.55.100) ; members(old:2 left:1)<br class="">Oct 13 08:29:43 CentTest1 corosync[29835]: [MAIN ] Completed service synchronization, ready to provide service.<br class="">Oct 13 08:29:43 CentTest1 kernel: dlm: closing connection to node 2<br class="">Oct 13 08:29:43 CentTest1 crmd[30096]: notice: cman_event_callback: Node <a href="http://centtest2.ravnalaska.net" class="">centtest2.ravnalaska.net</a>[2] - state is now lost (was member)<br class="">Oct 13 08:29:43 CentTest1 crmd[30096]: notice: do_shutdown of <a href="http://centtest2.ravnalaska.net" class="">centtest2.ravnalaska.net</a> (op 30) is complete<br class="">Oct 13 08:29:43 CentTest1 pacemakerd[30085]: notice: cman_event_callback: Node <a href="http://centtest2.ravnalaska.net" class="">centtest2.ravnalaska.net</a>[2] - state is now lost (was member)<br class="">Oct 13 08:29:43 CentTest1 crmd[30096]: notice: State transition S_IDLE -> S_INTEGRATION [ input=I_NODE_JOIN cause=C_FSA_INTERNAL origin=check_join_state ]<br class="">Oct 13 08:29:43 CentTest1 attrd[30094]: notice: Sending full refresh (origin=crmd)<br class="">Oct 13 08:29:43 CentTest1 attrd[30094]: notice: Sending flush op to all hosts for: pgsql_96-status (HS:alone)<br class="">Oct 13 08:29:43 CentTest1 attrd[30094]: notice: Sending flush op to all hosts for: shutdown (0)<br class="">Oct 13 08:29:43 CentTest1 attrd[30094]: notice: Sending flush op to all hosts for: master-pgsql_96 (-INFINITY)<br class="">Oct 13 08:29:43 CentTest1 attrd[30094]: notice: Sending flush op to all hosts for: pgsql_96-master-baseline (000000000700B5C8)<br class="">Oct 13 08:29:43 CentTest1 pengine[30095]: notice: On loss of CCM Quorum: Ignore<br class="">Oct 13 08:29:43 CentTest1 pengine[30095]: notice: Calculated Transition 196: /var/lib/pacemaker/pengine/pe-input-503.bz2<br class="">Oct 13 08:29:43 CentTest1 crmd[30096]: notice: Transition 196 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-input-503.bz2): Complete<br class="">Oct 13 08:29:43 CentTest1 crmd[30096]: notice: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]<br class="">Oct 13 08:29:47 CentTest1 pgsql(pgsql_96)[19602]: INFO: Master does not exist.<br class="">Oct 13 08:29:47 CentTest1 pgsql(pgsql_96)[19602]: WARNING: My data is out-of-date. status=DISCONNECT<br class="">Oct 13 08:29:51 CentTest1 pgsql(pgsql_96)[19730]: INFO: Master does not exist.<br class="">Oct 13 08:29:51 CentTest1 pgsql(pgsql_96)[19730]: WARNING: My data is out-of-date. status=DISCONNECT</div><div class=""><br class=""></div><div class="">Those last two lines repeat indefinitely, but there is no indication that the cluster ever tries to promote centtest1 to master. Even if I completely shut down the cluster, and bring it back up only on centtest1, pacemaker refuses to start postgresql on centtest1 as a master.</div><div class=""><br class=""></div><div class="">What can I do to fix this? What troubleshooting steps can I follow? Thanks.</div><div class=""><br class=""></div><div class=""><br class=""></div><div class="">
<div style="color: rgb(0, 0, 0); font-family: Helvetica; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; text-align: -webkit-auto; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; orphans: 2; widows: 2; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div class=""><div style="font-family: Helvetica, sans-serif; font-size: 12px;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">-----------------------------------------------<o:p class=""></o:p></span></div></div><div style="font-family: Helvetica, sans-serif; font-size: 12px;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">Israel Brewster<o:p class=""></o:p></span></div></div><div style="font-family: Helvetica, sans-serif; font-size: 12px;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">Systems Analyst II<o:p class=""></o:p></span></div></div><div style="font-family: Helvetica, sans-serif; font-size: 12px;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">Ravn Alaska<o:p class=""></o:p></span></div></div><div style="font-family: Helvetica, sans-serif; font-size: 12px;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">5245 Airport Industrial Rd<o:p class=""></o:p></span></div></div><div style="font-family: Helvetica, sans-serif; font-size: 12px;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">Fairbanks, AK 99709<o:p class=""></o:p></span></div></div><div style="font-family: Helvetica, sans-serif; font-size: 12px;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">(907) 450-7293<o:p class=""></o:p></span></div></div><div style="font-family: Helvetica, sans-serif; font-size: 12px;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">-----------------------------------------------</span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""></span><br class="Apple-interchange-newline"></div></div></div></div><span style="color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant: normal; font-weight: normal; letter-spacing: normal; line-height: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;"><br class="Apple-interchange-newline"><span></span></span></div></div></body></html>