<html><head><meta http-equiv="Content-Type" content="text/html charset=us-ascii"></head><body style="word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class="">On Oct 13, 2016, at 9:41 AM, Ken Gaillot <<a href="mailto:kgaillot@redhat.com" class="">kgaillot@redhat.com</a>> wrote:<br class=""><div><blockquote type="cite" class=""><br class="Apple-interchange-newline"><div class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">On 10/13/2016 12:04 PM, Israel Brewster wrote:</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">Summary: Two-node cluster setup with latest pgsql resource agent.<br class="">Postgresql starts initially, but failover never happens.<br class=""><br class="">Details:<br class=""><br class="">I'm trying to get a cluster set up with Postgresql 9.6 in a streaming<br class="">replication using named slots scenario. I'm using the latest pgsql<br class="">Resource Agent, which does appear to support the named replication slot<br class="">feature, and I've pulled in the various utility functions the RA uses<br class="">that weren't available in my base install, so the RA itself no longer<br class="">gives me errors.<br class=""><br class="">Setup: Two machines, centtest1 and centtest2. Both are running CentOS<br class="">6.8. Centtest1 has an IP of 10.211.55.100, and centtest2 has an IP of<br class="">10.211.55.101. The cluster is set up and functioning, with a shared<br class="">virtual IP resource at 10.211.55.200. Postgresql has been set up and<br class="">tested functioning properly on both nodes with centtest1 as the master<br class="">and centtest2 as the streaming replica slave.<span class="Apple-converted-space"> </span><br class=""><br class="">I then set up the postgresql master/slave resource using the following<br class="">commands:<br class=""><br class="">pcs resource create pgsql_96 pgsql \<br class="">pgctl="/usr/pgsql-9.6/bin/pg_ctl" \<br class="">logfile="/var/log/pgsql/test2.log" \<br class="">psql="/usr/pgsql-9.6/bin/psql" \<br class="">pgdata="/pgsql96/data" \<br class="">rep_mode="async" \<br class="">repuser="postgres" \<br class="">node_list="<a href="http://tcentest1.ravnalaska.net/" class="">tcentest1.ravnalaska.net</a><span class="Apple-converted-space"> </span><<a href="http://tcentest1.ravnalaska.net/" class="">http://tcentest1.ravnalaska.net</a>><br class=""><a href="http://centtest2.ravnalaska.net/" class="">centtest2.ravnalaska.net</a><span class="Apple-converted-space"> </span><<a href="http://centtest2.ravnalaska.net/" class="">http://centtest2.ravnalaska.net</a>>" \<br class="">master_ip="10.211.55.200" \<br class="">archive_cleanup_command="" \<br class="">restart_on_promote="true" \<br class="">replication_slot_name="centtest_2_slot" \<br class="">monitor_user="postgres" \<br class="">monitor_password="SuperSecret" \<br class="">op start timeout="60s" interval="0s" on-fail="restart" \<br class="">op monitor timeout="60s" interval="4s" on-fail="restart" \<br class="">op monitor timeout="60s" interval="3s" on-fail="restart" role="Master" \<br class="">op promote timeout="60s" interval="0s" on-fail="restart" \<br class="">op demote timeout="60s" interval="0s" on-fail=stop \<br class="">op stop timeout="60s" interval="0s" on-fail="block" \<br class="">op notify timeout="60s" interval="0s";<br class=""><br class="">pcs resource master msPostgresql pgsql_96 master-max=1 master-node-max=1<br class="">clone-max=2 clone-node-max=1 notify=true<br class=""><br class="">pcs constraint colocation add virtual_ip with Master msPostgresql INFINITY<br class="">pcs constraint order promote msPostgresql then start virtual_ip<br class="">symmetrical=false score=INFINITY<br class="">pcs constraint order demote msPostgresql then stop virtual_ip<br class="">symmetrical=false score=0<br class=""><br class="">My preference would be that the master runs on centtest1, so I add the<br class="">following constraint as well:<br class=""><br class="">pcs constraint location --master msPostgresql prefers<br class=""><a href="http://centtest1.ravnalaska.net/" class="">centtest1.ravnalaska.net</a>=50<br class=""><br class="">When I then start the cluster, I first see *both* machines come up as<br class="">"slave", which I feel is somewhat odd, however the cluster software<br class="">quickly figures things out and promotes centtest2 to master. I've tried<br class=""></blockquote><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">This is inherent to pacemaker's model of multistate resources. Instances</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">are always started in slave mode, and then promotion to master is a</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">separate step.</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""></div></blockquote><div><br class=""></div><div>Ok, that makes sense, and appears to be working properly then.</div><br class=""><blockquote type="cite" class=""><div class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">this a dozen different times, and it *always* promotes centtest2 to<br class="">master - even if I put INFINITY in for the location constraint.<br class=""></blockquote><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Surprisingly, location constraints do not directly support limiting to</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">one role (your "--master" option is ignored, and I'm surprised it</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">doesn't give an error). To do what you want, you need a rule, like:</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">pcs constraint location msPostgresql rule \</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class=""> role=master score=50 \</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class=""> #uname eq<span class="Apple-converted-space"> </span></span><a href="http://centtest1.ravnalaska.net/" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">centtest1.ravnalaska.net</a><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""></div></blockquote><div><br class=""></div><div>Interesting. I may give that a try, just to see what happens. Of course, ideally, the master could be on either node equally, so I shouldn't need to do something like this.</div><br class=""><blockquote type="cite" class=""><div class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">But whatever- this is a cluster, it doesn't really matter which node<br class="">things are running on, as long as they are running. So the cluster is<br class="">working - postgresql starts, the master process is on the same node as<br class="">the IP, you can connect, etc, everything looks good. Obviously the next<br class="">thing to try is failover - should the master node fail, the slave node<br class="">should be promoted to master. So I try testing this by shutting down the<br class="">cluster on the primary server: "pcs cluster stop"<br class="">...and nothing happens. The master shuts down (uncleanly, I might add -<br class="">it leaves behind a lock file that prevents it from starting again until<br class="">I manually remove said lock file), but the slave is never promoted to<br class=""></blockquote><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">This definitely needs to be corrected. What creates the lock file, and</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">how is that entity managed?</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""></div></blockquote><div><br class=""></div><div>The lock file entity is created/managed by the postgresql process itself. On launch, postgres creates the lock file to say it is running, and deletes said lock file when it shuts down. To my understanding, its role in life is to prevent a restart after an unclean shutdown so the admin is reminded to make sure that the data is in a consistent state before starting the server again.</div><br class=""><blockquote type="cite" class=""><div class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">master. Neither pcs status or crm_mon show any errors, but centtest1<br class="">never becomes master.<br class=""></blockquote><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">I remember a situation where a resource agent improperly set master</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">scores, that led to no master being promoted. I don't remember the</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">details, though.</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""></div></blockquote><div><br class=""></div><div>The interesting thing is that it's not a case of no master being promoted *in general*, but rather a case of no master being promoted on a specific node, ever. The other node works fine for promoting masters. And I set up both machines at the same time, in the same way, so they should be identical, unless I am missing something.</div><div><br class=""></div><div>The "Current DC" is centtest1, the node on which postgresql will not promote to master, if that makes any difference.</div><div><br class=""></div><blockquote type="cite" class=""><div class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class=""><br class="">If instead of stoping the cluster on centtest2, I try to simply move the<br class="">master using the command "pcs resource move --master msPostgresql", I<br class="">first run into the aforementioned unclean shutdown issue (lock file left<br class="">behind that has to be manually removed), and after removing the lock<br class="">file, I wind up with *both* nodes being slaves, and no master node. "pcs<br class="">resource clear --master msPostgresql" re-promotes centtest2 to master.<br class=""><br class="">What it looks like is that for some reason pacemaker/corosync is<br class="">absolutely refusing to ever make centtest1 a master - even when I<br class="">explicitly tell it to, or when it is the only node left.<br class=""><br class="">Looking at the messages log when I do the node shutdown test I see this:<br class=""><br class="">Oct 13 08:29:39 CentTest1 crmd[30096]: notice: State transition S_IDLE<br class="">-> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL<br class="">origin=abort_transition_graph ]<br class="">Oct 13 08:29:39 CentTest1 pengine[30095]: notice: On loss of CCM<br class="">Quorum: Ignore<br class="">Oct 13 08:29:39 CentTest1 pengine[30095]: notice: Stop<span class="Apple-converted-space"> </span><br class=""> virtual_ip#011(<a href="http://centtest2.ravnalaska.net/" class="">centtest2.ravnalaska.net</a><span class="Apple-converted-space"> </span><<a href="http://centtest2.ravnalaska.net/" class="">http://centtest2.ravnalaska.net</a>>)<br class="">Oct 13 08:29:39 CentTest1 pengine[30095]: notice:<br class="">Demote pgsql_96:0#011(Master -> Stopped<span class="Apple-converted-space"> </span><a href="http://centtest2.ravnalaska.net/" class="">centtest2.ravnalaska.net</a><br class=""><<a href="http://centtest2.ravnalaska.net/" class="">http://centtest2.ravnalaska.net</a>>)<br class="">Oct 13 08:29:39 CentTest1 pengine[30095]: notice: Calculated<br class="">Transition 193: /var/lib/pacemaker/pengine/pe-input-500.bz2<br class="">Oct 13 08:29:39 CentTest1 crmd[30096]: notice: Initiating action 43:<br class="">notify pgsql_96_pre_notify_demote_0 on<span class="Apple-converted-space"> </span><a href="http://centtest2.ravnalaska.net/" class="">centtest2.ravnalaska.net</a><br class=""><<a href="http://centtest2.ravnalaska.net/" class="">http://centtest2.ravnalaska.net</a>><br class="">Oct 13 08:29:39 CentTest1 crmd[30096]: notice: Initiating action 45:<br class="">notify pgsql_96_pre_notify_demote_0 on<span class="Apple-converted-space"> </span><a href="http://centtest1.ravnalaska.net/" class="">centtest1.ravnalaska.net</a><br class=""><<a href="http://centtest1.ravnalaska.net/" class="">http://centtest1.ravnalaska.net</a>> (local)<br class="">Oct 13 08:29:39 CentTest1 crmd[30096]: notice: Operation<br class="">pgsql_96_notify_0: ok (node=<a href="http://centtest1.ravnalaska.net/" class="">centtest1.ravnalaska.net</a><br class=""><<a href="http://centtest1.ravnalaska.net/" class="">http://centtest1.ravnalaska.net</a>>, call=230, rc=0, cib-update=0,<br class="">confirmed=true)<br class="">Oct 13 08:29:39 CentTest1 crmd[30096]: notice: Initiating action 6:<br class="">demote pgsql_96_demote_0 on<span class="Apple-converted-space"> </span><a href="http://centtest2.ravnalaska.net/" class="">centtest2.ravnalaska.net</a><br class=""><<a href="http://centtest2.ravnalaska.net/" class="">http://centtest2.ravnalaska.net</a>><br class="">Oct 13 08:29:39 CentTest1 crmd[30096]: notice: Transition aborted by<br class=""><a href="http://status-centtest2.ravnalaska.net/" class="">status-centtest2.ravnalaska.net</a><br class=""><<a href="http://status-centtest2.ravnalaska.net/" class="">http://status-centtest2.ravnalaska.net</a>>-master-pgsql_96,<br class="">master-pgsql_96=-INFINITY: Transient attribute change (modify<br class="">cib=0.170.19,<br class="">source=abort_unless_down:329, path=/cib/status/node_state[@id='<a href="http://centtest2.ravnalaska.net/" class="">centtest2.ravnalaska.net</a><br class=""><<a href="http://centtest2.ravnalaska.net/" class="">http://centtest2.ravnalaska.net</a>>']/transient_attributes[@id='<a href="http://centtest2.ravnalaska.net/" class="">centtest2.ravnalaska.net</a><br class=""><<a href="http://centtest2.ravnalaska.net/" class="">http://centtest2.ravnalaska.net</a>>']/instance_attributes[@id='<a href="http://status-centtest2.ravnalaska.net/" class="">status-centtest2.ravnalaska.net</a><br class=""><<a href="http://status-centtest2.ravnalaska.net/" class="">http://status-centtest2.ravnalaska.net</a>>']/nvpair[@id='<a href="http://status-centtest2.ravnalaska.net/" class="">status-centtest2.ravnalaska.net</a><br class=""><<a href="http://status-centtest2.ravnalaska.net/" class="">http://status-centtest2.ravnalaska.net</a>>-master-pgsql_96'], 0)<br class="">Oct 13 08:29:39 CentTest1 crmd[30096]: warning: No match for shutdown<br class="">action on<span class="Apple-converted-space"> </span><a href="http://centtest2.ravnalaska.net/" class="">centtest2.ravnalaska.net</a><span class="Apple-converted-space"> </span><<a href="http://centtest2.ravnalaska.net/" class="">http://centtest2.ravnalaska.net</a>><br class="">Oct 13 08:29:40 CentTest1 crmd[30096]: notice: Initiating action 44:<br class="">notify pgsql_96_post_notify_demote_0 on<span class="Apple-converted-space"> </span><a href="http://centtest2.ravnalaska.net/" class="">centtest2.ravnalaska.net</a><br class=""><<a href="http://centtest2.ravnalaska.net/" class="">http://centtest2.ravnalaska.net</a>><br class="">Oct 13 08:29:40 CentTest1 crmd[30096]: notice: Initiating action 46:<br class="">notify pgsql_96_post_notify_demote_0 on<span class="Apple-converted-space"> </span><a href="http://centtest1.ravnalaska.net/" class="">centtest1.ravnalaska.net</a><br class=""><<a href="http://centtest1.ravnalaska.net/" class="">http://centtest1.ravnalaska.net</a>> (local)<br class="">Oct 13 08:29:41 CentTest1 pgsql(pgsql_96)[19150]: INFO: My master<br class="">baseline : 000000000700B5C8.<br class="">Oct 13 08:29:41 CentTest1 attrd[30094]: notice: Sending flush op to<br class="">all hosts for: pgsql_96-master-baseline (000000000700B5C8)<br class="">Oct 13 08:29:41 CentTest1 attrd[30094]: notice: Sent update 227:<br class="">pgsql_96-master-baseline=000000000700B5C8<br class="">Oct 13 08:29:41 CentTest1 crmd[30096]: notice: Operation<br class="">pgsql_96_notify_0: ok (node=<a href="http://centtest1.ravnalaska.net/" class="">centtest1.ravnalaska.net</a><br class=""><<a href="http://centtest1.ravnalaska.net/" class="">http://centtest1.ravnalaska.net</a>>, call=231, rc=0, cib-update=0,<br class="">confirmed=true)<br class="">Oct 13 08:29:41 CentTest1 crmd[30096]: notice: Transition 193<br class="">(Complete=12, Pending=0, Fired=0, Skipped=3, Incomplete=11,<br class="">Source=/var/lib/pacemaker/pengine/pe-input-500.bz2): Stopped<br class="">Oct 13 08:29:41 CentTest1 pengine[30095]: notice: On loss of CCM<br class="">Quorum: Ignore<br class="">Oct 13 08:29:41 CentTest1 pengine[30095]: notice: Stop<span class="Apple-converted-space"> </span><br class=""> virtual_ip#011(<a href="http://centtest2.ravnalaska.net/" class="">centtest2.ravnalaska.net</a><span class="Apple-converted-space"> </span><<a href="http://centtest2.ravnalaska.net/" class="">http://centtest2.ravnalaska.net</a>>)<br class="">Oct 13 08:29:41 CentTest1 pengine[30095]: notice: Stop<span class="Apple-converted-space"> </span><br class=""> pgsql_96:0#011(<a href="http://centtest2.ravnalaska.net/" class="">centtest2.ravnalaska.net</a><span class="Apple-converted-space"> </span><<a href="http://centtest2.ravnalaska.net/" class="">http://centtest2.ravnalaska.net</a>>)<br class="">Oct 13 08:29:41 CentTest1 pengine[30095]: notice: Calculated<br class="">Transition 194: /var/lib/pacemaker/pengine/pe-input-501.bz2<br class="">Oct 13 08:29:41 CentTest1 crmd[30096]: notice: Initiating action 4:<br class="">stop virtual_ip_stop_0 on<span class="Apple-converted-space"> </span><a href="http://centtest2.ravnalaska.net/" class="">centtest2.ravnalaska.net</a><br class=""><<a href="http://centtest2.ravnalaska.net/" class="">http://centtest2.ravnalaska.net</a>><br class="">Oct 13 08:29:41 CentTest1 crmd[30096]: notice: Initiating action 38:<br class="">notify pgsql_96_pre_notify_stop_0 on<span class="Apple-converted-space"> </span><a href="http://centtest2.ravnalaska.net/" class="">centtest2.ravnalaska.net</a><br class=""><<a href="http://centtest2.ravnalaska.net/" class="">http://centtest2.ravnalaska.net</a>><br class="">Oct 13 08:29:41 CentTest1 crmd[30096]: notice: Initiating action 39:<br class="">notify pgsql_96_pre_notify_stop_0 on<span class="Apple-converted-space"> </span><a href="http://centtest1.ravnalaska.net/" class="">centtest1.ravnalaska.net</a><br class=""><<a href="http://centtest1.ravnalaska.net/" class="">http://centtest1.ravnalaska.net</a>> (local)<br class="">Oct 13 08:29:41 CentTest1 crmd[30096]: notice: Operation<br class="">pgsql_96_notify_0: ok (node=<a href="http://centtest1.ravnalaska.net/" class="">centtest1.ravnalaska.net</a><br class=""><<a href="http://centtest1.ravnalaska.net/" class="">http://centtest1.ravnalaska.net</a>>, call=232, rc=0, cib-update=0,<br class="">confirmed=true)<br class="">Oct 13 08:29:41 CentTest1 crmd[30096]: notice: Initiating action 5:<br class="">stop pgsql_96_stop_0 on<span class="Apple-converted-space"> </span><a href="http://centtest2.ravnalaska.net/" class="">centtest2.ravnalaska.net</a><br class=""><<a href="http://centtest2.ravnalaska.net/" class="">http://centtest2.ravnalaska.net</a>><br class="">Oct 13 08:29:41 CentTest1 crmd[30096]: notice: Initiating action 40:<br class="">notify pgsql_96_post_notify_stop_0 on<span class="Apple-converted-space"> </span><a href="http://centtest1.ravnalaska.net/" class="">centtest1.ravnalaska.net</a><br class=""><<a href="http://centtest1.ravnalaska.net/" class="">http://centtest1.ravnalaska.net</a>> (local)<br class="">Oct 13 08:29:41 CentTest1 crmd[30096]: notice: Operation<br class="">pgsql_96_notify_0: ok (node=<a href="http://centtest1.ravnalaska.net/" class="">centtest1.ravnalaska.net</a><br class=""><<a href="http://centtest1.ravnalaska.net/" class="">http://centtest1.ravnalaska.net</a>>, call=233, rc=0, cib-update=0,<br class="">confirmed=true)<br class="">Oct 13 08:29:41 CentTest1 crmd[30096]: notice: Transition 194<br class="">(Complete=12, Pending=0, Fired=0, Skipped=0, Incomplete=0,<br class="">Source=/var/lib/pacemaker/pengine/pe-input-501.bz2): Complete<br class="">Oct 13 08:29:41 CentTest1 crmd[30096]: notice: State transition<br class="">S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL<br class="">origin=notify_crmd ]<br class="">Oct 13 08:29:41 CentTest1 crmd[30096]: notice: State transition S_IDLE<br class="">-> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL<br class="">origin=abort_transition_graph ]<br class="">Oct 13 08:29:41 CentTest1 pengine[30095]: notice: On loss of CCM<br class="">Quorum: Ignore<br class="">Oct 13 08:29:41 CentTest1 pengine[30095]: notice: Scheduling Node<br class=""><a href="http://centtest2.ravnalaska.net/" class="">centtest2.ravnalaska.net</a><span class="Apple-converted-space"> </span><<a href="http://centtest2.ravnalaska.net/" class="">http://centtest2.ravnalaska.net</a>> for shutdown<br class="">Oct 13 08:29:41 CentTest1 pengine[30095]: notice: Calculated<br class="">Transition 195: /var/lib/pacemaker/pengine/pe-input-502.bz2<br class="">Oct 13 08:29:41 CentTest1 crmd[30096]: notice: Transition 195<br class="">(Complete=1, Pending=0, Fired=0, Skipped=0, Incomplete=0,<br class="">Source=/var/lib/pacemaker/pengine/pe-input-502.bz2): Complete<br class="">Oct 13 08:29:41 CentTest1 crmd[30096]: notice: State transition<br class="">S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL<br class="">origin=notify_crmd ]<br class="">Oct 13 08:29:41 CentTest1 crmd[30096]: notice: do_shutdown of<br class=""><a href="http://centtest2.ravnalaska.net/" class="">centtest2.ravnalaska.net</a><span class="Apple-converted-space"> </span><<a href="http://centtest2.ravnalaska.net/" class="">http://centtest2.ravnalaska.net</a>> (op 30) is<br class="">complete<br class="">Oct 13 08:29:41 CentTest1 attrd[30094]: notice: crm_update_peer_proc:<br class="">Node<span class="Apple-converted-space"> </span><a href="http://centtest2.ravnalaska.net/" class="">centtest2.ravnalaska.net</a><span class="Apple-converted-space"> </span><<a href="http://centtest2.ravnalaska.net/" class="">http://centtest2.ravnalaska.net</a>>[2] -<br class="">state is now lost (was member)<br class="">Oct 13 08:29:41 CentTest1 attrd[30094]: notice: Removing<br class=""><a href="http://centtest2.ravnalaska.net/2" class="">centtest2.ravnalaska.net/2</a><span class="Apple-converted-space"> </span><<a href="http://centtest2.ravnalaska.net/2" class="">http://centtest2.ravnalaska.net/2</a>> from the<br class="">membership list<br class="">Oct 13 08:29:41 CentTest1 attrd[30094]: notice: Purged 1 peers with<br class="">id=2 and/or uname=<a href="http://centtest2.ravnalaska.net/" class="">centtest2.ravnalaska.net</a><br class=""><<a href="http://centtest2.ravnalaska.net/" class="">http://centtest2.ravnalaska.net</a>> from the membership cache<br class="">Oct 13 08:29:41 CentTest1 stonith-ng[30092]: notice:<br class="">crm_update_peer_proc: Node<span class="Apple-converted-space"> </span><a href="http://centtest2.ravnalaska.net/" class="">centtest2.ravnalaska.net</a><br class=""><<a href="http://centtest2.ravnalaska.net/" class="">http://centtest2.ravnalaska.net</a>>[2] - state is now lost (was member)<br class="">Oct 13 08:29:41 CentTest1 stonith-ng[30092]: notice: Removing<br class=""><a href="http://centtest2.ravnalaska.net/2" class="">centtest2.ravnalaska.net/2</a><span class="Apple-converted-space"> </span><<a href="http://centtest2.ravnalaska.net/2" class="">http://centtest2.ravnalaska.net/2</a>> from the<br class="">membership list<br class="">Oct 13 08:29:41 CentTest1 stonith-ng[30092]: notice: Purged 1 peers<br class="">with id=2 and/or uname=<a href="http://centtest2.ravnalaska.net/" class="">centtest2.ravnalaska.net</a><br class=""><<a href="http://centtest2.ravnalaska.net/" class="">http://centtest2.ravnalaska.net</a>> from the membership cache<br class="">Oct 13 08:29:41 CentTest1 cib[30091]: notice: crm_update_peer_proc:<br class="">Node<span class="Apple-converted-space"> </span><a href="http://centtest2.ravnalaska.net/" class="">centtest2.ravnalaska.net</a><span class="Apple-converted-space"> </span><<a href="http://centtest2.ravnalaska.net/" class="">http://centtest2.ravnalaska.net</a>>[2] -<br class="">state is now lost (was member)<br class="">Oct 13 08:29:41 CentTest1 cib[30091]: notice: Removing<br class=""><a href="http://centtest2.ravnalaska.net/2" class="">centtest2.ravnalaska.net/2</a><span class="Apple-converted-space"> </span><<a href="http://centtest2.ravnalaska.net/2" class="">http://centtest2.ravnalaska.net/2</a>> from the<br class="">membership list<br class="">Oct 13 08:29:41 CentTest1 cib[30091]: notice: Purged 1 peers with id=2<br class="">and/or uname=<a href="http://centtest2.ravnalaska.net/" class="">centtest2.ravnalaska.net</a><span class="Apple-converted-space"> </span><<a href="http://centtest2.ravnalaska.net/" class="">http://centtest2.ravnalaska.net</a>><br class="">from the membership cache<br class="">Oct 13 08:29:43 CentTest1 pgsql(pgsql_96)[19472]: INFO: Master does not<br class="">exist.<br class="">Oct 13 08:29:43 CentTest1 pgsql(pgsql_96)[19472]: WARNING: My data is<br class="">out-of-date. status=DISCONNECT<br class=""></blockquote><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">I'm not familiar with pgsql, but this doesn't look right. The instance</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">on centtest2 was cleanly demoted and stopped, so I'd expect pgsql to be</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">consistent at this point.</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""></div></blockquote><div><br class=""></div><div>Agreed. The complaint about master not existing makes sense, but data being out-of-date? It might simply be complaining because it hasn't heard from the master in a while though.</div><div><br class=""></div><div><br class=""></div><div><div style="text-align: -webkit-auto; font-variant-ligatures: normal; font-variant-position: normal; font-variant-numeric: normal; font-variant-alternates: normal; font-variant-east-asian: normal; line-height: normal; orphans: 2; widows: 2; word-wrap: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space;" class=""><div style="font-family: Helvetica, sans-serif;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">-----------------------------------------------<o:p class=""></o:p></span></div></div><div style="font-family: Helvetica, sans-serif;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">Israel Brewster<o:p class=""></o:p></span></div></div><div style="font-family: Helvetica, sans-serif;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">Systems Analyst II<o:p class=""></o:p></span></div></div><div style="font-family: Helvetica, sans-serif;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">Ravn Alaska<o:p class=""></o:p></span></div></div><div style="font-family: Helvetica, sans-serif;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">5245 Airport Industrial Rd<o:p class=""></o:p></span></div></div><div style="font-family: Helvetica, sans-serif;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">Fairbanks, AK 99709<o:p class=""></o:p></span></div></div><div style="font-family: Helvetica, sans-serif;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">(907) 450-7293<o:p class=""></o:p></span></div></div><div style="font-family: Helvetica, sans-serif;" class=""><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class="">-----------------------------------------------</span></div><div style="margin: 0in 0in 0.0001pt; font-size: 12pt; font-family: 'Times New Roman', serif;" class=""><span style="font-size: 9pt; font-family: Helvetica, sans-serif;" class=""></span></div></div></div></div><br class=""><blockquote type="cite" class=""><div class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><blockquote type="cite" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">Oct 13 08:29:43 CentTest1 corosync[29835]: [QUORUM] Members[1]: 1<br class="">Oct 13 08:29:43 CentTest1 corosync[29835]: [TOTEM ] A processor joined<br class="">or left the membership and a new membership was formed.<br class="">Oct 13 08:29:43 CentTest1 corosync[29835]: [CPG ] chosen downlist:<br class="">sender r(0) ip(10.211.55.100) ; members(old:2 left:1)<br class="">Oct 13 08:29:43 CentTest1 corosync[29835]: [MAIN ] Completed service<br class="">synchronization, ready to provide service.<br class="">Oct 13 08:29:43 CentTest1 kernel: dlm: closing connection to node 2<br class="">Oct 13 08:29:43 CentTest1 crmd[30096]: notice: cman_event_callback:<br class="">Node<span class="Apple-converted-space"> </span><a href="http://centtest2.ravnalaska.net/" class="">centtest2.ravnalaska.net</a><span class="Apple-converted-space"> </span><<a href="http://centtest2.ravnalaska.net/" class="">http://centtest2.ravnalaska.net</a>>[2] -<br class="">state is now lost (was member)<br class="">Oct 13 08:29:43 CentTest1 crmd[30096]: notice: do_shutdown of<br class=""><a href="http://centtest2.ravnalaska.net/" class="">centtest2.ravnalaska.net</a><span class="Apple-converted-space"> </span><<a href="http://centtest2.ravnalaska.net/" class="">http://centtest2.ravnalaska.net</a>> (op 30) is<br class="">complete<br class="">Oct 13 08:29:43 CentTest1 pacemakerd[30085]: notice:<br class="">cman_event_callback: Node<span class="Apple-converted-space"> </span><a href="http://centtest2.ravnalaska.net/" class="">centtest2.ravnalaska.net</a><br class=""><<a href="http://centtest2.ravnalaska.net/" class="">http://centtest2.ravnalaska.net</a>>[2] - state is now lost (was member)<br class="">Oct 13 08:29:43 CentTest1 crmd[30096]: notice: State transition S_IDLE<br class="">-> S_INTEGRATION [ input=I_NODE_JOIN cause=C_FSA_INTERNAL<br class="">origin=check_join_state ]<br class="">Oct 13 08:29:43 CentTest1 attrd[30094]: notice: Sending full refresh<br class="">(origin=crmd)<br class="">Oct 13 08:29:43 CentTest1 attrd[30094]: notice: Sending flush op to<br class="">all hosts for: pgsql_96-status (HS:alone)<br class="">Oct 13 08:29:43 CentTest1 attrd[30094]: notice: Sending flush op to<br class="">all hosts for: shutdown (0)<br class="">Oct 13 08:29:43 CentTest1 attrd[30094]: notice: Sending flush op to<br class="">all hosts for: master-pgsql_96 (-INFINITY)<br class="">Oct 13 08:29:43 CentTest1 attrd[30094]: notice: Sending flush op to<br class="">all hosts for: pgsql_96-master-baseline (000000000700B5C8)<br class="">Oct 13 08:29:43 CentTest1 pengine[30095]: notice: On loss of CCM<br class="">Quorum: Ignore<br class="">Oct 13 08:29:43 CentTest1 pengine[30095]: notice: Calculated<br class="">Transition 196: /var/lib/pacemaker/pengine/pe-input-503.bz2<br class="">Oct 13 08:29:43 CentTest1 crmd[30096]: notice: Transition 196<br class="">(Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,<br class="">Source=/var/lib/pacemaker/pengine/pe-input-503.bz2): Complete<br class="">Oct 13 08:29:43 CentTest1 crmd[30096]: notice: State transition<br class="">S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL<br class="">origin=notify_crmd ]<br class="">Oct 13 08:29:47 CentTest1 pgsql(pgsql_96)[19602]: INFO: Master does not<br class="">exist.<br class="">Oct 13 08:29:47 CentTest1 pgsql(pgsql_96)[19602]: WARNING: My data is<br class="">out-of-date. status=DISCONNECT<br class="">Oct 13 08:29:51 CentTest1 pgsql(pgsql_96)[19730]: INFO: Master does not<br class="">exist.<br class="">Oct 13 08:29:51 CentTest1 pgsql(pgsql_96)[19730]: WARNING: My data is<br class="">out-of-date. status=DISCONNECT<br class=""><br class="">Those last two lines repeat indefinitely, but there is no indication<br class="">that the cluster ever tries to promote centtest1 to master. Even if I<br class="">completely shut down the cluster, and bring it back up only on<br class="">centtest1, pacemaker refuses to start postgresql on centtest1 as a master.<br class=""><br class="">What can I do to fix this? What troubleshooting steps can I follow? Thanks.<br class=""><br class=""><br class="">-----------------------------------------------<br class="">Israel Brewster<br class="">Systems Analyst II<br class="">Ravn Alaska<br class="">5245 Airport Industrial Rd<br class="">Fairbanks, AK 99709<br class="">(907) 450-7293<br class="">-----------------------------------------------<br class=""></blockquote><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">_______________________________________________</span><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Users mailing list:<span class="Apple-converted-space"> </span></span><a href="mailto:Users@clusterlabs.org" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">Users@clusterlabs.org</a><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><a href="http://clusterlabs.org/mailman/listinfo/users" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">http://clusterlabs.org/mailman/listinfo/users</a><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Project Home:<span class="Apple-converted-space"> </span></span><a href="http://www.clusterlabs.org/" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">http://www.clusterlabs.org</a><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Getting started:<span class="Apple-converted-space"> </span></span><a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px;" class=""><span style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-stroke-width: 0px; float: none; display: inline !important;" class="">Bugs:<span class="Apple-converted-space"> </span></span><a href="http://bugs.clusterlabs.org/" style="font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: normal; letter-spacing: normal; orphans: auto; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; widows: auto; word-spacing: 0px; -webkit-text-size-adjust: auto; -webkit-text-stroke-width: 0px;" class="">http://bugs.clusterlabs.org</a></div></blockquote></div><br class=""></body></html>