<div dir="ltr">Sequence seems to be:<div><ul><li>node02 is DC and master/primary, node01 is maintenance mode and slave/secondary</li><li>comms go down</li><li>node01 elects itself master, and deletes node01 status from its cib</li><li>comms come up</li><li>cluster starts reforming</li><li>node01 sends cib updates to node02</li><li>DC negotiations start, both nodes unset DC</li><li>node02 receives cib updates and process them, deleting its own status</li><li>DC negotiations complete with node02 winning</li><li>node02, having lost it's status, believes it cannot host resources and stops them all</li><li>for whatever reason, perhaps somehow due to the completely missing transient_attributes, node02 nevers schedules a probe for itself</li><li>we have to "refresh" manually</li></ul></div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon, Feb 1, 2021 at 11:31 AM Ken Gaillot <<a href="mailto:kgaillot@redhat.com">kgaillot@redhat.com</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">On Mon, 2021-02-01 at 11:09 -0500, Stuart Massey wrote:<br>
> Hi Ken,<br>
> Thanks. In this case, transient_attributes for node02 in the cib on<br>
> node02 which never lost quorum seem to be deleted by a request from<br>
> node01 when node01 rejoins the cluster - IF I understand the<br>
> pacemaker.log correctly. This causes node02 to stop resources, which<br>
> will not be restarted until we manually refresh on node02.<br>
<br>
Good point, it depends on which node is DC. When a cluster splits, each<br>
side sees the other side as the one that left. When the split heals,<br>
whichever side has the newly elected DC is the one that clears the<br>
other.<br>
<br>
However the DC should schedule probes for the other side, and probes<br>
generally set the promotion score, so manual intervention shouldn't be<br>
needed. I'd make sure that probes were scheduled, then investigate how<br>
the agent sets the score.<br>
<br>
> On Mon, Feb 1, 2021 at 10:59 AM Ken Gaillot <<a href="mailto:kgaillot@redhat.com" target="_blank">kgaillot@redhat.com</a>><br>
> wrote:<br>
> > On Fri, 2021-01-29 at 12:37 -0500, Stuart Massey wrote:<br>
> > > Can someone help me with this?<br>
> > > Background:<br>
> > > > "node01" is failing, and has been placed in "maintenance" mode.<br>
> > It<br>
> > > > occasionally loses connectivity.<br>
> > > > "node02" is able to run our resources<br>
> > > <br>
> > > Consider the following messages from pacemaker.log on "node02",<br>
> > just<br>
> > > after "node01" has rejoined the cluster (per "node02"):<br>
> > > > Jan 28 14:48:03 [21933] <a href="http://node02.example.com" rel="noreferrer" target="_blank">node02.example.com</a> cib: <br>
> > info:<br>
> > > > cib_perform_op: --<br>
> > > > /cib/status/node_state[@id='2']/transient_attributes[@id='2']<br>
> > > > Jan 28 14:48:03 [21933] <a href="http://node02.example.com" rel="noreferrer" target="_blank">node02.example.com</a> cib: <br>
> > info:<br>
> > > > cib_perform_op: + /cib: @num_updates=309<br>
> > > > Jan 28 14:48:03 [21933] <a href="http://node02.example.com" rel="noreferrer" target="_blank">node02.example.com</a> cib: <br>
> > info:<br>
> > > > cib_process_request: Completed cib_delete operation for<br>
> > section<br>
> > > > //node_state[@uname='<a href="http://node02.example.com" rel="noreferrer" target="_blank">node02.example.com</a>']/transient_attributes:<br>
> > OK<br>
> > > > (rc=0, origin=<a href="http://node01.example.com/crmd/3784" rel="noreferrer" target="_blank">node01.example.com/crmd/3784</a>, version=0.94.309)<br>
> > > > Jan 28 14:48:04 [21938] <a href="http://node02.example.com" rel="noreferrer" target="_blank">node02.example.com</a> crmd: <br>
> > info:<br>
> > > > abort_transition_graph: Transition aborted by deletion of<br>
> > > > transient_attributes[@id='2']: Transient attribute change |<br>
> > > > cib=0.94.309 source=abort_unless_down:357<br>
> > > ><br>
> > path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']<br>
> > > > complete=true<br>
> > > > Jan 28 14:48:05 [21937] <a href="http://node02.example.com" rel="noreferrer" target="_blank">node02.example.com</a> pengine: <br>
> > info:<br>
> > > > master_color: ms_drbd_ourApp: Promoted 0 instances of a<br>
> > possible 1<br>
> > > > to master<br>
> > > > <br>
> > > The implication, it seems to me, is that "node01" has asked<br>
> > "node02"<br>
> > > to delete the transient-attributes for "node02". The transient-<br>
> > > attributes should normally be:<br>
> > > <transient_attributes id="2"><br>
> > > <instance_attributes id="status-2"><br>
> > > <nvpair id="status-2-master-drbd_ourApp" name="master-<br>
> > > drbd_ourApp" value="10000"/><br>
> > > <nvpair id="status-2-pingd" name="pingd" value="100"/><br>
> > > </instance_attributes><br>
> > > </transient_attributes><br>
> > > <br>
> > > These attributes are necessary for "node02" to be Master/Primary,<br>
> > > correct? <br>
> > > <br>
> > > Why might this be happening and how do we prevent it?<br>
> > <br>
> > Transient attributes are always cleared when a node leaves the<br>
> > cluster<br>
> > (that's what makes them transient ...). It's probably coincidence<br>
> > it<br>
> > went through as the node rejoined.<br>
> > <br>
> > When the node rejoins, it will trigger another run of the<br>
> > scheduler,<br>
> > which will schedule a probe of all resources on the node. Those<br>
> > probes<br>
> > should reset the promotion score.<br>
> > _______________________________________________<br>
> > Manage your subscription:<br>
> > <a href="https://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.clusterlabs.org/mailman/listinfo/users</a><br>
> > <br>
> > ClusterLabs home: <a href="https://www.clusterlabs.org/" rel="noreferrer" target="_blank">https://www.clusterlabs.org/</a><br>
-- <br>
Ken Gaillot <<a href="mailto:kgaillot@redhat.com" target="_blank">kgaillot@redhat.com</a>><br>
<br>
_______________________________________________<br>
Manage your subscription:<br>
<a href="https://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.clusterlabs.org/mailman/listinfo/users</a><br>
<br>
ClusterLabs home: <a href="https://www.clusterlabs.org/" rel="noreferrer" target="_blank">https://www.clusterlabs.org/</a><br>
</blockquote></div>