<div dir="ltr">Ok, though I'm not entirely sure what your "X" and "Y" refer to in this case. I do hope it is possible to focus on the primary issue at hand while still mentioning other things encountered in the process, and while fully describing the current state of things (since I don't know what I don't know) even though much of that may not be germane to the primary issue at hand. I did not intend to imply that maintenance mode was a cause or expected solution to the issue, since the issue seems to occur with or without it. I did want to let expert readers know about it, in case it matters.<div><br></div><div>I am hopeful that Ken can glean something more than I from the logs I posted.</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Feb 9, 2021 at 9:09 AM Ulrich Windl <<a href="mailto:Ulrich.Windl@rz.uni-regensburg.de">Ulrich.Windl@rz.uni-regensburg.de</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">>>> Stuart Massey <<a href="mailto:djangoschef@gmail.com" target="_blank">djangoschef@gmail.com</a>> schrieb am 09.02.2021 um 14:07 in<br>
Nachricht<br>
<<a href="mailto:CABQ68NSgYgHdozo%2B1soYZ81-yyzpSbpcCHue5zwNqnFo1Vmu0g@mail.gmail.com" target="_blank">CABQ68NSgYgHdozo+1soYZ81-yyzpSbpcCHue5zwNqnFo1Vmu0g@mail.gmail.com</a>>:<br>
> Ulrich,<br>
> Thank you for clarifying what single-node maintenance mode entails. It is<br>
> surprising to learn that, though resource actions do not happen, a node<br>
> would send a cib update that deletes the transient attributes that are<br>
> maintained by resource monitoring activities. I think "maintenance mode"<br>
> is a distraction at this point: You are implying that this problem is<br>
> unrelated to having one node in maintenance mode, and that matches our<br>
> experience.<br>
> I don't want to do anything with the cluster with one node in maintenance<br>
> mode. I just want the currently healthy node which has DC and retains DC<br>
> and is running services not to get into this odd state where it shuts down<br>
> services and does not restart them. I want to prevent that happening again,<br>
> as similar problems and unexplained fail-overs or service restarts seem to<br>
> happen occasionally, whether anything is in maintenance mode or not.<br>
<br>
Hi!<br>
<br>
It sounds a bit like an "XY-Problem" (You are suffering from X, thinking Y is the thing to prevent X. But as Y does not prevent X you are wondering what's wrong with Y, while you should ask: "What's wrong with X?")<br>
<br>
Usually you can find a lot in the logs (talking about X). Also "crm_mon -1Arfj" gives you hints at "Failed Actions".<br>
You should investigate those. If they were one-time fault, you should consider cleanuing up the error state with "crm_resource -C ...". A failed resource also increments a "fail count" that you may want to reset either manually or automatically (like after a day). So subsequent failures will be treated differently.<br>
Once again: Get a clear image of your "X", then decide what Y should be ;-)<br>
<br>
Regards,<br>
Ulrich<br>
<br>
<br>
> Regards,<br>
> Stuart<br>
> <br>
> On Tue, Feb 9, 2021 at 2:34 AM Ulrich Windl <<br>
> <a href="mailto:Ulrich.Windl@rz.uni-regensburg.de" target="_blank">Ulrich.Windl@rz.uni-regensburg.de</a>> wrote:<br>
> <br>
>> Hi!<br>
>><br>
>> Maybe you just misunderstand what maintennce mode for a single node means:<br>
>> CIBS updates will still be performed, but not the resource actions. If CIB<br>
>> updates are sent to another node, that node will perform actions.<br>
>><br>
>> Maybe just explain what you really want to do with one node in maintenance<br>
>> mode.<br>
>> Don't expect the cluster to behave normally in maintenance mode...<br>
>><br>
>> Regards,<br>
>> Ulrich<br>
>><br>
>> >>> Stuart Massey <<a href="mailto:djangoschef@gmail.com" target="_blank">djangoschef@gmail.com</a>> schrieb am 08.02.2021 um 18:01<br>
>> in<br>
>> Nachricht<br>
>> <<a href="mailto:CABQ68NRhQ%2Bh%2BCgSBC-rAnwCd0zHUnA7XnayZLMFpYaNvi6gKKw@mail.gmail.com" target="_blank">CABQ68NRhQ+h+CgSBC-rAnwCd0zHUnA7XnayZLMFpYaNvi6gKKw@mail.gmail.com</a>>:<br>
>> > I'm wondering if anyone can advise us on next steps here and/or correct<br>
>> our<br>
>> > understanding. This seems like a race condition that causes resources to<br>
>> be<br>
>> > stopped unnecessarily. Is there a way to prevent a node from processing<br>
>> cib<br>
>> > updates from a peer while DC negotiations are underway? Our "node02" is<br>
>> > running resources fine, and since it winds up winning the DC election,<br>
>> > would continue to run them uninterrupted if it just ignored or pended the<br>
>> > cib updates it receives in the middle of the negotiation.<br>
>> > Very much appreciate all the help and discussion available on this board.<br>
>> > Regards,<br>
>> > Stuart<br>
>> ><br>
>> > On Mon, Feb 1, 2021 at 11:43 AM Stuart Massey <<a href="mailto:djangoschef@gmail.com" target="_blank">djangoschef@gmail.com</a>><br>
>> wrote:<br>
>> ><br>
>> >> Sequence seems to be:<br>
>> >><br>
>> >> - node02 is DC and master/primary, node01 is maintenance mode and<br>
>> >> slave/secondary<br>
>> >> - comms go down<br>
>> >> - node01 elects itself master, and deletes node01 status from its cib<br>
>> >> - comms come up<br>
>> >> - cluster starts reforming<br>
>> >> - node01 sends cib updates to node02<br>
>> >> - DC negotiations start, both nodes unset DC<br>
>> >> - node02 receives cib updates and process them, deleting its own<br>
>> status<br>
>> >> - DC negotiations complete with node02 winning<br>
>> >> - node02, having lost it's status, believes it cannot host resources<br>
>> >> and stops them all<br>
>> >> - for whatever reason, perhaps somehow due to the completely missing<br>
>> >> transient_attributes, node02 nevers schedules a probe for itself<br>
>> >> - we have to "refresh" manually<br>
>> >><br>
>> >><br>
>> >> On Mon, Feb 1, 2021 at 11:31 AM Ken Gaillot <<a href="mailto:kgaillot@redhat.com" target="_blank">kgaillot@redhat.com</a>><br>
>> wrote:<br>
>> >><br>
>> >>> On Mon, 2021-02-01 at 11:09 -0500, Stuart Massey wrote:<br>
>> >>> > Hi Ken,<br>
>> >>> > Thanks. In this case, transient_attributes for node02 in the cib on<br>
>> >>> > node02 which never lost quorum seem to be deleted by a request from<br>
>> >>> > node01 when node01 rejoins the cluster - IF I understand the<br>
>> >>> > pacemaker.log correctly. This causes node02 to stop resources, which<br>
>> >>> > will not be restarted until we manually refresh on node02.<br>
>> >>><br>
>> >>> Good point, it depends on which node is DC. When a cluster splits, each<br>
>> >>> side sees the other side as the one that left. When the split heals,<br>
>> >>> whichever side has the newly elected DC is the one that clears the<br>
>> >>> other.<br>
>> >>><br>
>> >>> However the DC should schedule probes for the other side, and probes<br>
>> >>> generally set the promotion score, so manual intervention shouldn't be<br>
>> >>> needed. I'd make sure that probes were scheduled, then investigate how<br>
>> >>> the agent sets the score.<br>
>> >>><br>
>> >>> > On Mon, Feb 1, 2021 at 10:59 AM Ken Gaillot <<a href="mailto:kgaillot@redhat.com" target="_blank">kgaillot@redhat.com</a>><br>
>> >>> > wrote:<br>
>> >>> > > On Fri, 2021-01-29 at 12:37 -0500, Stuart Massey wrote:<br>
>> >>> > > > Can someone help me with this?<br>
>> >>> > > > Background:<br>
>> >>> > > > > "node01" is failing, and has been placed in "maintenance" mode.<br>
>> >>> > > It<br>
>> >>> > > > > occasionally loses connectivity.<br>
>> >>> > > > > "node02" is able to run our resources<br>
>> >>> > > ><br>
>> >>> > > > Consider the following messages from pacemaker.log on "node02",<br>
>> >>> > > just<br>
>> >>> > > > after "node01" has rejoined the cluster (per "node02"):<br>
>> >>> > > > > Jan 28 14:48:03 [21933] <a href="http://node02.example.com" rel="noreferrer" target="_blank">node02.example.com</a> cib:<br>
>> >>> > > info:<br>
>> >>> > > > > cib_perform_op: --<br>
>> >>> > > > > /cib/status/node_state[@id='2']/transient_attributes[@id='2']<br>
>> >>> > > > > Jan 28 14:48:03 [21933] <a href="http://node02.example.com" rel="noreferrer" target="_blank">node02.example.com</a> cib:<br>
>> >>> > > info:<br>
>> >>> > > > > cib_perform_op: + /cib: @num_updates=309<br>
>> >>> > > > > Jan 28 14:48:03 [21933] <a href="http://node02.example.com" rel="noreferrer" target="_blank">node02.example.com</a> cib:<br>
>> >>> > > info:<br>
>> >>> > > > > cib_process_request: Completed cib_delete operation for<br>
>> >>> > > section<br>
>> >>> > > > > //node_state[@uname='<a href="http://node02.example.com" rel="noreferrer" target="_blank">node02.example.com</a><br>
>> ']/transient_attributes:<br>
>> >>> > > OK<br>
>> >>> > > > > (rc=0, origin=<a href="http://node01.example.com/crmd/3784" rel="noreferrer" target="_blank">node01.example.com/crmd/3784</a>, version=0.94.309)<br>
>> >>> > > > > Jan 28 14:48:04 [21938] <a href="http://node02.example.com" rel="noreferrer" target="_blank">node02.example.com</a> crmd:<br>
>> >>> > > info:<br>
>> >>> > > > > abort_transition_graph: Transition aborted by deletion of<br>
>> >>> > > > > transient_attributes[@id='2']: Transient attribute change |<br>
>> >>> > > > > cib=0.94.309 source=abort_unless_down:357<br>
>> >>> > > > ><br>
>> >>> > > path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']<br>
>> >>> > > > > complete=true<br>
>> >>> > > > > Jan 28 14:48:05 [21937] <a href="http://node02.example.com" rel="noreferrer" target="_blank">node02.example.com</a> pengine:<br>
>> >>> > > info:<br>
>> >>> > > > > master_color: ms_drbd_ourApp: Promoted 0 instances of a<br>
>> >>> > > possible 1<br>
>> >>> > > > > to master<br>
>> >>> > > > ><br>
>> >>> > > > The implication, it seems to me, is that "node01" has asked<br>
>> >>> > > "node02"<br>
>> >>> > > > to delete the transient-attributes for "node02". The transient-<br>
>> >>> > > > attributes should normally be:<br>
>> >>> > > > <transient_attributes id="2"><br>
>> >>> > > > <instance_attributes id="status-2"><br>
>> >>> > > > <nvpair id="status-2-master-drbd_ourApp" name="master-<br>
>> >>> > > > drbd_ourApp" value="10000"/><br>
>> >>> > > > <nvpair id="status-2-pingd" name="pingd" value="100"/><br>
>> >>> > > > </instance_attributes><br>
>> >>> > > > </transient_attributes><br>
>> >>> > > ><br>
>> >>> > > > These attributes are necessary for "node02" to be Master/Primary,<br>
>> >>> > > > correct?<br>
>> >>> > > ><br>
>> >>> > > > Why might this be happening and how do we prevent it?<br>
>> >>> > ><br>
>> >>> > > Transient attributes are always cleared when a node leaves the<br>
>> >>> > > cluster<br>
>> >>> > > (that's what makes them transient ...). It's probably coincidence<br>
>> >>> > > it<br>
>> >>> > > went through as the node rejoined.<br>
>> >>> > ><br>
>> >>> > > When the node rejoins, it will trigger another run of the<br>
>> >>> > > scheduler,<br>
>> >>> > > which will schedule a probe of all resources on the node. Those<br>
>> >>> > > probes<br>
>> >>> > > should reset the promotion score.<br>
>> >>> > > _______________________________________________<br>
>> >>> > > Manage your subscription:<br>
>> >>> > > <a href="https://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.clusterlabs.org/mailman/listinfo/users</a> <br>
>> >>> > ><br>
>> >>> > > ClusterLabs home: <a href="https://www.clusterlabs.org/" rel="noreferrer" target="_blank">https://www.clusterlabs.org/</a> <br>
>> >>> --<br>
>> >>> Ken Gaillot <<a href="mailto:kgaillot@redhat.com" target="_blank">kgaillot@redhat.com</a>><br>
>> >>><br>
>> >>> _______________________________________________<br>
>> >>> Manage your subscription:<br>
>> >>> <a href="https://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.clusterlabs.org/mailman/listinfo/users</a> <br>
>> >>><br>
>> >>> ClusterLabs home: <a href="https://www.clusterlabs.org/" rel="noreferrer" target="_blank">https://www.clusterlabs.org/</a> <br>
>> >>><br>
>> >><br>
>><br>
>><br>
>><br>
>> _______________________________________________<br>
>> Manage your subscription:<br>
>> <a href="https://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.clusterlabs.org/mailman/listinfo/users</a> <br>
>><br>
>> ClusterLabs home: <a href="https://www.clusterlabs.org/" rel="noreferrer" target="_blank">https://www.clusterlabs.org/</a> <br>
>><br>
<br>
<br>
<br>
_______________________________________________<br>
Manage your subscription:<br>
<a href="https://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.clusterlabs.org/mailman/listinfo/users</a><br>
<br>
ClusterLabs home: <a href="https://www.clusterlabs.org/" rel="noreferrer" target="_blank">https://www.clusterlabs.org/</a><br>
</blockquote></div>