<div dir="ltr">A reasonable question from a practical on-the-ground perspective. Several considerations contributed:<div><ol><li>We need to induce the hardware errors to demonstrate the problem to the hardware vendor, who is acting at their own pace. Leaving the node in it's semi-active state seems to achieve this goal.<br></li><li>We would like to retain as much as possible an up-to-date secondary of mission critical data. Leaving drbd up in secondary mode under HA control was the most direct way to do this.<br></li><li>We have had occasional fail-overs over time (prior to current hardware issues) which we have had difficulty fully explaining. We chalked them up to transient communications errors with sequences and effects we didn't fully comprehend, but have remained uneasy about our particular HA configuration and about crm/pacemaker/drbd generally. This seems like a good opportunity to try and "catch" and/or fully understand this issue.<br></li></ol>Regards</div></div><br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Tue, Feb 2, 2021 at 2:11 AM Ulrich Windl <<a href="mailto:Ulrich.Windl@rz.uni-regensburg.de">Ulrich.Windl@rz.uni-regensburg.de</a>> wrote:<br></div><blockquote class="gmail_quote" style="margin:0px 0px 0px 0.8ex;border-left-width:1px;border-left-style:solid;border-left-color:rgb(204,204,204);padding-left:1ex">>>> Ken Gaillot <<a href="mailto:kgaillot@redhat.com" target="_blank">kgaillot@redhat.com</a>> schrieb am 01.02.2021 um 17:27 in<br>
Nachricht<br>
<<a href="mailto:9b99d08faf4ddbe496ede10165f586afd81aa850.camel@redhat.com" target="_blank">9b99d08faf4ddbe496ede10165f586afd81aa850.camel@redhat.com</a>>:<br>
> On Mon, 2021-02-01 at 11:16 -0500, Stuart Massey wrote:<br>
>> Andrei,<br>
>> You are right, thank you. I have an earlier thread on which I posted<br>
>> a pacemaker.log for this issue, and didn't think to point to it here.<br>
>> The link is <br>
>> <a href="http://project.ibss.net/samples/deidPacemakerLog.2021-01-25.txtxt" rel="noreferrer" target="_blank">http://project.ibss.net/samples/deidPacemakerLog.2021-01-25.txtxt</a> .<br>
>> So, node01 is in maintenance mode, and constraints prevent any<br>
>> resources from running on it (other than drbd in Secondary). I would<br>
>> not want node01 to ston[node02]ith after a communications failure,<br>
>> especially not if all resources are running fine on node02.<br>
>> Also I did not think to wonder if node01 could become DC even though<br>
>> in maintenance mode.<br>
>> The logs seem to me to match this contention. The cib ops happen<br>
>> right in the middle of the DC negotiations.<br>
>> Is there a way to tell node01 that it cannot be DC? Like a<br>
>> constraint?<br>
> <br>
> No, though that's been suggested as a new feature.<br>
<br>
I wonder: If the node is running no resources, the node is in<br>
maintenance-mode, and the node shouldn't be DC wouldn't it be easiest to<br>
cleanly shutdown the node? What would be the difference? Or asked the other<br>
way: What's the purpose of such scenario?<br>
<br>
> <br>
> As a workaround, you could restart the cluster on the less preferred<br>
> node -- the controller with the most CPU time (i.e. up the longest)<br>
> will be preferred for DC (if pacemaker versions are equal).<br>
> <br>
>> Thanks again.<br>
>> <br>
>> <br>
>> <br>
>> On Sun, Jan 31, 2021 at 1:55 AM Andrei Borzenkov <<a href="mailto:arvidjaar@gmail.com" target="_blank">arvidjaar@gmail.com</a> <br>
>> > wrote:<br>
>> > 29.01.2021 20:37, Stuart Massey пишет:<br>
>> > > Can someone help me with this?<br>
>> > > Background:<br>
>> > > <br>
>> > > "node01" is failing, and has been placed in "maintenance" mode.<br>
>> > It<br>
>> > > occasionally loses connectivity.<br>
>> > > <br>
>> > > "node02" is able to run our resources<br>
>> > > <br>
>> > > Consider the following messages from pacemaker.log on "node02",<br>
>> > just after<br>
>> > > "node01" has rejoined the cluster (per "node02"):<br>
>> > > <br>
>> > > Jan 28 14:48:03 [21933] <a href="http://node02.example.com" rel="noreferrer" target="_blank">node02.example.com</a> cib: info:<br>
>> > > cib_perform_op: --<br>
>> > > /cib/status/node_state[@id='2']/transient_attributes[@id='2']<br>
>> > > Jan 28 14:48:03 [21933] <a href="http://node02.example.com" rel="noreferrer" target="_blank">node02.example.com</a> cib: info:<br>
>> > > cib_perform_op: + /cib: @num_updates=309<br>
>> > > Jan 28 14:48:03 [21933] <a href="http://node02.example.com" rel="noreferrer" target="_blank">node02.example.com</a> cib: info:<br>
>> > > cib_process_request: Completed cib_delete operation for section<br>
>> > > //node_state[@uname='<a href="http://node02.example.com" rel="noreferrer" target="_blank">node02.example.com</a>']/transient_attributes:<br>
>> > OK (rc=0,<br>
>> > > origin=<a href="http://node01.example.com/crmd/3784" rel="noreferrer" target="_blank">node01.example.com/crmd/3784</a>, version=0.94.309)<br>
>> > > Jan 28 14:48:04 [21938] <a href="http://node02.example.com" rel="noreferrer" target="_blank">node02.example.com</a> crmd: info:<br>
>> > > abort_transition_graph: Transition aborted by deletion of<br>
>> > > transient_attributes[@id='2']: Transient attribute change |<br>
>> > cib=0.94.309<br>
>> > > source=abort_unless_down:357<br>
>> > ><br>
>> > path=/cib/status/node_state[@id='2']/transient_attributes[@id='2']<br>
>> > > complete=true<br>
>> > > Jan 28 14:48:05 [21937] <a href="http://node02.example.com" rel="noreferrer" target="_blank">node02.example.com</a> pengine: info:<br>
>> > > master_color: ms_drbd_ourApp: Promoted 0 instances of a possible<br>
>> > 1 to master<br>
>> > > <br>
>> > > The implication, it seems to me, is that "node01" has asked<br>
>> > "node02" to<br>
>> > > delete the transient-attributes for "node02". The transient-<br>
>> > attributes<br>
>> > > should normally be:<br>
>> > > <transient_attributes id="2"><br>
>> > > <instance_attributes id="status-2"><br>
>> > > <nvpair id="status-2-master-drbd_ourApp"<br>
>> > > name="master-drbd_ourApp" value="10000"/><br>
>> > > <nvpair id="status-2-pingd" name="pingd" value="100"/><br>
>> > > </instance_attributes><br>
>> > > </transient_attributes><br>
>> > > <br>
>> > > These attributes are necessary for "node02" to be Master/Primary,<br>
>> > correct?<br>
>> > > <br>
>> > > Why might this be happening and how do we prevent it?<br>
>> > > <br>
>> > <br>
>> > You do not provide enough information to answer. At the very least<br>
>> > you<br>
>> > need to show full logs from both nodes around time it happens<br>
>> > (starting<br>
>> > with both nodes losing connectivity).<br>
>> > <br>
>> > But as a wild guess - you do not use stonith, node01 becomes DC and<br>
>> > clears other node state.<br>
>> > _______________________________________________<br>
>> > Manage your subscription:<br>
>> > <a href="https://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.clusterlabs.org/mailman/listinfo/users</a> <br>
>> > <br>
>> > ClusterLabs home: <a href="https://www.clusterlabs.org/" rel="noreferrer" target="_blank">https://www.clusterlabs.org/</a> <br>
>> <br>
>> _______________________________________________<br>
>> Manage your subscription:<br>
>> <a href="https://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.clusterlabs.org/mailman/listinfo/users</a> <br>
>> <br>
>> ClusterLabs home: <a href="https://www.clusterlabs.org/" rel="noreferrer" target="_blank">https://www.clusterlabs.org/</a> <br>
> -- <br>
> Ken Gaillot <<a href="mailto:kgaillot@redhat.com" target="_blank">kgaillot@redhat.com</a>><br>
> <br>
> _______________________________________________<br>
> Manage your subscription:<br>
> <a href="https://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.clusterlabs.org/mailman/listinfo/users</a> <br>
> <br>
> ClusterLabs home: <a href="https://www.clusterlabs.org/" rel="noreferrer" target="_blank">https://www.clusterlabs.org/</a> <br>
<br>
<br>
<br>
_______________________________________________<br>
Manage your subscription:<br>
<a href="https://lists.clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">https://lists.clusterlabs.org/mailman/listinfo/users</a><br>
<br>
ClusterLabs home: <a href="https://www.clusterlabs.org/" rel="noreferrer" target="_blank">https://www.clusterlabs.org/</a><br>
</blockquote></div>