<div dir="ltr"><br><div class="gmail_extra"><br><div class="gmail_quote">On Tue, Mar 8, 2016 at 7:03 AM, Ferenc Wágner <span dir="ltr"><<a href="mailto:wferi@niif.hu" target="_blank">wferi@niif.hu</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class="">Ken Gaillot <<a href="mailto:kgaillot@redhat.com">kgaillot@redhat.com</a>> writes:<br>
<br>
> On 03/07/2016 07:31 AM, Ferenc Wágner wrote:<br>
><br>
</span><span class="">>> 12:55:13 vhbl07 crmd[8484]: notice: Transition aborted by vm-eiffel_monitor_60000 'create' on vhbl05: Foreign event (magic=0:0;521:0:0:634eef05-39c1-4093-94d4-8d624b423bb7, cib=0.613.98, source=process_graph_event:600, 0)<br>
><br>
> That means the action was initiated by a different node (the previous DC<br>
> presumably),</span></blockquote><div><br></div><div>I suspect s/previous/other/</div><div><br></div><div>With a stuck machine its entirely possible that the other nodes elected a new leader.</div><div>Would I be right in guessing that fencing is disabled?</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex"><span class=""> so the new DC wants to recalculate everything.<br>
<br>
</span>Time travel was sort of possible in that situation, and recurring<br>
monitor operations are not logged, so this is indeed possible. The main<br>
thing is that it wasn't mishandled.<br>
<span class=""><br>
>> recovery actions turned into start actions for the resources stopped<br>
>> during the previous transition. However, almost all other recovery<br>
>> actions just disappeared without any comment. This was actually<br>
>> correct, but I really wonder why the cluster decided to paper over<br>
>> the previous monitor operation timeouts. Maybe the operations<br>
>> finished meanwhile and got accounted somehow, just not logged?<br>
><br>
> I'm not sure why the PE decided recovery was not necessary. Operation<br>
> results wouldn't be accepted without being logged.<br>
<br>
</span>At which logging level? I can't see recurring monitor operation logs in<br>
syslog (at default logging level: notice) nor in /var/log/pacemaker.log<br>
(which contains info level messages as well).<br></blockquote><div><br></div><div>The DC will log that the recurring monitor was successfully started, but due to noise it doesn't log subsequent successes.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<br>
However, the info level logs contain more "Transition aborted" lines, as<br>
if only the first of them got logged with notice level. This would make<br>
sense, since the later ones don't make any difference on an already<br>
aborted transition, so they aren't that important. And in fact such<br>
lines were suppressed from the syslog I checked first, for example:<br>
<br>
12:55:39 [8479] vhbl07 cib: info: cib_perform_op: Diff: --- 0.613.120 2<br>
12:55:39 [8479] vhbl07 cib: info: cib_perform_op: Diff: +++ 0.613.121 (null)<br>
12:55:39 [8479] vhbl07 cib: info: cib_perform_op: + /cib: @num_updates=121<br>
12:55:39 [8479] vhbl07 cib: info: cib_perform_op: ++ /cib/status/node_state[@id='167773707']/lrm[@id='167773707']/lrm_resources/lrm_resource[@id='vm-elm']: <lrm_rsc_op id="vm-elm_monitor_60000" operation_key="vm-elm_monitor_60000" operation="monitor" crm-debug-origin="do_update_resource" crm_feature_set="3.0.10" transition-key="473:0:0:634eef05-39c1-4093-94d4-8d624b423bb7" transition-magic="0:0;473:0:0:634eef05-39c1-4093-94d4-8d624b423bb7" on_node="vhbl05" call-id="645" rc-code="0" op-st<br>
12:55:39 [8479] vhbl07 cib: info: cib_process_request: Completed cib_modify operation for section status: OK (rc=0, origin=vhbl05/crmd/362, version=0.613.121)<br>
12:55:39 [8484] vhbl07 crmd: info: abort_transition_graph: Transition aborted by vm-elm_monitor_60000 'create' on vhbl05: Foreign event (magic=0:0;473:0:0:634eef05-39c1-4093-94d4-8d624b423bb7, cib=0.613.121, source=process_graph_event:600, 0)<br>
12:55:39 [8484] vhbl07 crmd: info: process_graph_event: Detected action (0.473) vm-elm_monitor_60000.645=ok: initiated by a different node<br>
<br>
I can very much imagine this cancelling the FAILED state induced by a<br>
monitor timeout like:<br>
<br>
12:54:52 [8479] vhbl07 cib: info: cib_perform_op: ++ <lrm_resource id="vm-elm" type="TransientDomain" class="ocf" provider="niif"><br>
12:54:52 [8479] vhbl07 cib: info: cib_perform_op: ++ <lrm_rsc_op id="vm-elm_last_failure_0" operation_key="vm-elm_monitor_60000" operation="monitor" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.10" transition-key="473:0:0:634eef05-39c1-4093-94d4-8d624b423bb7" transition-magic="2:1;473:0:0:634eef05-39c1-4093-94d4-8d624b423bb7" on_node="vhbl05" call-id="645" rc-code="1" op-status="2" interval="60000" last-rc-change="1456833279" exe<br>
12:54:52 [8479] vhbl07 cib: info: cib_perform_op: ++ <lrm_rsc_op id="vm-elm_last_0" operation_key="vm-elm_start_0" operation="start" crm-debug-origin="build_active_RAs" crm_feature_set="3.0.10" transition-key="472:0:0:634eef05-39c1-4093-94d4-8d624b423bb7" transition-magic="0:0;472:0:0:634eef05-39c1-4093-94d4-8d624b423bb7" on_node="vhbl05" call-id="602" rc-code="0" op-status="0" interval="0" last-run="1456091121" last-rc-change="1456091121" e<br>
12:54:52 [8479] vhbl07 cib: info: cib_perform_op: ++ </lrm_resource><br>
<br>
The transition-keys match, does this mean that the above is a late<br>
result from the monitor operation which was considered timed-out<br>
previously? How did it reach vhbl07, if the DC at that time was vhbl03?<br></blockquote><div><br></div><div>Everything goes into the cib (replicated datastore) and the DC(s) get notified.</div><div> </div><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<span class=""><br>
> The pe-input files from the transitions around here should help.<br>
<br>
</span>They are available. What shall I look for?<br>
<span class=""><br>
>> Basically, the cluster responded beyond my expectations, sparing lots of<br>
>> unnecessary recoveries or fencing. I'm happy, thanks for this wonderful<br>
>> software! But I'm left with these "Processing failed op monitor"<br>
>> warnings emitted every 15 minutes (timer pops). Is it safe and clever<br>
>> to cleanup the affected resources? Would that get rid of them without<br>
>> invoking other effects, like recoveries for example?<br>
><br>
> That's normal; it's how the cluster maintains the effect of a failure<br>
> that has not been cleared. The logs can be confusing, because it's not<br>
> apparent from that message alone whether the failure is new or old.<br>
<br>
</span>Ah, do you mean that these are the same thing that appears after "Failed<br>
Actions:" at the end of the crm_mon output? They certainly match, and<br>
the logs are confusing indeed.<br>
<span class=""><br>
> Cleaning up the resource will end the failure condition, so what happens<br>
> next depends on the configuration and state of the cluster. If the<br>
> failure was preventing a preferred node from running the resource, the<br>
> resource could move, depending on other factors such as stickiness.<br>
<br>
</span>These resources are (still) running fine, suffered only monitor failures<br>
and are node-neutral, so it should be safe to cleanup them, I suppose.<br>
<span class="HOEnZb"><font color="#888888">--<br>
Thanks for your quick and enlightening answer! I feared the mere length<br>
of my message would scare everybody away...<br>
Regards,<br>
Feri<br>
</font></span><div class="HOEnZb"><div class="h5"><br>
_______________________________________________<br>
Users mailing list: <a href="mailto:Users@clusterlabs.org">Users@clusterlabs.org</a><br>
<a href="http://clusterlabs.org/mailman/listinfo/users" rel="noreferrer" target="_blank">http://clusterlabs.org/mailman/listinfo/users</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" rel="noreferrer" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" rel="noreferrer" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" rel="noreferrer" target="_blank">http://bugs.clusterlabs.org</a><br>
</div></div></blockquote></div><br></div></div>