<div dir="ltr"> Hi,<div><div><br></div><div>I've checked the resource graphs I have, and the resources were fine, so I think it's not a problem due to a high use of memory or something like that.</div><div>And unfortunately I don't have a core dump to analize(I'll enable it for a future case) so the only thing I have are the logs.</div><div><br></div><div>For the line below, I though that was the process in charge to monitore nginx what was killed due to a segfault:</div><div><br></div><div>RA output: (Nginx-rsc:monitor:stderr) Segmentation fault</div><div><br></div><div> </div><div>I've checked the Nginx logs, and there is nothing worth there, actually there is no activity, so I think it has to be something internal what caused the failure.</div><div>I'll enable coredumps, it's the only thing I can do for now.</div><div><br></div><div>Thank you very much</div><div><br></div><div>Oscar</div></div></div><div class="gmail_extra"><br><div class="gmail_quote">2015-01-27 10:39 GMT+01:00 Dejan Muhamedagic <span dir="ltr"><<a href="mailto:dejanmm@fastmail.fm" target="_blank">dejanmm@fastmail.fm</a>></span>:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi,<br>
<div><div class="h5"><br>
On Mon, Jan 26, 2015 at 06:20:35PM +0100, Oscar Salvador wrote:<br>
> Hi!<br>
><br>
> I'm writing here because two days ago I experienced a strange problem in my<br>
> Pacemaker Cluster.<br>
> Everything was working fine, till suddenly a Segfault in Nginx monitor<br>
> resource happened:<br>
><br>
> Jan 25 03:55:24 lb02 crmd: [9975]: notice: run_graph: ==== Transition 7551<br>
> (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,<br>
> Source=/var/lib/pengine/pe-input-90.bz2): Complete<br>
> Jan 25 03:55:24 lb02 crmd: [9975]: notice: do_state_transition: State<br>
> transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS<br>
> cause=C_FSA_INTERNAL origin=notify_crmd ]<br>
> Jan 25 04:00:08 lb02 cib: [9971]: info: cib_stats: Processed 1 operations<br>
> (0.00us average, 0% utilization) in the last 10min<br>
> Jan 25 04:10:24 lb02 crmd: [9975]: info: crm_timer_popped: PEngine Recheck<br>
> Timer (I_PE_CALC) just popped (900000ms)<br>
> Jan 25 04:10:24 lb02 crmd: [9975]: notice: do_state_transition: State<br>
> transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_TIMER_POPPED<br>
> origin=crm_timer_popped ]<br>
> Jan 25 04:10:24 lb02 crmd: [9975]: info: do_state_transition: Progressed to<br>
> state S_POLICY_ENGINE after C_TIMER_POPPED<br>
> Jan 25 04:10:24 lb02 pengine: [10028]: WARN: unpack_rsc_op: Processing<br>
> failed op Ldirector-rsc_last_failure_0 on lb02: not running (7)<br>
> Jan 25 04:10:24 lb02 pengine: [10028]: notice: common_apply_stickiness:<br>
> Ldirector-rsc can fail 999997 more times on lb02 before being forced off<br>
> Jan 25 04:10:24 lb02 crmd: [9975]: notice: do_state_transition: State<br>
> transition S_POLICY_ENGINE -> S_TRANSITION_ENGINE [ input=I_PE_SUCCESS<br>
> cause=C_IPC_MESSAGE origin=handle_response ]<br>
> Jan 25 04:10:24 lb02 pengine: [10028]: notice: process_pe_message:<br>
> Transition 7552: PEngine Input stored in: /var/lib/pengine/pe-input-90.bz2<br>
> Jan 25 04:10:24 lb02 crmd: [9975]: info: do_te_invoke: Processing graph<br>
> 7552 (ref=pe_calc-dc-1422155424-7644) derived from<br>
> /var/lib/pengine/pe-input-90.bz2<br>
> Jan 25 04:10:24 lb02 crmd: [9975]: notice: run_graph: ==== Transition 7552<br>
> (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0,<br>
> Source=/var/lib/pengine/pe-input-90.bz2): Complete<br>
> Jan 25 04:10:24 lb02 crmd: [9975]: notice: do_state_transition: State<br>
> transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS<br>
> cause=C_FSA_INTERNAL origin=notify_crmd ]<br>
><br>
><br>
> Jan 25 04:10:30 lb02 lrmd: [9972]: info: RA output:<br>
> (Nginx-rsc:monitor:stderr) Segmentation fault ******* here it starts<br>
<br>
</div></div>What exactly did segfault? Do you have a core dump to examine?<br>
<span class=""><br>
> As you can see, the last line.<br>
> And then:<br>
><br>
> Jan 25 04:10:30 lb02 lrmd: [9972]: info: RA output:<br>
> (Nginx-rsc:monitor:stderr) Killed<br>
> /usr/lib/ocf/resource.d//heartbeat/nginx: 910:<br>
> /usr/lib/ocf/resource.d//heartbeat/nginx: Cannot fork<br>
<br>
</span>This could be related to the segfault, or due to other serious<br>
system error.<br>
<div><div class="h5"><br>
> I guess here Nginx was killed.<br>
><br>
> And then I have some others errors till Pacemaker decide to move the<br>
> resources to the node:<br>
><br>
> Jan 25 04:10:30 lb02 crmd: [9975]: info: process_lrm_event: LRM operation<br>
> Nginx-rsc_monitor_10000 (call=52, rc=2, cib-update=7633, confirmed=false)<br>
> invalid parameter<br>
> Jan 25 04:10:30 lb02 crmd: [9975]: info: process_graph_event: Detected<br>
> action Nginx-rsc_monitor_10000 from a different transition: 5739 vs. 7552<br>
> Jan 25 04:10:30 lb02 crmd: [9975]: info: abort_transition_graph:<br>
> process_graph_event:476 - Triggered transition abort (complete=1,<br>
> tag=lrm_rsc_op, id=Nginx-rsc_last_failure_0,<br>
> magic=0:2;4:5739:0:42d1ed53-9686-4174-84e7-d2c230ed8832, cib=<br>
> 3.14.40) : Old event<br>
> Jan 25 04:10:30 lb02 crmd: [9975]: WARN: update_failcount: Updating<br>
> failcount for Nginx-rsc on lb02 after failed monitor: rc=2 (update=value++,<br>
> time=1422155430)<br>
> Jan 25 04:10:30 lb02 crmd: [9975]: notice: do_state_transition: State<br>
> transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_FSA_INTERNAL<br>
> origin=abort_transition_graph ]<br>
> Jan 25 04:10:30 lb02 attrd: [9974]: info: log-rotate detected on logfile<br>
> /var/log/ha-log<br>
> Jan 25 04:10:30 lb02 attrd: [9974]: notice: attrd_trigger_update: Sending<br>
> flush op to all hosts for: fail-count-Nginx-rsc (1)<br>
> Jan 25 04:10:30 lb02 pengine: [10028]: ERROR: unpack_rsc_op: Preventing<br>
> Nginx-rsc from re-starting on lb02: operation monitor failed 'invalid<br>
> parameter' (rc=2)<br>
> Jan 25 04:10:30 lb02 pengine: [10028]: WARN: unpack_rsc_op: Processing<br>
> failed op Nginx-rsc_last_failure_0 on lb02: invalid parameter (2)<br>
> Jan 25 04:10:30 lb02 pengine: [10028]: WARN: unpack_rsc_op: Processing<br>
> failed op Ldirector-rsc_last_failure_0 on lb02: not running (7)<br>
> Jan 25 04:10:30 lb02 pengine: [10028]: notice: common_apply_stickiness:<br>
> Ldirector-rsc can fail 999997 more times on lb02 before being forced off<br>
> Jan 25 04:10:30 lb02 pengine: [10028]: notice: LogActions: Stop<br>
> IP-rsc_mysql (lb02)<br>
> Jan 25 04:10:30 lb02 pengine: [10028]: notice: LogActions: Stop<br>
> IP-rsc_nginx (lb02)<br>
> Jan 25 04:10:30 lb02 pengine: [10028]: notice: LogActions: Stop<br>
> IP-rsc_nginx6 (lb02)<br>
> Jan 25 04:10:30 lb02 pengine: [10028]: notice: LogActions: Stop<br>
> IP-rsc_elasticsearch (lb02)<br>
> Jan 25 04:10:30 lb02 pengine: [10028]: notice: LogActions: Move<br>
> Ldirector-rsc (Started lb02 -> lb01)<br>
> Jan 25 04:10:30 lb02 pengine: [10028]: notice: LogActions: Move<br>
> Nginx-rsc (Started lb02 -> lb01)<br>
> Jan 25 04:10:30 lb02 attrd: [9974]: notice: attrd_perform_update: Sent<br>
> update 23: fail-count-Nginx-rsc=1<br>
> Jan 25 04:10:30 lb02 attrd: [9974]: notice: attrd_trigger_update: Sending<br>
> flush op to all hosts for: last-failure-Nginx-rsc (1422155430)<br>
><br>
> I see that Pacemaker is complaining about some errors like "invalid<br>
> paraemter", for example in these lines:<br>
<br>
</div></div>That error code is what the nginx RA exited with. It's unusual,<br>
but perhaps also due to the segfault.<br>
<br>
Thanks,<br>
<br>
Dejan<br>
<div class="HOEnZb"><div class="h5"><br>
> Jan 25 04:10:30 lb02 crmd: [9975]: info: process_lrm_event: LRM operation<br>
> Nginx-rsc_monitor_10000 (call=52, rc=2, cib-update=7633, confirmed=false)<br>
> invalid parameter<br>
><br>
> Jan 25 04:10:30 lb02 pengine: [10028]: ERROR: unpack_rsc_op: Preventing<br>
> Nginx-rsc from re-starting on lb02: operation monitor failed 'invalid<br>
> parameter' (rc=2)<br>
><br>
> It sounds(for me) like a syntax problem defining the resources, but I've<br>
> checked the confic with crm_verify and there is no error:<br>
><br>
> root# (S) crm_verify -LVV<br>
> root# (S)<br>
><br>
> So I'm just wondering why pacemaker is complaining about an invalid<br>
> parameter.<br>
><br>
> This is my CIB objetcs:<br>
><br>
> node $id="43b2c5a1-9552-4438-962b-6e98a2dd67c7" lb01<br>
> node $id="68328520-68e0-42fd-9adf-062655691643" lb02<br>
> primitive IP-rsc_elasticsearch ocf:heartbeat:IPaddr2 \<br>
> params ip="xx.xx.xx.xx" nic="eth0" cidr_netmask="255.255.255.224"<br>
> primitive IP-rsc_elasticsearch6 ocf:heartbeat:IPv6addr \<br>
> params ipv6addr="xxxxxxxxxxxxxxxx" \<br>
> op monitor interval="10s"<br>
> primitive IP-rsc_mysql ocf:heartbeat:IPaddr2 \<br>
> params ip="xx.xx.xx.xx" nic="eth0" cidr_netmask="255.255.255.224"<br>
> primitive IP-rsc_mysql6 ocf:heartbeat:IPv6addr \<br>
> params ipv6addr="xxxxxxxxxxxxxx" \<br>
> op monitor interval="10s"<br>
> primitive IP-rsc_nginx ocf:heartbeat:IPaddr2 \<br>
> params ip="xx.xx.xx.xx" nic="eth0" cidr_netmask="255.255.255.224"<br>
> primitive IP-rsc_nginx6 ocf:heartbeat:IPv6addr \<br>
> params ipv6addr="xxxxxxxxxxxxxx" \<br>
> op monitor interval="10s"<br>
> primitive Ldirector-rsc ocf:heartbeat:ldirectord \<br>
> op monitor interval="10s" timeout="30s"<br>
> primitive Nginx-rsc ocf:heartbeat:nginx \<br>
> op monitor interval="10s" timeout="30s"<br>
> location cli-standby-IP-rsc_elasticsearch6 IP-rsc_elasticsearch6 \<br>
> rule $id="cli-standby-rule-IP-rsc_elasticsearch6" -inf: #uname eq lb01<br>
> location cli-standby-IP-rsc_mysql IP-rsc_mysql \<br>
> rule $id="cli-standby-rule-IP-rsc_mysql" -inf: #uname eq lb01<br>
> location cli-standby-IP-rsc_mysql6 IP-rsc_mysql6 \<br>
> rule $id="cli-standby-rule-IP-rsc_mysql6" -inf: #uname eq lb01<br>
> location cli-standby-IP-rsc_nginx IP-rsc_nginx \<br>
> rule $id="cli-standby-rule-IP-rsc_nginx" -inf: #uname eq lb01<br>
> location cli-standby-IP-rsc_nginx6 IP-rsc_nginx6 \<br>
> rule $id="cli-standby-rule-IP-rsc_nginx6" -inf: #uname eq lb01<br>
> colocation hcu_c inf: Nginx-rsc Ldirector-rsc IP-rsc_mysql IP-rsc_nginx<br>
> IP-rsc_nginx6 IP-rsc_elasticsearch<br>
> order hcu_o inf: IP-rsc_nginx IP-rsc_nginx6 IP-rsc_mysql Ldirector-rsc<br>
> Nginx-rsc IP-rsc_elasticsearch<br>
> property $id="cib-bootstrap-options" \<br>
> dc-version="1.1.7-ee0730e13d124c3d58f00016c3376a1de5323cff" \<br>
> cluster-infrastructure="Heartbeat" \<br>
> stonith-enabled="false<br>
><br>
><br>
> Do you have some hints that I can follow?<br>
><br>
> Thanks in advance!<br>
><br>
> Oscar<br>
<br>
</div></div><div class="HOEnZb"><div class="h5">> _______________________________________________<br>
> Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br>
> <a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
><br>
> Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
> Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
> Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a><br>
<br>
<br>
_______________________________________________<br>
Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br>
<a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a><br>
</div></div></blockquote></div><br></div>