<html><head></head><body><div style="color:#000; background-color:#fff; font-family:HelveticaNeue, Helvetica Neue, Helvetica, Arial, Lucida Grande, sans-serif;font-size:12px"><div id="yui_3_16_0_ym19_1_1473677093016_5412">Hi all,</div><div id="yui_3_16_0_ym19_1_1473677093016_5413"><br></div><div id="yui_3_16_0_ym19_1_1473677093016_5414">I am having a problem with one of our pacemaker clusters that is running in an active-active configuration.</div><div id="yui_3_16_0_ym19_1_1473677093016_5415"><br></div><div id="yui_3_16_0_ym19_1_1473677093016_5416">Sometimes the Website monitor will timeout, triggering and apache restart that fails. That will increase the fail-count to INFINITY for the Website resource and make in un-managed. I have tried the following changes:</div><div id="yui_3_16_0_ym19_1_1473677093016_5417"><br></div><div id="yui_3_16_0_ym19_1_1473677093016_5418" dir="ltr">pcs property set start-failure-is-fatal=false<br></div><div id="yui_3_16_0_ym19_1_1473677093016_5418" dir="ltr"><br></div><div id="yui_3_16_0_ym19_1_1473677093016_5418" dir="ltr">increasing the stop timeout monitor on the Website resource:</div><div id="yui_3_16_0_ym19_1_1473677093016_5418" dir="ltr">pcs resource op add Website stop interval=0s timeout=60s<br></div><div id="yui_3_16_0_ym19_1_1473677093016_5418" dir="ltr"><br></div><div id="yui_3_16_0_ym19_1_1473677093016_5418" dir="ltr">Here is the resource configuration:</div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5585"> Resource: Website (class=ocf provider=heartbeat type=apache)</div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5586"> Attributes: configfile=/etc/httpd/conf/httpd.conf statusurl=http://localhost/server-status </div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5587"> Operations: start on-fail=restart interval=0s timeout=60s (Website-name-start-interval-0s-on-fail-restart-timeout-60s)</div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5588"> monitor on-fail=restart interval=1min timeout=40s (Website-name-monitor-interval-1min-on-fail-restart-timeout-40s)</div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5589"> stop interval=0s timeout=60s (Website-name-stop-interval-0s-timeout-60s)</div><div id="yui_3_16_0_ym19_1_1473677093016_5418" dir="ltr"><br></div><div id="yui_3_16_0_ym19_1_1473677093016_5418" dir="ltr">Here is what I see in the logs when it fails:<br></div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5874">Sep 10 17:34:25 pcs-wwwclu01-02 lrmd[2268]: warning: child_timeout_callback: Website_monitor_60000 process (PID 10352) timed out</div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5875">Sep 10 17:34:25 pcs-wwwclu01-02 lrmd[2268]: warning: operation_finished: Website_monitor_60000:10352 - timed out after 40000ms</div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5876">Sep 10 17:34:25 pcs-wwwclu01-02 crmd[2271]: error: process_lrm_event: LRM operation Website_monitor_60000 (32) Timed Out (timeout=40000ms)</div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5877">Sep 10 17:34:25 pcs-wwwclu01-02 crmd[2271]: warning: update_failcount: Updating failcount for Website on pcs-wwwclu01-02 after failed monitor: rc=1 (update=value++, time=1473543265)</div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5878">Sep 10 17:34:25 pcs-wwwclu01-02 attrd[2269]: notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-Website (1)</div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5879">Sep 10 17:34:25 pcs-wwwclu01-02 attrd[2269]: notice: attrd_perform_update: Sent update 27: fail-count-Website=1</div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5880">Sep 10 17:34:25 pcs-wwwclu01-02 attrd[2269]: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-Website (1473543265)</div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5881">Sep 10 17:34:25 pcs-wwwclu01-02 pengine[2270]: warning: unpack_rsc_op: Processing failed op monitor for Website:0 on pcs-wwwclu01-02: unknown error (1)</div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5882">Sep 10 17:34:25 pcs-wwwclu01-02 pengine[2270]: notice: LogActions: Recover Website:0#011(Started pcs-wwwclu01-02)</div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5883">Sep 10 17:34:25 pcs-wwwclu01-02 attrd[2269]: notice: attrd_perform_update: Sent update 30: last-failure-Website=1473543265</div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5884">Sep 10 17:34:25 pcs-wwwclu01-02 pengine[2270]: warning: unpack_rsc_op: Processing failed op monitor for Website:0 on pcs-wwwclu01-02: unknown error (1)</div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5885">Sep 10 17:34:25 pcs-wwwclu01-02 pengine[2270]: notice: LogActions: Recover Website:0#011(Started pcs-wwwclu01-02)</div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5886">Sep 10 17:34:25 pcs-wwwclu01-02 pengine[2270]: warning: unpack_rsc_op: Processing failed op monitor for Website:0 on pcs-wwwclu01-02: unknown error (1)</div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5887">Sep 10 17:34:25 pcs-wwwclu01-02 pengine[2270]: notice: LogActions: Recover Website:0#011(Started pcs-wwwclu01-02)</div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5888">Sep 10 17:34:25 pcs-wwwclu01-02 crmd[2271]: notice: te_rsc_command: Initiating action 2: stop Website_stop_0 on pcs-wwwclu01-02 (local)</div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5889">Sep 10 17:34:25 pcs-wwwclu01-02 apache(Website)[10443]: INFO: Attempting graceful stop of apache PID 3561</div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5890">Sep 10 17:34:55 pcs-wwwclu01-02 apache(Website)[10443]: INFO: Killing apache PID 3561</div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5891">Sep 10 17:35:04 pcs-wwwclu01-02 apache(Website)[10443]: INFO: apache still running (3561). Killing pid failed.</div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5892">Sep 10 17:35:04 pcs-wwwclu01-02 apache(Website)[10443]: INFO: apache children were signalled (SIGTERM)</div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5893">Sep 10 17:35:06 pcs-wwwclu01-02 apache(Website)[10443]: INFO: apache children were signalled (SIGHUP)</div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5894">Sep 10 17:35:07 pcs-wwwclu01-02 crmd[2271]: notice: process_lrm_event: LRM operation Website_stop_0 (call=34, rc=1, cib-update=3097, confirmed=true) unknown error</div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5895">Sep 10 17:35:07 pcs-wwwclu01-02 crmd[2271]: warning: status_from_rc: Action 2 (Website_stop_0) on pcs-wwwclu01-02 failed (target: 0 vs. rc: 1): Error</div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5896">Sep 10 17:35:07 pcs-wwwclu01-02 crmd[2271]: warning: update_failcount: Updating failcount for Website on pcs-wwwclu01-02 after failed stop: rc=1 (update=INFINITY, time=1473543307)</div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5897">Sep 10 17:35:07 pcs-wwwclu01-02 attrd[2269]: notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-Website (INFINITY)</div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5898">Sep 10 17:35:07 pcs-wwwclu01-02 attrd[2269]: notice: attrd_perform_update: Sent update 32: fail-count-Website=INFINITY</div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5899">Sep 10 17:35:07 pcs-wwwclu01-02 attrd[2269]: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-Website (1473543307)</div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5900">Sep 10 17:35:07 pcs-wwwclu01-02 attrd[2269]: notice: attrd_perform_update: Sent update 34: last-failure-Website=1473543307</div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5901">Sep 10 17:35:07 pcs-wwwclu01-02 attrd[2269]: notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-Website (INFINITY)</div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5902">Sep 10 17:35:07 pcs-wwwclu01-02 pengine[2270]: warning: unpack_rsc_op: Processing failed op stop for Website:0 on pcs-wwwclu01-02: unknown error (1)</div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5903">Sep 10 17:35:07 pcs-wwwclu01-02 attrd[2269]: notice: attrd_perform_update: Sent update 36: fail-count-Website=INFINITY</div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5904">Sep 10 17:35:07 pcs-wwwclu01-02 attrd[2269]: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-Website (1473543307)</div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5905">Sep 10 17:35:07 pcs-wwwclu01-02 attrd[2269]: notice: attrd_perform_update: Sent update 38: last-failure-Website=1473543307</div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5906">Sep 10 17:35:07 pcs-wwwclu01-02 pengine[2270]: warning: unpack_rsc_op: Processing failed op stop for Website:0 on pcs-wwwclu01-02: unknown error (1)</div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5907">Sep 10 17:35:07 pcs-wwwclu01-02 pengine[2270]: warning: common_apply_stickiness: Forcing Website-clone away from pcs-wwwclu01-02 after 1000000 failures (max=1000000)</div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5908">Sep 10 17:35:07 pcs-wwwclu01-02 pengine[2270]: warning: unpack_rsc_op: Processing failed op stop for Website:0 on pcs-wwwclu01-02: unknown error (1)</div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5909">Sep 10 17:35:07 pcs-wwwclu01-02 pengine[2270]: warning: common_apply_stickiness: Forcing Website-clone away from pcs-wwwclu01-02 after 1000000 failures (max=1000000)</div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5910"><br></div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5775">I dont see that pacemaker is waiting for 60 seconds for the apache to stop.</div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5775"><br></div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5775">Has anyone encountered something like this before? Or am I missing something in the configuration?</div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5775"><br></div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5775">Thank you,</div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5775">Alex</div><div dir="ltr" id="yui_3_16_0_ym19_1_1473677093016_5775"><br></div><div id="yui_3_16_0_ym19_1_1473677093016_5418" dir="ltr"><br></div></div></body></html>