<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri",sans-serif;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
span.EmailStyle19
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="EN-US" link="#0563C1" vlink="#954F72">
<div class="WordSection1">
<p class="MsoNormal" style="margin-bottom:12.0pt">Hi Team,<o:p></o:p></p>
<p class="MsoNormal">Problem Statement: <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">pcs resource monitor got timed out after 120000ms and tried to recover resource(application) by stopping and starting first occurrence itself. Due to this restart resource which caused traffic impact momently in their environment. And,
we suspect reason for timed out is at <b>time of monitor function execution, process checks command got hanged & delayed due to system resource unavailability.
</b><o:p></o:p></p>
<p class="MsoNormal"><br>
We are not able to confirm exactly same thing is happened or not. we got only info like HAZELCAST_occ12_monitor_10000:59159 - terminated with signal 9.
<o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><b><u>Error message we seen in customer node: </u></b><o:p></o:p></p>
<p class="MsoNormal"><b> </b><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">zrangun@seliius25303[16:40][var/log/pacemaker]$ grep -ia HAZELCAST pacemaker.log</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">Nov 15 22:42:33 occ12 pacemaker-execd [2796] (child_timeout_callback) warning: HAZELCAST_occ12_monitor_10000 process (PID 57827) timed out</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">Nov 15 22:42:33 occ12 pacemaker-execd [2796] (operation_finished) warning: HAZELCAST_occ12_monitor_10000:57827 - timed out after 120000ms</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">Nov 15 22:42:47 occ12 pacemaker-execd [2796] (cancel_recurring_action) info: Cancelling ocf operation HAZELCAST_occ12_monitor_10000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">Nov 15 22:42:47 occ12 pacemaker-execd [2796] (services_action_cancel) info: Terminating in-flight op HAZELCAST_occ12_monitor_10000 (pid 59159) early because it was cancelled</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">Nov 15 22:42:47 occ12 pacemaker-execd [2796] (operation_finished) info: HAZELCAST_occ12_monitor_10000:59159 - terminated with signal 9</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">Nov 15 22:42:47 occ12 pacemaker-execd [2796] (cancel_recurring_action) info: Cancelling ocf operation HAZELCAST_occ12_monitor_10000</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">Nov 15 22:42:47 occ12 pacemaker-execd [2796] (log_execute) info: executing - rsc:HAZELCAST_occ12 action:stop call_id:391</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">Nov 15 22:43:41 occ12 pacemaker-execd [2796] (log_finished) info: finished - rsc:HAZELCAST_occ12 action:stop call_id:391 pid:59476 exit-code:0 exec-time:53623ms queue-time:0ms</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">Nov 15 22:43:42 occ12 pacemaker-execd [2796] (log_execute) info: executing - rsc:HAZELCAST_occ12 action:start call_id:392</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">Nov 15 22:43:46 occ12 pacemaker-execd [2796] (operation_finished) notice: HAZELCAST_occ12_start_0:61681:stderr [ touch: cannot touch '/usr/var/run/resource-agents/hazelcast-HAZELCAST_occ12.state':
No such file or directory ]</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">Nov 15 22:43:46 occ12 pacemaker-execd [2796] (log_finished) info: finished - rsc:HAZELCAST_occ12 action:start call_id:392 pid:61681 exit-code:1 exec-time:3525ms queue-time:0ms</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">Nov 15 22:43:46 occ12 pacemaker-execd [2796] (log_execute) info: executing - rsc:HAZELCAST_occ12 action:stop call_id:393</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">Nov 15 22:43:47 occ12 pacemaker-execd [2796] (log_finished) info: finished - rsc:HAZELCAST_occ12 action:stop call_id:393 pid:64134 exit-code:0 exec-time:695ms queue-time:0ms</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">Nov 15 22:43:50 occ12 pacemaker-execd [2796] (log_execute) info: executing - rsc:HAZELCAST_occ12 action:start call_id:394</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">Nov 15 22:45:15 occ12 pacemaker-execd [2796] (log_finished) info: finished - rsc:HAZELCAST_occ12 action:start call_id:394 pid:64410 exit-code:0 exec-time:85211ms queue-time:1ms</span><o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal">We have shared resource configuration setup and dummy_monitor function from local node FYR. <o:p></o:p></p>
<p class="MsoNormal"> <br>
<b><u>Resource setup:</u></b> <o:p></o:p></p>
<p class="MsoNormal"> <o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">[root@vmc0137 ~]# pcs resource show HAZELCAST_vmc0137</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">Resource: HAZELCAST_vmc0137 (class=ocf provider=provider type=HazelCast_RA)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> Meta Attrs: failure-timeout=120s migration-threshold=5 priority=50</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> Operations: migrate_from interval=0s timeout=20 (HAZELCAST_vmc0137-migrate_from-interval-0s)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> migrate_to interval=0s timeout=20 (HAZELCAST_vmc0137-migrate_to-interval-0s)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> monitor interval=10s on-fail=restart timeout=120s (HAZELCAST_vmc0137-monitor-interval-10s)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> reload interval=0s timeout=20 (HAZELCAST_vmc0137-reload-interval-0s)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> start interval=0s on-fail=restart timeout=120s (HAZELCAST_vmc0137-start-interval-0s)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> stop interval=0s timeout=120s (HAZELCAST_vmc0137-stop-interval-0s)</span><br>
<br>
<o:p></o:p></p>
<p class="MsoNormal"><b><u>Monitor function input:<o:p></o:p></u></b></p>
<p class="MsoNormal"><br>
<span style="font-family:"Courier New"">dummy_monitor() {</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> # Monitor _MUST!_ differentiate correctly between running</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> # (SUCCESS), failed (ERROR) or _cleanly_ stopped (NOT RUNNING).</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> # That is THREE states, not just yes/no</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> #sleep ${OCF_RESKEY_op_sleep}</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">output=$(su - ogw -c "/opt/occ/$PRODUCT_NUMBER/bin/RCControl status SERVER")</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">number=$(grep "Running as PID" <<< "$output" | wc -l)</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">PID=`pgrep -f "<a href="https://protect2.fireeye.com/v1/url?k=c0fcaeb7-9f679786-c0fcee2c-86e2237f51fb-f9fa99fdfd024a85&q=1&e=6f1aa0d1-5fc8-4c2c-a73e-983c5f3bfab6&u=http%3A%2F%2Fdcmg.component.name%2F">Dcmg.component.name</a>=SERVER"`</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> if [ $number == 1 ] || [ -n "$PID" ] ; then</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> if [ ! -f /opt/occ/var/pid/SERVER.`hostname`.pid ]; then</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> NOW=$(date +"%b %d %H:%M:%S")</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> echo "$PID" > /opt/occ/var/pid/SERVER.`hostname`.pid</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> chown ogw:med /opt/occ/var/pid/SERVER.`hostname`.pid</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> echo "$NOW Monitor found SERVER pid file not exist and going to create it" >>/var/log/cluster/corosync.log</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> fi</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> return $OCF_SUCCESS</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> fi</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> NOW=$(date +"%b %d %H:%M:%S")</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> echo "$NOW Monitor found SERVER component is not running and going for the restart" >>/var/log/cluster/corosync.log</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> return $OCF_NOT_RUNNING</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New"">}</span><o:p></o:p></p>
<p class="MsoNormal"><span style="font-family:"Courier New""> </span><o:p></o:p></p>
<p class="MsoNormal">So, we need to support and answer to avoid above scenarios in future , kindly let us know if any additional logs required.
<br>
<br>
1) Is there any options available to set fail-retry conditions for resource monitor? So, if two times resource monitor fails continuously, then only it should go for recover. Other wise pacemaker should initiate recover for the resource. Please
confirm. <br>
2) Is there any other better option available to avoid timed out issues in first occurrence itself.?<br>
3) we thought of increasing resource timeout value to 300s and adding retry logic in dummy_monitor function itself on the RA files with timeout command. So, In this case pgrep command will be killed if couldn’t get response within 30s and retry
with next loop. Will this solution help for us..?<br>
<br>
Thanks & Regards,<o:p></o:p></p>
<p class="MsoNormal">S Sathish S<o:p></o:p></p>
</div>
</body>
</html>