<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=utf-8">
<meta name="Generator" content="Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:宋体;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:等线;
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"\@等线";
panose-1:2 1 6 0 3 1 1 1 1 1;}
@font-face
{font-family:"\@宋体";
panose-1:2 1 6 0 3 1 1 1 1 1;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
text-align:justify;
text-justify:inter-ideograph;
font-size:10.5pt;
font-family:等线;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoPlainText, li.MsoPlainText, div.MsoPlainText
{mso-style-priority:99;
mso-style-link:"纯文本 字符";
margin:0cm;
margin-bottom:.0001pt;
font-size:10.5pt;
font-family:"Calibri",sans-serif;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin:0cm;
margin-bottom:.0001pt;
text-align:justify;
text-justify:inter-ideograph;
text-indent:21.0pt;
font-size:10.5pt;
font-family:等线;}
p.msonormal0, li.msonormal0, div.msonormal0
{mso-style-name:msonormal;
mso-margin-top-alt:auto;
margin-right:0cm;
mso-margin-bottom-alt:auto;
margin-left:0cm;
font-size:12.0pt;
font-family:宋体;}
span.EmailStyle18
{mso-style-type:personal-compose;
font-family:等线;
color:windowtext;}
span.a
{mso-style-name:"纯文本 字符";
mso-style-priority:99;
mso-style-link:纯文本;
font-family:"Calibri",sans-serif;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
font-family:等线;}
/* Page Definitions */
@page WordSection1
{size:612.0pt 792.0pt;
margin:72.0pt 90.0pt 72.0pt 90.0pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:827019543;
mso-list-type:hybrid;
mso-list-template-ids:1347450746 -1753185294 67698713 67698715 67698703 67698713 67698715 67698703 67698713 67698715;}
@list l0:level1
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:18.0pt;
text-indent:-18.0pt;}
@list l0:level2
{mso-level-number-format:alpha-lower;
mso-level-text:"%2\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:42.0pt;
text-indent:-21.0pt;}
@list l0:level3
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:63.0pt;
text-indent:-21.0pt;}
@list l0:level4
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:84.0pt;
text-indent:-21.0pt;}
@list l0:level5
{mso-level-number-format:alpha-lower;
mso-level-text:"%5\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:105.0pt;
text-indent:-21.0pt;}
@list l0:level6
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:126.0pt;
text-indent:-21.0pt;}
@list l0:level7
{mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:147.0pt;
text-indent:-21.0pt;}
@list l0:level8
{mso-level-number-format:alpha-lower;
mso-level-text:"%8\)";
mso-level-tab-stop:none;
mso-level-number-position:left;
margin-left:168.0pt;
text-indent:-21.0pt;}
@list l0:level9
{mso-level-number-format:roman-lower;
mso-level-tab-stop:none;
mso-level-number-position:right;
margin-left:189.0pt;
text-indent:-21.0pt;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="ZH-CN" link="#0563C1" vlink="#954F72" style="text-justify-trim:punctuation">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US">Hello,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">This issue only appears when we run performance test and the CPU is high. The cluster and log is as below. The Pacemaker will restart the Slave Side
</span><span lang="EN-US">pgsql-ha resource about every two minutes.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Take the following scenario for example:</span>（<span lang="EN-US">when the pgsqlms RA is called, we print the log “<span style="color:#92D050">execute the command start (command)</span>”. When the command is returned,
we print the log “<span style="color:#92D050">execute the command stop (Command) (result)</span>”</span>）<span lang="EN-US"><o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:-18.0pt;mso-list:l0 level1 lfo1">
<![if !supportLists]><span lang="EN-US"><span style="mso-list:Ignore">1.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span lang="EN-US">We could see that pacemaker call “pgsqlms monitor” about every 15 seconds. And it return $OCF_SUCCESS<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:-18.0pt;mso-list:l0 level1 lfo1">
<![if !supportLists]><span lang="EN-US"><span style="mso-list:Ignore">2.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span lang="EN-US">In calls monitor command again at 13:56:16, and then it reports timeout error error 13:56:18. It is only 2 seconds but it reports “timeout=10000ms”<o:p></o:p></span></p>
<p class="MsoListParagraph" style="margin-left:18.0pt;text-indent:-18.0pt;mso-list:l0 level1 lfo1">
<![if !supportLists]><span lang="EN-US"><span style="mso-list:Ignore">3.<span style="font:7.0pt "Times New Roman"">
</span></span></span><![endif]><span lang="EN-US">In other logs, sometimes after 15 minutes, there is no “</span><span lang="EN-US" style="color:#00B050">execute the command start monitor</span><span lang="EN-US">” printed and it reports timeout error directly.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Could you please tell how to debug or resolve such issue?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US">The log:<o:p></o:p></span></b></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#92D050">Jan 10 13:55:35 sds2 pgsqlms(pgsqld)[5240]: INFO: execute the command start monitor<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#92D050">Jan 10 13:55:35 sds2 pgsqlms(pgsqld)[5240]: INFO: _confirm_role start<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#92D050">Jan 10 13:55:35 sds2 pgsqlms(pgsqld)[5240]: INFO: _confirm_role stop 0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#92D050">Jan 10 13:55:35 sds2 pgsqlms(pgsqld)[5240]: INFO: execute the command stop monitor 0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#00B050">Jan 10 13:55:52 sds2 pgsqlms(pgsqld)[5477]: INFO: execute the command start monitor<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#00B050">Jan 10 13:55:52 sds2 pgsqlms(pgsqld)[5477]: INFO: _confirm_role start<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#00B050">Jan 10 13:55:52 sds2 pgsqlms(pgsqld)[5477]: INFO: _confirm_role stop 0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:#00B050">Jan 10 13:55:52 sds2 pgsqlms(pgsqld)[5477]: INFO: execute the command stop monitor 0<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Jan 10 13:56:02 sds2 crmd[26096]: notice: High CPU load detected: 426.779999<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:red">Jan 10 13:56:16 sds2 pgsqlms(pgsqld)[5606]: INFO: execute the command start monitor<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US" style="color:red">Jan 10 13:56:18 sds2 lrmd[26093]: warning: pgsqld_monitor_16000 process (PID 5606) timed out<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Jan 10 13:56:18 sds2 lrmd[26093]: warning: pgsqld_monitor_16000:5606 - timed out after 10000ms<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Jan 10 13:56:18 sds2 crmd[26096]: error: Result of monitor operation for pgsqld on db2: Timed Out | call=102 key=pgsqld_monitor_16000 timeout=10000ms<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Jan 10 13:56:18 sds2 crmd[26096]: notice: db2-pgsqld_monitor_16000:102 [ /tmp:5432 - accepting connections\n ]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Jan 10 13:56:18 sds2 crmd[26096]: notice: State transition S_IDLE -> S_POLICY_ENGINE | input=I_PE_CALC cause=C_FSA_INTERNAL origin=abort_transition_graph<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Jan 10 13:56:19 sds2 pengine[26095]: warning: Processing failed op monitor for pgsqld:0 on db2: unknown error (1)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Jan 10 13:56:19 sds2 pengine[26095]: warning: Processing failed op start for pgsqld:1 on db1: unknown error (1)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Jan 10 13:56:19 sds2 pengine[26095]: warning: Forcing pgsql-ha away from db1 after 1000000 failures (max=1000000)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Jan 10 13:56:19 sds2 pengine[26095]: warning: Forcing pgsql-ha away from db1 after 1000000 failures (max=1000000)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Jan 10 13:56:19 sds2 pengine[26095]: notice: Recover pgsqld:0#011(Slave db2)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Jan 10 13:56:19 sds2 pengine[26095]: notice: Calculated transition 37, saving inputs in /var/lib/pacemaker/pengine/pe-input-1251.bz2<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US">The Cluster Configuration:<o:p></o:p></span></b></p>
<p class="MsoNormal"><span lang="EN-US">2 nodes and 13 resources configured<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Online: [ db1 db2 ]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Full list of resources:<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Clone Set: dlm-clone [dlm]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> Started: [ db1 db2 ]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Clone Set: clvmd-clone [clvmd]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> Started: [ db1 db2 ]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">ipmi_node1 (stonith:fence_ipmilan): Started db2<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">ipmi_node2 (stonith:fence_ipmilan): Started db1<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Clone Set: clusterfs-clone [clusterfs]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> Started: [ db1 db2 ]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Master/Slave Set: pgsql-ha [pgsqld]><o:p></o:p></span></p>
<p class="MsoPlainText"><span lang="EN-US"> Masters: [ db1 ] <o:p></o:p></span></p>
<p class="MsoPlainText" style="text-indent:31.5pt"><span lang="EN-US">Slaves: [ db2 ]<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Resource Group: mastergroup<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> db1-vip (ocf::heartbeat:IPaddr2): Started<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> rep-vip (ocf::heartbeat:IPaddr2): Started<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Resource Group: slavegroup<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> db2-vip (ocf::heartbeat:IPaddr2): Started<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><b><span lang="EN-US">pcs resource show pgsql-ha<o:p></o:p></span></b></p>
<p class="MsoNormal"><span lang="EN-US">Master: pgsql-ha<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> Meta Attrs: interleave=true notify=true
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> Resource: pgsqld (class=ocf provider=heartbeat type=pgsqlms)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> Attributes: bindir=/usr/local/pgsql/bin pgdata=/home/postgres/data<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> Operations: start interval=0s timeout=160s (pgsqld-start-interval-0s)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> stop interval=0s timeout=60s (pgsqld-stop-interval-0s)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> promote interval=0s timeout=130s (pgsqld-promote-interval-0s)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> demote interval=0s timeout=120s (pgsqld-demote-interval-0s)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> monitor interval=15s role=Master timeout=10s (pgsqld-monitor-interval-15s)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> monitor interval=16s role=Slave timeout=10s (pgsqld-monitor-interval-16s)<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"> notify interval=0s timeout=60s (pgsqld-notify-interval-0s)<o:p></o:p></span></p>
</div>
</body>
</html>