<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40"><head><meta http-equiv=Content-Type content="text/html; charset=us-ascii"><meta name=Generator content="Microsoft Word 15 (filtered medium)"><style><!--
/* Font Definitions */
@font-face
{font-family:Wingdings;
panose-1:5 0 0 0 0 0 0 0 0 0;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:#0563C1;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:#954F72;
text-decoration:underline;}
p.MsoListParagraph, li.MsoListParagraph, div.MsoListParagraph
{mso-style-priority:34;
margin-top:0cm;
margin-right:0cm;
margin-bottom:0cm;
margin-left:36.0pt;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
/* List Definitions */
@list l0
{mso-list-id:2097748326;
mso-list-type:hybrid;
mso-list-template-ids:-2095690438 -926109156 67698691 67698693 67698689 67698691 67698693 67698689 67698691 67698693;}
@list l0:level1
{mso-level-start-at:4367;
mso-level-number-format:bullet;
mso-level-text:-;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";
mso-fareast-font-family:Calibri;}
@list l0:level2
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level3
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level4
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level5
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level6
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
@list l0:level7
{mso-level-number-format:bullet;
mso-level-text:\F0B7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Symbol;}
@list l0:level8
{mso-level-number-format:bullet;
mso-level-text:o;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:"Courier New";}
@list l0:level9
{mso-level-number-format:bullet;
mso-level-text:\F0A7;
mso-level-tab-stop:none;
mso-level-number-position:left;
text-indent:-18.0pt;
font-family:Wingdings;}
ol
{margin-bottom:0cm;}
ul
{margin-bottom:0cm;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]--></head><body lang=EN-US link="#0563C1" vlink="#954F72"><div class=WordSection1><p class=MsoNormal><span style='font-family:"Courier New"'>Hello,<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>We have a strange issue with Corosync/Pacemaker.<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>From time to time, something unexpected happens and suddenly the crm_mon output remains static.<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>When I check the cpu usage, I see that one of the cores uses 100% cpu, but cannot actually match it to either the corosync or one of the pacemaker processes.<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>In such a case, this high CPU usage is happening on all 7 nodes.<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>I have to manually go to each node, stop pacemaker, restart corosync, then start pacemeker. Stoping pacemaker and corosync does not work in most of the cases, usually a kill -9 is needed.<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>Using corosync 2.3.0, pacemaker 1.1.10 on Ubuntu trusty.<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>Using udpu as transport, two rings on Gigabit ETH, rro_mode passive.<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>Logs are usually flooded with CPG related messages, such as:<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>Mar 06 18:10:49 [1316] ctsip1 crmd: info: crm_cs_flush: Sent 0 CPG messages (1 remaining, last=8): Try again (6)<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>Mar 06 18:10:49 [1316] ctsip1 crmd: info: crm_cs_flush: Sent 0 CPG messages (1 remaining, last=8): Try again (6)<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>Mar 06 18:10:50 [1316] ctsip1 crmd: info: crm_cs_flush: Sent 0 CPG messages (1 remaining, last=8): Try again (6)<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>Mar 06 18:10:50 [1316] ctsip1 crmd: info: crm_cs_flush: Sent 0 CPG messages (1 remaining, last=8): Try again (6)<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>OR<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>Mar 06 17:46:24 [1341] ctdb1 cib: info: crm_cs_flush: Sent 0 CPG messages (1 remaining, last=10933): Try again (<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>Mar 06 17:46:24 [1341] ctdb1 cib: info: crm_cs_flush: Sent 0 CPG messages (1 remaining, last=10933): Try again (<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>Mar 06 17:46:24 [1341] ctdb1 cib: info: crm_cs_flush: Sent 0 CPG messages (1 remaining, last=10933): Try again (<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>HTOP show something like this (sorted by TIME+ descending):<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> 1 [||||||||||||||||||||||||||||||||||||||||100.0%] Tasks: 59, 4 thr; 2 running<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> 2 [| 0.7%] Load average: 1.00 0.99 1.02<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> Mem[|||||||||||||||||||||||||||||||| 165/994MB] Uptime: 1 day, 10:22:03<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> Swp[ 0/509MB]<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> PID USER PRI NI VIRT RES SHR S CPU% MEM% TIME+ Command<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> 921 root 20 0 188M 49220 33856 R 0.0 4.8 3h33:58 /usr/sbin/corosync<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> 1277 snmp 20 0 45708 4248 1472 S 0.0 0.4 1:33.07 /usr/sbin/snmpd -Lsd -Lf /dev/null -u snmp -g snm<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> 1311 hacluster 20 0 109M 16160 9640 S 0.0 1.6 1:12.71 /usr/lib/pacemaker/cib<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> 1312 root 20 0 104M 7484 3780 S 0.0 0.7 0:38.06 /usr/lib/pacemaker/stonithd<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> 1611 root -2 0 4408 2356 2000 S 0.0 0.2 0:24.15 /usr/sbin/watchdog<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> 1316 hacluster 20 0 122M 9756 5924 S 0.0 1.0 0:22.62 /usr/lib/pacemaker/crmd<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> 1313 root 20 0 81784 3800 2876 S 0.0 0.4 0:18.64 /usr/lib/pacemaker/lrmd<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> 1314 hacluster 20 0 96616 4132 2604 S 0.0 0.4 0:16.01 /usr/lib/pacemaker/attrd<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> 1309 root 20 0 104M 4804 2580 S 0.0 0.5 0:15.56 pacemakerd<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> 1250 root 20 0 33000 1192 928 S 0.0 0.1 0:13.59 ha_logd: read process<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> 1315 hacluster 20 0 73892 2652 1952 S 0.0 0.3 0:13.25 /usr/lib/pacemaker/pengine<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> 1252 root 20 0 33000 712 456 S 0.0 0.1 0:13.03 ha_logd: write process<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> 1835 ntp 20 0 27216 1980 1408 S 0.0 0.2 0:11.80 /usr/sbin/ntpd -p /var/run/ntpd.pid -g -u 105:112<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> 899 root 20 0 19168 700 488 S 0.0 0.1 0:09.75 /usr/sbin/irqbalance<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> 1642 root 20 0 30696 1556 912 S 0.0 0.2 0:06.49 /usr/bin/monit -c /etc/monit/monitrc<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> 4374 kamailio 20 0 291M 7272 2188 S 0.0 0.7 0:02.77 /usr/local/sbin/kamailio -f /etc/kamailio/kamaili<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> 3079 root 0 -20 16864 4592 3508 S 0.0 0.5 0:01.51 /usr/bin/atop -a -w /var/log/atop/atop_20140306 6<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> 445 syslog 20 0 249M 6276 976 S 0.0 0.6 0:01.16 rsyslogd<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> 4373 kamailio 20 0 291M 7492 2396 S 0.0 0.7 0:01.03 /usr/local/sbin/kamailio -f /etc/kamailio/kamaili<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> 1 root 20 0 33376 2632 1404 S 0.0 0.3 0:00.63 /sbin/init<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> 453 syslog 20 0 249M 6276 976 S 0.0 0.6 0:00.63 rsyslogd<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> 451 syslog 20 0 249M 6276 976 S 0.0 0.6 0:00.53 rsyslogd<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> 4379 kamailio 20 0 291M 6224 1132 S 0.0 0.6 0:00.38 /usr/local/sbin/kamailio -f /etc/kamailio/kamaili<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> 4380 kamailio 20 0 291M 8516 3084 S 0.0 0.8 0:00.38 /usr/local/sbin/kamailio -f /etc/kamailio/kamaili<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> 4381 kamailio 20 0 291M 8252 2828 S 0.0 0.8 0:00.37 /usr/local/sbin/kamailio -f /etc/kamailio/kamaili<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>23315 root 20 0 24872 2476 1412 R 0.7 0.2 0:00.37 htop<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'> 4367 kamailio 20 0 291M 10000 4864 S 0.0 1.0 0:00.36 /usr/local/sbin/kamailio -f /etc/kamailio/kamaili<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>My questions:<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l0 level1 lfo1'><![if !supportLists]><span style='font-family:"Courier New"'><span style='mso-list:Ignore'>-<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='font-family:"Courier New"'>Is this a cororync or pacameker issue?<o:p></o:p></span></p><p class=MsoListParagraph style='text-indent:-18.0pt;mso-list:l0 level1 lfo1'><![if !supportLists]><span style='font-family:"Courier New"'><span style='mso-list:Ignore'>-<span style='font:7.0pt "Times New Roman"'> </span></span></span><![endif]><span style='font-family:"Courier New"'>What are the CPG messages? Is it possible that we have a firewall issue?<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>Any hints would be great!<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'><o:p> </o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>Thanks,<o:p></o:p></span></p><p class=MsoNormal><span style='font-family:"Courier New"'>Attila<o:p></o:p></span></p></div></body></html>