<div dir="ltr"><div dir="ltr"><div>Hi -</div><div><br></div><div>Yesterday, I noticed that when I am trying to execute 'crm node standby' command on one of my cluster nodes, it was failing with </div><div><br></div><div dir="ltr">"Error performing operation: Communication error on send . Return code is 70"<br></div><div dir="ltr"><br></div><div>My corosync logs had these entries during that time:</div><div dir="ltr"><br></div><div dir="ltr">

<span style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Sep 20 22:14:54 [4454] vm5c336912f1       crmd:   notice: throttle_handle_load: High CPU load detected: 1.850000</span><br style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Sep 20 22:14:57 [4449] vm5c336912f1        cib:     info: cib_process_ping:     Reporting our current digest to vmb546073338: 8fe67fcfcd20515c246c225a124a89</span><wbr style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">02 for 0.481.2 (0x2742230 0)</span><br style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Sep 20 22:15:09 [4449] vm5c336912f1        cib:     info: cib_process_request:  Forwarding cib_modify operation for section nodes to master (origin=local/crm_attribute/4)</span><br style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Sep 20 22:15:24 [4454] vm5c336912f1       crmd:   notice: throttle_handle_load: High CPU load detected: 1.640000</span><br style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Sep 20 22:15:54 [4454] vm5c336912f1       crmd:     info: throttle_handle_load: Moderate CPU load detected: 0.990000</span><br style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Sep 20 22:15:54 [4454] vm5c336912f1       crmd:     info: throttle_send_command:        New throttle mode: 0010 (was 0100)</span><br style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Sep 20 22:16:24 [4454] vm5c336912f1       crmd:     info: throttle_send_command:        New throttle mode: 0001 (was 0010)</span><br style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Sep 20 22:16:54 [4454] vm5c336912f1       crmd:     info: throttle_send_command:        New throttle mode: 0000 (was 0001)</span><br style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Sep 20 22:17:09 [4449] vm5c336912f1        cib:     info: cib_process_request:  Forwarding cib_modify operation for section nodes to master (origin=local/crm_attribute/4)</span><br style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Sep 20 22:19:10 [4449] vm5c336912f1        cib:     info: cib_process_request:  Forwarding cib_modify operation for section nodes to master (origin=local/crm_attribute/4)</span><br style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Sep 20 22:23:08 [4449] vm5c336912f1        cib:     info: cib_perform_op:       Diff: --- 0.481.2 2</span><br style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Sep 20 22:23:08 [4449] vm5c336912f1        cib:     info: cib_perform_op:       Diff: +++ 0.482.0 9bacc862b8713430c81ea91694942a</span><wbr style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">41</span><br style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial"><span style="font-size:12.8px;background-color:rgb(255,255,255);text-decoration-style:initial;text-decoration-color:initial;float:none;display:inline">Sep 20 22:23:08 [4449] vm5c336912f1        cib:     info: cib_perform_op:       +  /cib:  @epoch=482, @num_updates=0</span>

<br></div><div dir="ltr"><br></div><div dir="ltr"><br></div><div>Is the above behavior due to pacemaker thinking that cluster is highly loaded and trying to throttle the execution of commands ? What is the best way to resolve or work-around such problems. We do have high io load on our cluster - which hosts mysql database.</div><div><br></div><div>Also from the thread,</div><div dir="ltr"><div><a href="https://lists.clusterlabs.org/pipermail/users/2017-May/005702.html">https://lists.clusterlabs.org/pipermail/users/2017-May/005702.html</a><br></div><div><br></div><div>it was asked :<br></div><div>

<pre style="white-space:pre-wrap;color:rgb(0,0,0);text-decoration-style:initial;text-decoration-color:initial"><i>>There is not much detail about “load-threshold”.</i><i>
</i>><i> Please can someone share steps or any commands to modify “load-threshold”.</i></pre>Could someone advise whether this is the way to control the throttling of cluster operations and how to set this parameter ?</div><div><br></div><div>Thanks in advance,</div><div>Prasad</div></div></div></div>