Hi Andrew,<br><br>I changed cluster option to batch-limit=3,I re-tried it.<br>However, similar time-out occurs.<br><br>I measured processing just before the time-out(120s) in systemtap.<br>The following only the function long time.<br>
-----<br>probe start! ---------------------------------<br> cib_process_request [call-count:179][117,540,173,155 nsec]<br> cib_process_command [call:179] [116,471,047,275 nsec]<br>cib_process_command call function ---<br>
cib_config_changed [call:179] [101,169,909,572 nsec]<br>cib_config_changed call function ---<br> calculate_xml_digest [call:179] [ 68,820,560,745 nsec]<br> create_xml_node [call:3012263] [ 19,855,469,976 nsec]$B"((B<br>
xpath_search [call:179] [ 145,030,232 nsec]<br> diff_xml_object [call:179] [ 32,677,359,476 nsec]$B"((B<br>calculate_xml_digest call function ---<br> sorted_xml [call:1505799] [ 52,512,465,838 nsec]$B"((B<br>
copy_xml [call:179] [ 3,692,232,073 nsec]<br> dump_xml [call:536] [ 6,177,606,232 nsec]<br>-----<br>Is there the method to make these processing early?<br><br><br><div class="gmail_quote">
2010/6/14 <span dir="ltr"><<a href="mailto:renayama19661014@ybb.ne.jp">renayama19661014@ybb.ne.jp</a>></span><br><blockquote class="gmail_quote" style="margin: 0pt 0pt 0pt 0.8ex; border-left: 1px solid rgb(204, 204, 204); padding-left: 1ex;">
Hi Andrew,<br>
<br>
Thank you for comment.<br>
<div class="im"><br>
> More likely of the underlying messaging infrastructure, but I'll take a look.<br>
> Perhaps the default cib operation timeouts are too low for larger clusters.<br>
><br>
> ><br>
> > The log attached it to next Bugzilla.<br>
</div>> > �* <a href="http://developerbugs.linux-foundation.org/show_bug.cgi?id=2443" target="_blank">http://developerbugs.linux-foundation.org/show_bug.cgi?id=2443</a><br>
<div class="im">><br>
> Ok, I'll follow up there.<br>
<br>
</div>If it is necessary for us to work for the solution of the problem, please order it.<br>
<br>
Best Regards,<br>
Hideo Yamauchi.<br>
<div><div></div><div class="h5"><br>
--- Andrew Beekhof <<a href="mailto:andrew@beekhof.net">andrew@beekhof.net</a>> wrote:<br>
<br>
> On Mon, Jun 14, 2010 at 4:46 AM, <<a href="mailto:renayama19661014@ybb.ne.jp">renayama19661014@ybb.ne.jp</a>> wrote:<br>
> > We tested 16 node constitution (15+1).<br>
> ><br>
> > We carried out the next procedure.<br>
> ><br>
> > Step1) Start 16 nodes.<br>
> > Step2) Send cib after a DC node was decided.<br>
> ><br>
> > An error occurs by the update of the attribute of pingd after Probe processing was over.<br>
> ><br>
> ><br>
><br>
----------------------------------------------------------------------------------------------------------------------------------------<br>
> > Jun 14 10:58:03 hb0102 pingd: [2465]: info: ping_read: Retrying...<br>
> > Jun 14 10:58:13 hb0102 attrd: [2155]: WARN: attrd_cib_callback: Update 337 for<br>
> default_ping_set=1600<br>
> > failed: Remote node did not respond<br>
> > Jun 14 10:58:13 hb0102 attrd: [2155]: WARN: attrd_cib_callback: Update 340 for<br>
> default_ping_set=1600<br>
> > failed: Remote node did not respond<br>
> > Jun 14 10:58:13 hb0102 attrd: [2155]: WARN: attrd_cib_callback: Update 343 for<br>
> default_ping_set=1600<br>
> > failed: Remote node did not respond<br>
> > Jun 14 10:58:13 hb0102 attrd: [2155]: WARN: attrd_cib_callback: Update 346 for<br>
> default_ping_set=1600<br>
> > failed: Remote node did not respond<br>
> > Jun 14 10:58:13 hb0102 attrd: [2155]: WARN: attrd_cib_callback: Update 349 for<br>
> default_ping_set=1600<br>
> > failed: Remote node did not respond<br>
> ><br>
><br>
----------------------------------------------------------------------------------------------------------------------------------------<br>
> ><br>
> > In the middle of this error, I carried out a cibadmin(-Q optin) command, but time-out<br>
> occurred.<br>
> > In addition, cib of the DC node seemed to move by the top command very busily.<br>
> ><br>
> ><br>
> > In addition, a communication error with cib occurs in the DC node, and crmd reboots.<br>
> ><br>
> ><br>
><br>
----------------------------------------------------------------------------------------------------------------------------------------<br>
> > Jun 14 10:58:09 hb0101 attrd: [2278]: WARN: xmlfromIPC: No message received in the required<br>
> interval<br>
> > (120s)<br>
> > Jun 14 10:58:09 hb0101 attrd: [2278]: info: attrd_perform_update: Sent update -41:<br>
> > default_ping_set=1600<br>
> > (snip)<br>
> > Jun 14 10:59:07 hb0101 crmd: [2280]: info: do_exit: [crmd] stopped (2)<br>
</div></div>> > Jun 14 10:59:07 hb0101 corosync[2269]: � [pcmk �] plugin.c:858 info: pcmk_ipc_exit:<br>
<div class="im">Client<br>
> crmd<br>
> > (conn=0x106a2bf0, async-conn=0x106a2bf0) left<br>
</div>> > Jun 14 10:59:08 hb0101 corosync[2269]: � [pcmk �] plugin.c:481 ERROR:<br>
<div class="im">pcmk_wait_dispatch:<br>
> Child<br>
> > process crmd exited (pid=2280, rc=2)<br>
</div>> > Jun 14 10:59:08 hb0101 corosync[2269]: � [pcmk �] plugin.c:498 notice:<br>
<div class="im">pcmk_wait_dispatch:<br>
> Respawning<br>
> > failed child process: crmd<br>
</div>> > Jun 14 10:59:08 hb0101 corosync[2269]: � [pcmk �] utils.c:131 info: spawn_child:<br>
<div class="im">Forked child<br>
> 2680 for<br>
> > process crmd<br>
> > Jun 14 10:59:08 hb0101 crmd: [2680]: info: Invoked: /usr/lib64/heartbeat/crmd<br>
> > Jun 14 10:59:08 hb0101 crmd: [2680]: info: main: CRM Hg Version:<br>
> > 9f04fa88cfd3da553e977cc79983d1c494c8b502<br>
> > Jun 14 10:59:08 hb0101 crmd: [2680]: info: crmd_init: Starting crmd<br>
> > Jun 14 10:59:08 hb0101 crmd: [2680]: info: G_main_add_SignalHandler: Added signal handler for<br>
> signal<br>
> > 17<br>
> ><br>
><br>
----------------------------------------------------------------------------------------------------------------------------------------<br>
> ><br>
> > There seems to be a problem in cib of the DC node somehow or other.<br>
> > We hope that an attribute change is completed in 16 nodes definitely.<br>
</div>> > �* Is this phenomenon a limit of the current cib process?<br>
<div class="im">><br>
> More likely of the underlying messaging infrastructure, but I'll take a look.<br>
> Perhaps the default cib operation timeouts are too low for larger clusters.<br>
><br>
> ><br>
> > The log attached it to next Bugzilla.<br>
</div>> > �* <a href="http://developerbugs.linux-foundation.org/show_bug.cgi?id=2443" target="_blank">http://developerbugs.linux-foundation.org/show_bug.cgi?id=2443</a><br>
<div><div></div><div class="h5">><br>
> Ok, I'll follow up there.<br>
><br>
> ><br>
> > Best Regards,<br>
> > Hideo Yamauchi.<br>
> ><br>
> ><br>
> ><br>
> > _______________________________________________<br>
> > Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br>
> > <a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
> ><br>
> > Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
> > Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
> > Bugs: <a href="http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker" target="_blank">http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker</a><br>
> ><br>
><br>
> _______________________________________________<br>
> Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br>
> <a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
><br>
> Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
> Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
> Bugs: <a href="http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker" target="_blank">http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker</a><br>
><br>
<br>
<br>
_______________________________________________<br>
Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br>
<a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker" target="_blank">http://developerbugs.linux-foundation.org/enter_bug.cgi?product=Pacemaker</a><br>
</div></div></blockquote></div><br>