Hi,<div><br></div><div>thank you for your answer.</div><div><br></div><div>Do you think that if I increase the resource timeout that's failling it will solve the problem?</div><div><br></div><div>Regards,</div><div>Pedro Sousa<br>
<br><div class="gmail_quote">On Mon, Nov 26, 2012 at 12:23 AM, Andrew Beekhof <span dir="ltr"><<a href="mailto:andrew@beekhof.net" target="_blank">andrew@beekhof.net</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="HOEnZb"><div class="h5">On Wed, Nov 21, 2012 at 2:02 AM, Pedro Sousa <<a href="mailto:pgsousa@gmail.com">pgsousa@gmail.com</a>> wrote:<br>
> Hi all,<br>
><br>
> some strange behavior is happening when I do some more intensive work on my<br>
> cluster like running a bash script or wireshark, some pacemaker resources<br>
> start to time out and fail back to the other node. I was running this<br>
> script:<br>
><br>
> # find /sharedstorage/var/log/asterisk/cdr-csv/ -type f -size 0 -exec rm -f<br>
> {} \;<br>
><br>
> to clean some unused 0-byte files on my drbd shared storage when I saw this<br>
> on my logs and some resources failling:<br>
><br>
> Nov 20 12:32:19 nd02 lrmd: [27143]: WARN: res_IPaddr2_Sip:monitor process<br>
> (PID 30312) timed out (try 1). Killing with signal SIGTERM (15).<br>
> Nov 20 12:32:19 nd02 lrmd: [27143]: WARN: res_IPaddr2_Admin:monitor process<br>
> (PID 30313) timed out (try 1). Killing with signal SIGTERM (15).<br>
> Nov 20 12:32:19 nd02 lrmd: [27143]: WARN: res_IPaddr2_Asterisk:monitor<br>
> process (PID 30314) timed out (try 1). Killing with signal SIGTERM (15).<br>
> Nov 20 12:32:20 nd02 lrmd: [27143]: WARN: operation monitor[416] on<br>
> ocf::IPaddr2::res_IPaddr2_Sip for client 27146, its parameters:<br>
> CRM_meta_name=[monitor] CRM_meta_start_delay=[0] crm_feature_set=[3.0.5]<br>
> CRM_meta_timeout=[20000] CRM_meta_interval=[10000] iflabel=[Sip]<br>
> ip=[10.100.251.30] : pid [30312] timed out<br>
> Nov 20 12:32:20 nd02 lrmd: [27143]: WARN: operation monitor[428] on<br>
> ocf::IPaddr2::res_IPaddr2_Admin for client 27146, its parameters:<br>
> CRM_meta_name=[monitor] CRM_meta_start_delay=[0] crm_feature_set=[3.0.5]<br>
> CRM_meta_timeout=[20000] CRM_meta_interval=[10000] iflabel=[Admin]<br>
> ip=[10.100.252.30] : pid [30313] timed out<br>
> Nov 20 12:32:20 nd02 lrmd: [27143]: WARN: operation monitor[442] on<br>
> ocf::IPaddr2::res_IPaddr2_Asterisk for client 27146, its parameters:<br>
> CRM_meta_name=[monitor] CRM_meta_start_delay=[0] crm_feature_set=[3.0.5]<br>
> CRM_meta_timeout=[20000] CRM_meta_interval=[10000] iflabel=[Asterisk]<br>
> ip=[10.100.251.100] : pid [30314] timed out<br>
> Nov 20 12:32:20 nd02 lrmd: [27143]: WARN: G_SIG_dispatch: Dispatch function<br>
> for SIGCHLD took too long to execute: 970 ms (> 300 ms) (GSource: 0x1ab0d60)<br>
> Nov 20 12:32:20 nd02 lrmd: [27143]: WARN: perform_ra_op: the operation<br>
> operation monitor[434] on lsb::ntpd::res_ntpd_Sip for client 27146, its<br>
> parameters: CRM_meta_name=[monitor] CRM_meta_start_delay=[15000]<br>
> crm_feature_set=[3.0.5] CRM_meta_timeout=[30000] CRM_meta_interval=[15000]<br>
> stayed in operation list for 21100 ms (longer than 10000 ms)<br>
> Nov 20 12:32:20 nd02 lrmd: [27143]: WARN: perform_ra_op: the operation<br>
> operation monitor[429] on lsb::dhcpd::res_dhcpd_Sip for client 27146, its<br>
> parameters: CRM_meta_name=[monitor] CRM_meta_start_delay=[15000]<br>
> crm_feature_set=[3.0.5] CRM_meta_timeout=[30000] CRM_meta_interval=[15000]<br>
> stayed in operation list for 19910 ms (longer than 10000 ms)<br>
><br>
> Any hint on what can be causing this? Can anybody help? Thanks.<br>
<br>
</div></div>I wouldn't have thought that one should be able to affect the other<br>
unless the find was running as a high priority task.<br>
But if it is a large filesystem then all those directory listings and<br>
file lookups would keep the kernel quite busy, long enough to delay a<br>
few socket()/bind()/recvmsg() calls I suppose.<br>
<div><div class="h5"><br>
><br>
> Here's my configuration:<br>
><br>
> 2 x node cluster Centos 6.0 64-bit 2GB RAM<br>
> pacemaker-1.1.6-3.el6.x86_64<br>
> pacemaker-libs-1.1.6-3.el6.x86_64<br>
> pacemaker-cli-1.1.6-3.el6.x86_64<br>
> pacemaker-cluster-libs-1.1.6-3.el6.x86_64<br>
> corosync-1.4.1-4.el6.x86_64<br>
> corosynclib-1.4.1-4.el6.x86_64<br>
> openaislib-1.1.1-7.el6.x86_64<br>
> openais-1.1.1-7.el6.x86_64<br>
><br>
> corosync.conf:<br>
><br>
> aisexec {<br>
> user: root<br>
> group: root<br>
> }<br>
><br>
> corosync {<br>
> user: root<br>
> group: root<br>
> }<br>
><br>
> amf {<br>
> mode: disabled<br>
> }<br>
><br>
> logging {<br>
> to_stderr: yes<br>
> debug: off<br>
> timestamp: on<br>
> to_file: no<br>
> to_syslog: yes<br>
> syslog_facility: daemon<br>
> }<br>
><br>
> totem {<br>
> version: 2<br>
> token: 3000<br>
> token_retransmits_before_loss_const: 10<br>
> join: 60<br>
> consensus: 4000<br>
> vsftype: none<br>
> max_messages: 20<br>
> clear_node_high_bit: yes<br>
> secauth: on<br>
> threads: 0<br>
> # nodeid: 1234<br>
> rrp_mode: active<br>
><br>
> interface {<br>
> ringnumber: 0<br>
> bindnetaddr: 10.0.0.2<br>
> mcastaddr: 226.94.1.1<br>
> mcastport: 4000<br>
> }<br>
><br>
> }<br>
><br>
> service {<br>
> ver: 0<br>
> name: pacemaker<br>
> use_mgmtd: yes<br>
> }<br>
><br>
> pacemaker configuration<br>
><br>
> # crm configure edit<br>
><br>
> node nd01.lab<br>
> node nd02.lab \<br>
> attributes standby="off"<br>
> primitive res_Filesystem_Sip ocf:heartbeat:Filesystem \<br>
> params device="/dev/drbd0" directory="/sharedstorage" fstype="ext4"<br>
> \<br>
> operations $id="res_Filesystem_Sip-operations" \<br>
> op start interval="0" timeout="60" \<br>
> op stop interval="0" timeout="60" \<br>
> op monitor interval="20" timeout="40" start-delay="0" \<br>
> op notify interval="0" timeout="60"<br>
> primitive res_IPaddr2_Admin ocf:heartbeat:IPaddr2 \<br>
> params ip="10.100.252.30" iflabel="Admin" \<br>
> operations $id="res_IPaddr2_Admin-operations" \<br>
> op start interval="0" timeout="20" \<br>
> op stop interval="0" timeout="20" \<br>
> op monitor interval="10" timeout="20" start-delay="0"<br>
> primitive res_IPaddr2_Asterisk ocf:heartbeat:IPaddr2 \<br>
> params ip="10.100.251.100" iflabel="Asterisk" \<br>
> operations $id="res_IPaddr2_Asterisk-operations" \<br>
> op start interval="0" timeout="20" \<br>
> op stop interval="0" timeout="20" \<br>
> op monitor interval="10" timeout="20" start-delay="0"<br>
> primitive res_IPaddr2_Sip ocf:heartbeat:IPaddr2 \<br>
> params ip="10.100.251.30" iflabel="Sip" \<br>
> operations $id="res_IPaddr2_Sip-operations" \<br>
> op start interval="0" timeout="20" \<br>
> op stop interval="0" timeout="20" \<br>
> op monitor interval="10" timeout="20" start-delay="0"<br>
> primitive res_asterisk_Asterisk lsb:asterisk \<br>
> operations $id="res_asterisk_Asterisk-operations" \<br>
> op start interval="0" timeout="15" \<br>
> op stop interval="0" timeout="15" \<br>
> op monitor interval="15" timeout="15" start-delay="15" \<br>
> meta target-role="Started"<br>
> primitive res_dhcpd_Sip lsb:dhcpd \<br>
> operations $id="res_dhcpd_Sip-operations" \<br>
> op start interval="0" timeout="15" \<br>
> op stop interval="0" timeout="15" \<br>
> op monitor interval="15" timeout="15" start-delay="15" \<br>
> meta is-managed="true" target-role="Started"<br>
> primitive res_drbd_1 ocf:linbit:drbd \<br>
> params drbd_resource="r0" \<br>
> operations $id="res_drbd_1-operations" \<br>
> op start interval="0" timeout="240" \<br>
> op promote interval="0" timeout="90" \<br>
> op demote interval="0" timeout="90" \<br>
> op stop interval="0" timeout="100" \<br>
> op monitor interval="10" timeout="20" start-delay="0" \<br>
> op notify interval="0" timeout="90"<br>
> primitive res_drbdlinks_Sip heartbeat:drbdlinks \<br>
> params 1="-c" 2="/etc/drbdlinks.conf" \<br>
> operations $id="res_drbdlinks_Sip-operations" \<br>
> op start interval="0" timeout="15" \<br>
> op stop interval="0" timeout="15" \<br>
> op monitor interval="15" timeout="15" start-delay="15" \<br>
> meta target-role="started"<br>
> primitive res_faxmodems_Sip lsb:faxmodems \<br>
> operations $id="res_faxmodems_Sip-operations" \<br>
> op start interval="0" timeout="15" \<br>
> op stop interval="0" timeout="15" \<br>
> op monitor interval="15" timeout="15" start-delay="15" \<br>
> meta is-managed="true" target-role="Started"<br>
> primitive res_httpd_Sip lsb:httpd \<br>
> operations $id="res_httpd_Sip-operations" \<br>
> op start interval="0" timeout="15" \<br>
> op stop interval="0" timeout="15" \<br>
> op monitor interval="15" timeout="15" start-delay="15" \<br>
> meta is-managed="true" target-role="Started"<br>
> primitive res_hylafax_Sip lsb:hylafax \<br>
> operations $id="res_hylafax_Sip-operations" \<br>
> op start interval="0" timeout="15" \<br>
> op stop interval="0" timeout="15" \<br>
> op monitor interval="15" timeout="15" start-delay="15" \<br>
> meta target-role="Started"<br>
> primitive res_iaxmodem_Sip lsb:iaxmodem \<br>
> operations $id="res_iaxmodem_Sip-operations" \<br>
> op start interval="0" timeout="15" \<br>
> op stop interval="0" timeout="15" \<br>
> op monitor interval="15" timeout="15" start-delay="15" \<br>
> meta is-managed="true" target-role="Started"<br>
> primitive res_kamailio_Sip lsb:kamailio \<br>
> operations $id="res_kamailio_Sip-operations" \<br>
> op start interval="0" timeout="15" \<br>
> op stop interval="0" timeout="15" \<br>
> op monitor interval="15" timeout="15" start-delay="15" \<br>
> meta target-role="Started"<br>
> primitive res_mysqld_Sip lsb:mysqld \<br>
> operations $id="res_mysqld_Sip-operations" \<br>
> op start interval="0" timeout="15" \<br>
> op stop interval="0" timeout="15" \<br>
> op monitor interval="15" timeout="15" start-delay="15"<br>
> primitive res_named_Sip lsb:named \<br>
> operations $id="res_named_Sip-operations" \<br>
> op start interval="0" timeout="15" \<br>
> op stop interval="0" timeout="15" \<br>
> op monitor interval="15" timeout="15" start-delay="15" \<br>
> meta target-role="started" is-managed="true"<br>
> primitive res_nfs_Sip lsb:nfs \<br>
> operations $id="res_nfs_Sip-operations" \<br>
> op start interval="0" timeout="15" \ op stop interval="0"<br>
> timeout="15" \<br>
> op monitor interval="15" timeout="15" start-delay="15"<br>
> primitive res_ntpd_Sip lsb:ntpd \<br>
> operations $id="res_ntpd_Sip-operations" \<br>
> op start interval="0" timeout="15" \<br>
> op stop interval="0" timeout="15" \<br>
> op monitor interval="15" timeout="15" start-delay="15"<br>
> primitive res_postfix_Sip lsb:postfix \<br>
> operations $id="res_postfix_Sip-operations" \<br>
> op start interval="0" timeout="15" \<br>
> op stop interval="0" timeout="15" \<br>
> op monitor interval="15" timeout="15" start-delay="15"<br>
> ms ms_drbd_1 res_drbd_1 \<br>
> meta clone-max="2" notify="true"<br>
> colocation col_res_Filesystem_Sip_ms_drbd_1 inf: res_Filesystem_Sip<br>
> ms_drbd_1:Master<br>
> colocation col_res_IPaddr2_Admin_res_Filesystem_Sip inf: res_IPaddr2_Admin<br>
> res_Filesystem_Sip<br>
> colocation col_res_IPaddr2_Sip_res_Filesystem_Sip inf: res_IPaddr2_Sip<br>
> res_Filesystem_Sip<br>
> colocation col_res_asterisk_Asterisk_res_IPaddr2_Asterisk inf:<br>
> res_asterisk_Asterisk res_IPaddr2_Asterisk<br>
> colocation col_res_drbdlinks_Sip_res_IPaddr2_Asterisk inf:<br>
> res_IPaddr2_Asterisk res_drbdlinks_Sip<br>
> colocation col_res_drbdlinks_Sip_res_IPaddr2_Sip inf: res_drbdlinks_Sip<br>
> res_IPaddr2_Sip<br>
> colocation col_res_drbdlinks_Sip_res_dhcpd_Sip inf: res_dhcpd_Sip<br>
> res_drbdlinks_Sip<br>
> colocation col_res_faxmodems_Sip_res_drbdlinks_Sip inf: res_faxmodems_Sip<br>
> res_drbdlinks_Sip<br>
> colocation col_res_httpd_Sip_res_drbdlinks_Sip inf: res_httpd_Sip<br>
> res_drbdlinks_Sip<br>
> colocation col_res_hylafax_Sip_res_drbdlinks_Sip inf: res_hylafax_Sip<br>
> res_drbdlinks_Sip<br>
> colocation col_res_iaxmodem_Sip_res_drbdlinks_Sip inf: res_iaxmodem_Sip<br>
> res_drbdlinks_Sip<br>
> colocation col_res_kamailio_Sip_res_IPaddr2_Asterisk inf:<br>
> res_IPaddr2_Asterisk res_kamailio_Sip<br>
> colocation col_res_kamailio_Sip_res_drbdlinks_Sip inf: res_kamailio_Sip<br>
> res_drbdlinks_Sip<br>
> colocation col_res_mysqld_Sip_res_drbdlinks_Sip inf: res_mysqld_Sip<br>
> res_drbdlinks_Sip<br>
> colocation col_res_mysqld_Sip_res_kamailio_Sip inf: res_kamailio_Sip<br>
> res_mysqld_Sip<br>
> colocation col_res_named_Sip_res_drbdlinks_Sip inf: res_named_Sip<br>
> res_drbdlinks_Sip<br>
> colocation col_res_named_Sip_res_kamailio_Sip inf: res_kamailio_Sip<br>
> res_named_Sip<br>
> colocation col_res_nfs_Sip_res_drbdlinks_Sip inf: res_nfs_Sip<br>
> res_drbdlinks_Sip<br>
> colocation col_res_ntpd_Sip_res_drbdlinks_Sip inf: res_ntpd_Sip<br>
> res_drbdlinks_Sip<br>
> colocation col_res_postfix_Sip_res_drbdlinks_Sip inf: res_postfix_Sip<br>
> res_drbdlinks_Sip<br>
> order ord_ms_drbd_1_res_Filesystem_Sip inf: ms_drbd_1:promote<br>
> res_Filesystem_Sip:start<br>
> order ord_res_Filesystem_Sip_res_IPaddr2_Admin inf: res_Filesystem_Sip<br>
> res_IPaddr2_Admin<br>
> order ord_res_Filesystem_Sip_res_IPaddr2_Sip inf: res_Filesystem_Sip<br>
> res_IPaddr2_Sip<br>
> order ord_res_IPaddr2_Asterisk_res_asterisk_Asterisk inf:<br>
> res_IPaddr2_Asterisk res_asterisk_Asterisk<br>
> order ord_res_IPaddr2_Sip_res_drbdlinks_Sip inf: res_IPaddr2_Sip<br>
> res_drbdlinks_Sip<br>
> order ord_res_drbdlinks_Sip_res_IPaddr2_Asterisk inf: res_drbdlinks_Sip<br>
> res_IPaddr2_Asterisk<br>
> order ord_res_drbdlinks_Sip_res_dhcpd_Sip inf: res_drbdlinks_Sip<br>
> res_dhcpd_Sip<br>
> order ord_res_drbdlinks_Sip_res_faxmodems_Sip inf: res_drbdlinks_Sip<br>
> res_faxmodems_Sip<br>
> order ord_res_drbdlinks_Sip_res_httpd_Sip inf: res_drbdlinks_Sip<br>
> res_httpd_Sip<br>
> order ord_res_drbdlinks_Sip_res_hylafax_Sip inf: res_drbdlinks_Sip<br>
> res_hylafax_Sip<br>
> order ord_res_drbdlinks_Sip_res_iaxmodem_Sip inf: res_drbdlinks_Sip<br>
> res_iaxmodem_Sip<br>
> order ord_res_drbdlinks_Sip_res_kamailio_Sip inf: res_drbdlinks_Sip<br>
> res_kamailio_Sip<br>
> order ord_res_drbdlinks_Sip_res_mysqld_Sip inf: res_drbdlinks_Sip<br>
> res_mysqld_Sip<br>
> order ord_res_drbdlinks_Sip_res_named_Sip inf: res_drbdlinks_Sip<br>
> res_named_Sip<br>
> order ord_res_drbdlinks_Sip_res_nfs_Sip inf: res_drbdlinks_Sip res_nfs_Sip<br>
> order ord_res_drbdlinks_Sip_res_ntpd_Sip inf: res_drbdlinks_Sip res_ntpd_Sip<br>
> order ord_res_drbdlinks_Sip_res_postfix_Sip inf: res_drbdlinks_Sip<br>
> res_postfix_Sip<br>
> order ord_res_kamailio_Sip_res_IPaddr2_Asterisk inf: res_kamailio_Sip<br>
> res_IPaddr2_Asterisk<br>
> order ord_res_mysqld_Sip_res_kamailio_Sip inf: res_mysqld_Sip<br>
> res_kamailio_Sip<br>
> order ord_res_named_Sip_res_kamailio_Sip inf: res_named_Sip res_kamailio_Sip<br>
> property $id="cib-bootstrap-options" \<br>
> expected-quorum-votes="2" \<br>
> stonith-enabled="false" \<br>
> dc-version="1.1.6-3.el6-a02c0f19a00c1eb2527ad38f146ebc0834814558" \<br>
> no-quorum-policy="ignore" \<br>
> cluster-infrastructure="openais" \<br>
> last-lrm-refresh="1353415104"<br>
> rsc_defaults $id="rsc-options" \<br>
> resource-stickiness="100"<br>
><br>
><br>
> Regards,<br>
> Pedro Sousa<br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
><br>
</div></div>> _______________________________________________<br>
> Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br>
> <a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
><br>
> Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
> Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
> Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a><br>
><br>
<br>
_______________________________________________<br>
Pacemaker mailing list: <a href="mailto:Pacemaker@oss.clusterlabs.org">Pacemaker@oss.clusterlabs.org</a><br>
<a href="http://oss.clusterlabs.org/mailman/listinfo/pacemaker" target="_blank">http://oss.clusterlabs.org/mailman/listinfo/pacemaker</a><br>
<br>
Project Home: <a href="http://www.clusterlabs.org" target="_blank">http://www.clusterlabs.org</a><br>
Getting started: <a href="http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf" target="_blank">http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf</a><br>
Bugs: <a href="http://bugs.clusterlabs.org" target="_blank">http://bugs.clusterlabs.org</a><br>
</blockquote></div><br></div>