[ClusterLabs] Antw: Re: Resources start serial, not parralel
Oleg Ilyin
on.ilyin at gmail.com
Wed Dec 16 09:44:20 UTC 2015
Hi guys,
have you any ideas for root cause of trouble?
I will appreciate any clues for investigation.
2015-12-14 17:40 GMT+09:00 Oleg Ilyin <on.ilyin at gmail.com>:
> Hi Ulrich,
>
> thank you for your answer,
>
> what about which limits do you talk?
>
> The pacemaker software runs by root user, so limits should be increased
> for root.
> There is output from one of my server:
>
> -bash-4.1# id
> uid=0(root) gid=0(root) groups=0(root)
>
> -bash-4.1# ulimit -a
> core file size (blocks, -c) 0
> data seg size (kbytes, -d) unlimited
> scheduling priority (-e) 0
> file size (blocks, -f) unlimited
> pending signals (-i) 124801
> max locked memory (kbytes, -l) 64
> max memory size (kbytes, -m) unlimited
> open files (-n) 4096
> pipe size (512 bytes, -p) 8
> POSIX message queues (bytes, -q) 819200
> real-time priority (-r) 0
> stack size (kbytes, -s) 10240
> cpu time (seconds, -t) unlimited
> max user processes (-u) 124801
> virtual memory (kbytes, -v) unlimited
> file locks (-x) unlimited
>
> -bash-4.1# ps -ef |grep pacemaker
> root 17947 1 0 Dec11 ? 00:00:09 pacemakerd
> 189 17953 17947 0 Dec11 ? 00:00:11 /usr/libexec/pacemaker/cib
> root 17954 17947 0 Dec11 ? 00:00:14
> /usr/libexec/pacemaker/stonithd
> root 17955 17947 0 Dec11 ? 00:00:11 /usr/libexec/pacemaker/lrmd
> 189 17956 17947 0 Dec11 ? 00:00:09
> /usr/libexec/pacemaker/attrd
> 189 17957 17947 0 Dec11 ? 00:00:09
> /usr/libexec/pacemaker/pengine
> root 17958 17947 0 Dec11 ? 00:00:16 /usr/libexec/pacemaker/crmd
>
>
> With settings in pacemaker or system can be changed for start heavy
> application ( java ) at the same time?
>
>
>
>
> 2015-12-14 16:29 GMT+09:00 Ulrich Windl <Ulrich.Windl at rz.uni-regensburg.de
> >:
>
>> Hi!
>>
>> There is one feature in Linux that may affect you: If processes block on
>> I/O (NFS also), the load increases, and the load is the _sum_, and not the
>> _average_ of all CPUs. So if you have many CPUs, your abservable load will
>> typically increase. Recently we had a load of 60, but nobody actually
>> noticed ;-)
>>
>> So maybe you just need to adjust the limits for pacemaker...
>>
>> Regards,
>> Ulrich
>>
>> >>> Oleg Ilyin <on.ilyin at gmail.com> schrieb am 13.12.2015 um 15:00 in
>> Nachricht
>> <CANLWoXrpvLAbcgCOE073iAOJz3zL7RAUB4UQC=0iPfGyTvnidw at mail.gmail.com>:
>> > There are errors in my /var/log/messages
>> >
>> >
>> > grep -e crmd\\[ -e crmd: /var/log/messages
>> > Dec 13 00:01:09 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 3.290000
>> > Dec 13 00:01:39 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 4.430000
>> > Dec 13 00:02:09 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 3.900000
>> > Dec 13 00:02:39 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 3.810000
>> > Dec 13 00:04:25 server_name_1 crmd[9941]: notice: do_state_transition:
>> > State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_
>> > TIMER_POPPED origin=crm_timer_popped ]
>> > Dec 13 00:04:26 server_name_1 crmd[9941]: notice: run_graph:
>> Transition
>> > 185166 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Sou
>> > rce=/var/lib/pacemaker/pengine/pe-input-656.bz2):
>> > Complete
>> > Dec 13 00:04:26 server_name_1 crmd[9941]: notice: do_state_transition:
>> > State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS c
>> > ause=C_FSA_INTERNAL origin=notify_crmd ]
>> > Dec 13 00:08:09 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 3.480000
>> > Dec 13 00:09:09 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 5.820000
>> > Dec 13 00:09:39 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 4.130000
>> > Dec 13 00:10:09 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 4.650000
>> > Dec 13 00:10:39 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 6.000000
>> > Dec 13 00:11:09 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 5.270000
>> > Dec 13 00:11:39 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 4.260000
>> > Dec 13 00:12:09 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 3.600000
>> > Dec 13 00:12:39 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 3.970000
>> > Dec 13 00:13:09 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 3.930000
>> > Dec 13 00:14:09 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 3.590000
>> > Dec 13 00:17:09 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 3.350000
>> > Dec 13 00:18:09 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 3.220000
>> > Dec 13 00:18:39 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 3.830000
>> > Dec 13 00:19:26 server_name_1 crmd[9941]: notice: do_state_transition:
>> > State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_
>> > TIMER_POPPED origin=crm_timer_popped ]
>> > Dec 13 00:19:26 server_name_1 crmd[9941]: notice: run_graph:
>> Transition
>> > 185167 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Sou
>> > rce=/var/lib/pacemaker/pengine/pe-input-656.bz2):
>> > Complete
>> > Dec 13 00:19:26 server_name_1 crmd[9941]: notice: do_state_transition:
>> > State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS c
>> > ause=C_FSA_INTERNAL origin=notify_crmd ]
>> > Dec 13 00:24:09 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 3.770000
>> > Dec 13 00:24:39 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 3.360000
>> > Dec 13 00:25:09 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 3.790000
>> > Dec 13 00:26:09 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 3.470000
>> > Dec 13 00:27:09 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 4.890000
>> > Dec 13 00:27:39 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 3.290000
>> > Dec 13 00:28:09 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 4.040000
>> > Dec 13 00:29:09 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 3.340000
>> > Dec 13 00:29:39 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 3.250000
>> > Dec 13 00:30:39 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 3.310000
>> > Dec 13 00:31:09 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 4.880000
>> > Dec 13 00:31:39 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 3.890000
>> > Dec 13 00:32:09 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 5.320000
>> > Dec 13 00:32:39 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 4.380000
>> > Dec 13 00:33:09 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 4.430000
>> > Dec 13 00:33:39 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 3.480000
>> > Dec 13 00:34:26 server_name_1 crmd[9941]: notice: do_state_transition:
>> > State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_
>> > TIMER_POPPED origin=crm_timer_popped ]
>> > Dec 13 00:34:26 server_name_1 crmd[9941]: notice: run_graph:
>> Transition
>> > 185168 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Sou
>> > rce=/var/lib/pacemaker/pengine/pe-input-656.bz2):
>> > Complete
>> > Dec 13 00:34:26 server_name_1 crmd[9941]: notice: do_state_transition:
>> > State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS c
>> > ause=C_FSA_INTERNAL origin=notify_crmd ]
>> > Dec 13 00:34:39 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 3.580000
>> > Dec 13 00:35:09 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 3.350000
>> > Dec 13 00:37:39 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 3.380000
>> > Dec 13 00:38:09 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 4.400000
>> > Dec 13 00:38:39 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 5.020000
>> > Dec 13 00:39:09 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 4.080000
>> > Dec 13 00:40:09 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 3.410000
>> > Dec 13 00:40:39 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 3.490000
>> > Dec 13 00:41:09 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 4.410000
>> > Dec 13 00:41:39 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 5.120000
>> > Dec 13 00:42:09 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 8.120000
>> > Dec 13 00:42:39 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 9.040000
>> > Dec 13 00:43:09 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 6.900000
>> > Dec 13 00:43:39 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 5.830000
>> > Dec 13 00:44:09 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 6.280000
>> > Dec 13 00:44:39 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 4.540000
>> > Dec 13 00:45:09 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 5.050000
>> > Dec 13 00:45:39 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 4.170000
>> > Dec 13 00:46:09 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 4.930000
>> > Dec 13 00:46:39 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 3.520000
>> > Dec 13 00:47:09 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 5.680000
>> > Dec 13 00:47:39 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 5.050000
>> > Dec 13 00:48:09 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 4.750000
>> > Dec 13 00:48:39 server_name_1 crmd[9941]: notice:
>> throttle_handle_load:
>> > High CPU load detected: 3.370000
>> > Dec 13 00:49:26 server_name_1 crmd[9941]: notice: do_state_transition:
>> > State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_
>> > TIMER_POPPED origin=crm_timer_popped ]
>> > Dec 13 00:49:26 server_name_1 crmd[9941]: notice: run_graph:
>> Transition
>> > 185169 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Sou
>> > rce=/var/lib/pacemaker/pengine/pe-input-656.bz2):
>> > Complete
>> > Dec 13 00:49:26 server_name_1 crmd[9941]: notice: do_state_transition:
>> > State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS c
>> > ause=C_FSA_INTERNAL origin=notify_crmd ]
>> >
>> >
>> > 2015-12-13 21:11 GMT+09:00 Oleg Ilyin <on.ilyin at gmail.com>:
>> >
>> >> Hello,
>> >> could you help please with issue on my environment
>> >> RHEL 6.6
>> >> corosync-1.4.7
>> >> pacemaker-1.1.12
>> >>
>> >> When I start several resources in the same time by " pcs resource
>> enable "
>> >> command,
>> >> resource doesn't start parralel, they start serial.
>> >>
>> >> If I start resources by one, thay start parralel. I didn't do any
>> special
>> >> settings.
>> >>
>> >> Please,
>> >> which parameters can affect to this behavior of cluster?
>> >>
>>
>>
>>
>>
>> _______________________________________________
>> Users mailing list: Users at clusterlabs.org
>> http://clusterlabs.org/mailman/listinfo/users
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>>
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20151216/7853b139/attachment-0002.html>
More information about the Users
mailing list