[ClusterLabs] Antw: Re: Resources start serial, not parralel
Ulrich Windl
Ulrich.Windl at rz.uni-regensburg.de
Mon Dec 14 02:29:43 EST 2015
Hi!
There is one feature in Linux that may affect you: If processes block on I/O (NFS also), the load increases, and the load is the _sum_, and not the _average_ of all CPUs. So if you have many CPUs, your abservable load will typically increase. Recently we had a load of 60, but nobody actually noticed ;-)
So maybe you just need to adjust the limits for pacemaker...
Regards,
Ulrich
>>> Oleg Ilyin <on.ilyin at gmail.com> schrieb am 13.12.2015 um 15:00 in Nachricht
<CANLWoXrpvLAbcgCOE073iAOJz3zL7RAUB4UQC=0iPfGyTvnidw at mail.gmail.com>:
> There are errors in my /var/log/messages
>
>
> grep -e crmd\\[ -e crmd: /var/log/messages
> Dec 13 00:01:09 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 3.290000
> Dec 13 00:01:39 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 4.430000
> Dec 13 00:02:09 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 3.900000
> Dec 13 00:02:39 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 3.810000
> Dec 13 00:04:25 server_name_1 crmd[9941]: notice: do_state_transition:
> State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_
> TIMER_POPPED origin=crm_timer_popped ]
> Dec 13 00:04:26 server_name_1 crmd[9941]: notice: run_graph: Transition
> 185166 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Sou
> rce=/var/lib/pacemaker/pengine/pe-input-656.bz2):
> Complete
> Dec 13 00:04:26 server_name_1 crmd[9941]: notice: do_state_transition:
> State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS c
> ause=C_FSA_INTERNAL origin=notify_crmd ]
> Dec 13 00:08:09 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 3.480000
> Dec 13 00:09:09 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 5.820000
> Dec 13 00:09:39 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 4.130000
> Dec 13 00:10:09 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 4.650000
> Dec 13 00:10:39 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 6.000000
> Dec 13 00:11:09 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 5.270000
> Dec 13 00:11:39 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 4.260000
> Dec 13 00:12:09 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 3.600000
> Dec 13 00:12:39 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 3.970000
> Dec 13 00:13:09 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 3.930000
> Dec 13 00:14:09 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 3.590000
> Dec 13 00:17:09 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 3.350000
> Dec 13 00:18:09 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 3.220000
> Dec 13 00:18:39 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 3.830000
> Dec 13 00:19:26 server_name_1 crmd[9941]: notice: do_state_transition:
> State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_
> TIMER_POPPED origin=crm_timer_popped ]
> Dec 13 00:19:26 server_name_1 crmd[9941]: notice: run_graph: Transition
> 185167 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Sou
> rce=/var/lib/pacemaker/pengine/pe-input-656.bz2):
> Complete
> Dec 13 00:19:26 server_name_1 crmd[9941]: notice: do_state_transition:
> State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS c
> ause=C_FSA_INTERNAL origin=notify_crmd ]
> Dec 13 00:24:09 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 3.770000
> Dec 13 00:24:39 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 3.360000
> Dec 13 00:25:09 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 3.790000
> Dec 13 00:26:09 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 3.470000
> Dec 13 00:27:09 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 4.890000
> Dec 13 00:27:39 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 3.290000
> Dec 13 00:28:09 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 4.040000
> Dec 13 00:29:09 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 3.340000
> Dec 13 00:29:39 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 3.250000
> Dec 13 00:30:39 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 3.310000
> Dec 13 00:31:09 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 4.880000
> Dec 13 00:31:39 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 3.890000
> Dec 13 00:32:09 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 5.320000
> Dec 13 00:32:39 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 4.380000
> Dec 13 00:33:09 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 4.430000
> Dec 13 00:33:39 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 3.480000
> Dec 13 00:34:26 server_name_1 crmd[9941]: notice: do_state_transition:
> State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_
> TIMER_POPPED origin=crm_timer_popped ]
> Dec 13 00:34:26 server_name_1 crmd[9941]: notice: run_graph: Transition
> 185168 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Sou
> rce=/var/lib/pacemaker/pengine/pe-input-656.bz2):
> Complete
> Dec 13 00:34:26 server_name_1 crmd[9941]: notice: do_state_transition:
> State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS c
> ause=C_FSA_INTERNAL origin=notify_crmd ]
> Dec 13 00:34:39 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 3.580000
> Dec 13 00:35:09 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 3.350000
> Dec 13 00:37:39 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 3.380000
> Dec 13 00:38:09 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 4.400000
> Dec 13 00:38:39 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 5.020000
> Dec 13 00:39:09 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 4.080000
> Dec 13 00:40:09 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 3.410000
> Dec 13 00:40:39 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 3.490000
> Dec 13 00:41:09 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 4.410000
> Dec 13 00:41:39 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 5.120000
> Dec 13 00:42:09 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 8.120000
> Dec 13 00:42:39 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 9.040000
> Dec 13 00:43:09 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 6.900000
> Dec 13 00:43:39 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 5.830000
> Dec 13 00:44:09 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 6.280000
> Dec 13 00:44:39 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 4.540000
> Dec 13 00:45:09 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 5.050000
> Dec 13 00:45:39 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 4.170000
> Dec 13 00:46:09 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 4.930000
> Dec 13 00:46:39 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 3.520000
> Dec 13 00:47:09 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 5.680000
> Dec 13 00:47:39 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 5.050000
> Dec 13 00:48:09 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 4.750000
> Dec 13 00:48:39 server_name_1 crmd[9941]: notice: throttle_handle_load:
> High CPU load detected: 3.370000
> Dec 13 00:49:26 server_name_1 crmd[9941]: notice: do_state_transition:
> State transition S_IDLE -> S_POLICY_ENGINE [ input=I_PE_CALC cause=C_
> TIMER_POPPED origin=crm_timer_popped ]
> Dec 13 00:49:26 server_name_1 crmd[9941]: notice: run_graph: Transition
> 185169 (Complete=0, Pending=0, Fired=0, Skipped=0, Incomplete=0, Sou
> rce=/var/lib/pacemaker/pengine/pe-input-656.bz2):
> Complete
> Dec 13 00:49:26 server_name_1 crmd[9941]: notice: do_state_transition:
> State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS c
> ause=C_FSA_INTERNAL origin=notify_crmd ]
>
>
> 2015-12-13 21:11 GMT+09:00 Oleg Ilyin <on.ilyin at gmail.com>:
>
>> Hello,
>> could you help please with issue on my environment
>> RHEL 6.6
>> corosync-1.4.7
>> pacemaker-1.1.12
>>
>> When I start several resources in the same time by " pcs resource enable "
>> command,
>> resource doesn't start parralel, they start serial.
>>
>> If I start resources by one, thay start parralel. I didn't do any special
>> settings.
>>
>> Please,
>> which parameters can affect to this behavior of cluster?
>>
More information about the Users
mailing list