[Pacemaker] S_POLICY_ENGINE state continues being maintained

Andrew Beekhof andrew at beekhof.net
Fri May 24 00:38:21 EDT 2013 

On 24/05/2013, at 2:19 PM, Andrew Beekhof <andrew at beekhof.net> wrote:

> 
> On 23/05/2013, at 4:44 PM, Kazunori INOUE <inouekazu at intellilink.co.jp> wrote:
> 
>> Hi,
>> 
>> I'm using pacemaker-1.1 (c3486a4a8d. the latest devel).
>> After fencing caused by split-brain failed 11 times, S_POLICY_ENGINE state is kept even if I recover split-brain.
> 
> Odd, I get:
> 
> May 24 00:17:08 corosync-host-1 crmd[3056]:   notice: tengine_stonith_callback: Stonith operation 12/69:23:0:9b069b96-3565-4219-85a5-8782bdb5d9d3: No route to host (-113)
> May 24 00:17:08 corosync-host-1 crmd[3056]:   notice: tengine_stonith_callback: Stonith operation 12 for corosync-host-6 failed (No route to host): aborting transition.
> May 24 00:17:08 corosync-host-1 crmd[3056]:   notice: run_graph: Transition 23 (Complete=1, Pending=0, Fired=0, Skipped=2, Incomplete=0, Source=/var/lib/pacemaker/pengine/pe-warn-110.bz2): Stopped
> May 24 00:17:08 corosync-host-1 crmd[3056]:   notice: too_many_st_failures: Too many failures to fence corosync-host-6 (11), giving up
> May 24 00:17:08 corosync-host-1 crmd[3056]:   notice: do_state_transition: State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]
> May 24 00:17:08 corosync-host-1 crmd[3056]:   notice: tengine_stonith_notify: Peer corosync-host-6 was not terminated (reboot) by corosync-host-1 for corosync-host-1: No route to host (ref=9dd3711e-c87d-4b2e-acd1-854391a6fa9d) by client crmd.3056

Same for you:

May 23 13:17:28 [24868] dev1       crmd:   notice: too_many_st_failures: 	Too many failures to fence dev2 (11), giving up
May 23 13:17:28 [24868] dev1       crmd:    debug: notify_crmd: 	Transition 10 status: restart - Stonith failed
May 23 13:17:28 [24868] dev1       crmd:    debug: s_crmd_fsa: 	Processing I_TE_SUCCESS: [ state=S_TRANSITION_ENGINE cause=C_FSA_INTERNAL origin=notify_crmd ]
May 23 13:17:28 [24868] dev1       crmd:     info: do_log: 	FSA: Input I_TE_SUCCESS from notify_crmd() received in state S_TRANSITION_ENGINE

and

May 23 13:17:28 [7107] dev2       crmd:   notice: too_many_st_failures: 	Too many failures to fence dev1 (11), giving up
May 23 13:17:28 [7107] dev2       crmd:    debug: notify_crmd: 	Transition 13 status: restart - Stonith failed
May 23 13:17:28 [7107] dev2       crmd:    debug: s_crmd_fsa: 	Processing I_TE_SUCCESS: [ state=S_TRANSITION_ENGINE cause=C_FSA_INTERNAL origin=notify_crmd ]
May 23 13:17:28 [7107] dev2       crmd:     info: do_log: 	FSA: Input I_TE_SUCCESS from notify_crmd() received in state S_TRANSITION_ENGINE
May 23 13:17:28 [7107] dev2       crmd:   notice: do_state_transition: 	State transition S_TRANSITION_ENGINE -> S_IDLE [ input=I_TE_SUCCESS cause=C_FSA_INTERNAL origin=notify_crmd ]

oh, but not here:

May 23 13:24:23 [7107] dev2       crmd:    debug: do_te_invoke: 	Cancelling the transition: inactive
May 23 13:24:23 [7107] dev2       crmd:     info: abort_transition_graph: 	do_te_invoke:155 - Triggered transition abort (complete=1) : Peer Cancelled
May 23 13:24:23 [7107] dev2       crmd:   notice: too_many_st_failures: 	Too many failures to fence dev1 (11), giving up
May 23 13:24:23 [7107] dev2       crmd:    debug: s_crmd_fsa: 	Processing I_TE_SUCCESS: [ state=S_POLICY_ENGINE cause=C_FSA_INTERNAL origin=abort_transition_graph ]
May 23 13:24:23 [7107] dev2       crmd:  warning: do_log: 	FSA: Input I_TE_SUCCESS from abort_transition_graph() received in state S_POLICY_ENGINE
May 23 13:24:23 [7107] dev2       crmd:    debug: te_update_diff: 	Processing diff (cib_modify): 0.5.24 -> 0.5.25 (S_POLICY_ENGINE)
May 23 13:24:23 [7107] dev2       crmd:    debug: te_update_diff: 	Processing diff (cib_modify): 0.5.25 -> 0.5.26 (S_POLICY_ENGINE)
May 23 13:24:23 [7107] dev2       crmd:    debug: join_update_complete_callback: 	Join update 95 complete
May 23 13:24:23 [7107] dev2       crmd:    debug: check_join_state: 	Invoked by join_update_complete_callback in state: S_POLICY_ENGINE
May 23 13:47:54 [7107] dev2       crmd:   notice: handle_request: 	Current ping state: S_POLICY_ENGINE

Can you try the following patch?

diff --git a/crmd/te_utils.c b/crmd/te_utils.c
index ae4c5de..f3e0d9f 100644
--- a/crmd/te_utils.c
+++ b/crmd/te_utils.c
@@ -408,15 +408,11 @@ abort_transition_graph(int abort_priority, enum transition_action abort_action,
     fsa_pe_ref = NULL;
 
     if (transition_graph->complete) {
-        if (too_many_st_failures() == FALSE) {
-            if (transition_timer->period_ms > 0) {
-                crm_timer_stop(transition_timer);
-                crm_timer_start(transition_timer);
-            } else {
-                register_fsa_input(C_FSA_INTERNAL, I_PE_CALC, NULL);
-            }
+        if (transition_timer->period_ms > 0) {
+            crm_timer_stop(transition_timer);
+            crm_timer_start(transition_timer);
         } else {
-            register_fsa_input(C_FSA_INTERNAL, I_TE_SUCCESS, NULL);
+            register_fsa_input(C_FSA_INTERNAL, I_PE_CALC, NULL);
         }
         return;
     }


The expected behavior is that after too_many_st_failures() returns true, we will retry once per re-check interval until either the node is confirmed down with stonith_admin -C or fencing succeeds.
If the node comes back and fencing is no longer needed, but has still not been confirmed to work, then the count in too_many_st_failures() is not cleared.

Make sense?


> 
> 
> 
>> 
>> 1. disconnect network connection
>> [dev1 ~]$ crm_mon
>> Last updated: Thu May 23 13:16:41 2013
>> Last change: Thu May 23 13:15:30 2013 via cibadmin on dev1
>> Stack: corosync
>> Current DC: dev1 (3232261525) - partition WITHOUT quorum
>> Version: 1.1.10-0.122.c3486a4.git.el6-c3486a4
>> 2 Nodes configured, unknown expected votes
>> 2 Resources configured.
>> 
>> 
>> Node dev2 (3232261523): UNCLEAN (offline)
>> Online: [ dev1 ]
>> 
>> f1      (stonith:external/libvirt.NG):  Started dev2
>> f2      (stonith:external/libvirt.NG):  Started dev1
>> 
>> [dev2 ~]$ crm_mon
>> Last updated: Thu May 23 13:16:41 2013
>> Last change: Thu May 23 13:15:30 2013 via cibadmin on dev1
>> Stack: corosync
>> Current DC: dev2 (3232261523) - partition WITHOUT quorum
>> Version: 1.1.10-0.122.c3486a4.git.el6-c3486a4
>> 2 Nodes configured, unknown expected votes
>> 2 Resources configured.
>> 
>> 
>> Node dev1 (3232261525): UNCLEAN (offline)
>> Online: [ dev2 ]
>> 
>> f1      (stonith:external/libvirt.NG):  Started dev2
>> f2      (stonith:external/libvirt.NG):  Started dev1
>> 
>> 
>> 2. wait until fencing failed 11 times
>> [dev1 ~]$ egrep "CRIT|too_many_st_failures" /var/log/ha-log
>> May 23 13:16:46 dev1 stonith: [24981]: CRIT: external_reset_req: 'libvirt.NG reset' for host dev2 failed with rc 1
>> (snip)
>> May 23 13:17:24 dev1 stonith: [25105]: CRIT: external_reset_req: 'libvirt.NG reset' for host dev2 failed with rc 1
>> May 23 13:17:28 dev1 stonith: [25118]: CRIT: external_reset_req: 'libvirt.NG reset' for host dev2 failed with rc 1
>> May 23 13:17:28 dev1 crmd[24868]:   notice: too_many_st_failures: Too many failures to fence dev2 (11), giving up
>> 
>> [dev2 ~]$ egrep "CRIT|too_many_st_failures" /var/log/ha-log
>> May 23 13:16:46 dev2 stonith: [7177]: CRIT: external_reset_req: 'libvirt.NG reset' for host dev1 failed with rc 1
>> (snip)
>> May 23 13:17:23 dev2 stonith: [7295]: CRIT: external_reset_req: 'libvirt.NG reset' for host dev1 failed with rc 1
>> May 23 13:17:28 dev2 stonith: [7309]: CRIT: external_reset_req: 'libvirt.NG reset' for host dev1 failed with rc 1
>> May 23 13:17:28 dev2 crmd[7107]:   notice: too_many_st_failures: Too many failures to fence dev1 (11), giving up
>> 
>> 
>> 3. recover network disconnection
>> [dev1 ~]$ crm_mon
>> Last updated: Thu May 23 13:24:23 2013
>> Last change: Thu May 23 13:15:30 2013 via cibadmin on dev1
>> Stack: corosync
>> Current DC: dev2 (3232261523) - partition with quorum
>> Version: 1.1.10-0.122.c3486a4.git.el6-c3486a4
>> 2 Nodes configured, unknown expected votes
>> 2 Resources configured.
>> 
>> 
>> Online: [ dev1 dev2 ]
>> 
>> f1      (stonith:external/libvirt.NG):  Started dev2
>> f2      (stonith:external/libvirt.NG):  Started dev1
>> 
>> 
>> S_POLICY_ENGINE state continues being maintained although a member's join seems to have succeeded.
>> 
>> [13:47:54 root at dev1 ~]$ crmadmin -S dev2
>> Status of crmd at dev2: S_POLICY_ENGINE (ok)
>> 
>> 
>> Best Regards,
>> Kazunori INOUE
>> <keeping-S_POLICY_ENGINE.tar.bz2>_______________________________________________
>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>> 
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
>

More information about the Pacemaker mailing list