<html xmlns:v="urn:schemas-microsoft-com:vml" xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:w="urn:schemas-microsoft-com:office:word" xmlns:m="http://schemas.microsoft.com/office/2004/12/omml" xmlns="http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv="Content-Type" content="text/html; charset=us-ascii">
<meta name="Generator" content="Microsoft Word 14 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0cm;
margin-bottom:.0001pt;
font-size:11.0pt;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
a:link, span.MsoHyperlink
{mso-style-priority:99;
color:blue;
text-decoration:underline;}
a:visited, span.MsoHyperlinkFollowed
{mso-style-priority:99;
color:purple;
text-decoration:underline;}
span.EmailStyle17
{mso-style-type:personal-compose;
font-family:"Calibri","sans-serif";
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-family:"Calibri","sans-serif";
mso-fareast-language:EN-US;}
@page WordSection1
{size:612.0pt 792.0pt;
margin:70.85pt 70.85pt 70.85pt 70.85pt;}
div.WordSection1
{page:WordSection1;}
--></style><!--[if gte mso 9]><xml>
<o:shapedefaults v:ext="edit" spidmax="1026" />
</xml><![endif]--><!--[if gte mso 9]><xml>
<o:shapelayout v:ext="edit">
<o:idmap v:ext="edit" data="1" />
</o:shapelayout></xml><![endif]-->
</head>
<body lang="FR" link="blue" vlink="purple">
<div class="WordSection1">
<p class="MsoNormal"><span lang="EN-US">Dear all,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">We are running corosync-1.4.1-4.el6.x86_64 on RHEL 6.2. We have a cluster with 2 nodes that run as services a Virtual IP and around 20 custom LSB scripts.
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">We had some network issue 2 weeks ago and we got a strange behavior.
<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">On the logs below we can see that at 06 08:36:29 that the connection between the nodes has been lost, then we see a strange log “</span><b><i><span lang="EN-US" style="font-size:10.0pt;color:red">pengine: [2621]: ERROR:
unpack_rsc_op: Hard error - 5-controller_last_failure_0 failed with rc=3: Preventing 5-controller from re-starting on srv01”.<br>
</span></i></b><span lang="EN-US"><br>
Only one of our LSB script “5-controller” got an “Hard error” and this service got the status “unmanaged”. This means that once the network got back, the 2 nodes communicate again, and all services have been restarted , excepted “5-controller”.<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Do you have an idea about what happened ?<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Thanks a lot,<o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US">Jerome <o:p></o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><span lang="EN-US"><o:p> </o:p></span></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:31:35 srv01 cib: [2618]: info: cib_stats: Processed 71 operations (5633.00us average, 0% utilization) in the last 10min<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:28 corosync [TOTEM ] A processor failed, forming new configuration.<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 corosync [pcmk ] notice: pcmk_peer_update: Transitional membership event on ring 156: memb=1, new=0, lost=1<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 corosync [pcmk ] info: pcmk_peer_update: memb: srv01 1661074698<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 corosync [pcmk ] info: pcmk_peer_update: lost: srv02 1677851914<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 corosync [pcmk ] notice: pcmk_peer_update: Stable membership event on ring 156: memb=1, new=0, lost=0<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 corosync [pcmk ] info: pcmk_peer_update: MEMB: srv01 1661074698<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 corosync [pcmk ] info: ais_mark_unseen_peer_dead: Node srv02 was not seen in the previous transition<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 corosync [pcmk ] info: update_member: Node 1677851914/srv02 is now: lost<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 corosync [pcmk ] info: send_member_notification: Sending membership update 156 to 2 children<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 srv01 cib: [2618]: notice: ais_dispatch_message: Membership 156: quorum lost<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 srv01 cib: [2618]: info: crm_update_peer: Node srv02: id=1677851914 state=lost (new) addr=r(0) ip(10.10.1.100) votes=1 born=76 seen=84 proc=00000000000000000000000000111312<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 corosync [TOTEM ] A processor joined or left the membership and a new membership was formed.<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 corosync [CPG ] chosen downlist: sender r(0) ip(10.10.1.99) ; members(old:2 left:1)<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 corosync [MAIN ] Completed service synchronization, ready to provide service.<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 srv01 crmd: [2622]: notice: ais_dispatch_message: Membership 156: quorum lost<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 srv01 crmd: [2622]: info: ais_status_callback: status: srv02 is now lost (was member)<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 srv01 crmd: [2622]: info: crm_update_peer: Node srv02: id=1677851914 state=lost (new) addr=r(0) ip(10.10.1.100) votes=1 born=76 seen=84 proc=00000000000000000000000000111312<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 srv01 crmd: [2622]: WARN: check_dead_member: Our DC node (srv02) left the cluster<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 srv01 crmd: [2622]: info: do_state_transition: State transition S_NOT_DC -> S_ELECTION [ input=I_ELECTION cause=C_FSA_INTERNAL origin=check_dead_member ]<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 srv01 crmd: [2622]: info: update_dc: Unset DC srv02<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 srv01 crmd: [2622]: info: do_state_transition: State transition S_ELECTION -> S_INTEGRATION [ input=I_ELECTION_DC cause=C_FSA_INTERNAL origin=do_election_check ]<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 srv01 crmd: [2622]: info: do_te_control: Registering TE UUID: ceab74dd-6a27-4eda-ba66-b63d77beaee5<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 srv01 crmd: [2622]: info: set_graph_functions: Setting custom graph functions<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 srv01 crmd: [2622]: info: unpack_graph: Unpacked transition -1: 0 actions in 0 synapses<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 srv01 crmd: [2622]: info: do_dc_takeover: Taking over DC status for this partition<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 srv01 cib: [2618]: info: cib_process_readwrite: We are now in R/W mode<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 srv01 cib: [2618]: info: cib_process_request: Operation complete: op cib_master for section 'all' (origin=local/crmd/709, version=0.2504.157): ok (rc=0)<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 srv01 cib: [2618]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/710, version=0.2504.158): ok (rc=0)<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 srv01 cib: [2618]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/712, version=0.2504.159): ok (rc=0)<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 srv01 crmd: [2622]: info: join_make_offer: Making join offers based on membership 156<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 srv01 crmd: [2622]: info: do_dc_join_offer_all: join-1: Waiting on 1 outstanding join acks<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 srv01 cib: [2618]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/714, version=0.2504.160): ok (rc=0)<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 srv01 crmd: [2622]: info: ais_dispatch_message: Membership 156: quorum still lost<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 srv01 crmd: [2622]: info: crmd_ais_dispatch: Setting expected votes to 2<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 srv01 crmd: [2622]: info: update_dc: Set DC to srv01 (3.0.5)<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 srv01 crmd: [2622]: info: config_query_callback: Shutdown escalation occurs after: 1200000ms<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 srv01 crmd: [2622]: info: config_query_callback: Checking for expired actions every 180000ms<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 srv01 crmd: [2622]: info: config_query_callback: Sending expected-votes=2 to corosync<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 srv01 crmd: [2622]: info: ais_dispatch_message: Membership 156: quorum still lost<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 srv01 cib: [2618]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/717, version=0.2504.161): ok (rc=0)<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 srv01 crmd: [2622]: info: crmd_ais_dispatch: Setting expected votes to 2<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 srv01 cib: [2618]: info: cib_process_request: Operation complete: op cib_modify for section crm_config (origin=local/crmd/720, version=0.2504.162): ok (rc=0)<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 srv01 crmd: [2622]: info: do_state_transition: State transition S_INTEGRATION -> S_FINALIZE_JOIN [ input=I_INTEGRATED cause=C_FSA_INTERNAL origin=check_join_state ]<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 srv01 crmd: [2622]: info: do_state_transition: All 1 cluster nodes responded to the join offer.<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 srv01 crmd: [2622]: info: do_dc_join_finalize: join-1: Syncing the CIB from srv01 to the rest of the cluster<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 srv01 cib: [2618]: info: cib_process_request: Operation complete: op cib_sync for section 'all' (origin=local/crmd/721, version=0.2504.162): ok (rc=0)<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:29 srv01 cib: [2618]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/722, version=0.2504.163): ok (rc=0)<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:30 srv01 crmd: [2622]: info: do_dc_join_ack: join-1: Updating node state to member for srv01<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:30 srv01 cib: [2618]: info: cib_process_request: Operation complete: op cib_delete for section //node_state[@uname='srv01']/lrm (origin=local/crmd/723, version=0.2504.164): ok
(rc=0)<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:30 srv01 crmd: [2622]: info: erase_xpath_callback: Deletion of "//node_state[@uname='srv01']/lrm": ok (rc=0)<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:30 srv01 crmd: [2622]: info: do_state_transition: State transition S_FINALIZE_JOIN -> S_POLICY_ENGINE [ input=I_FINALIZED cause=C_FSA_INTERNAL origin=check_join_state ]<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:30 srv01 crmd: [2622]: info: do_state_transition: All 1 cluster nodes are eligible to run resources.<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:30 srv01 crmd: [2622]: info: do_dc_join_final: Ensuring DC, quorum and node attributes are up-to-date<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:30 srv01 crmd: [2622]: info: crm_update_quorum: Updating quorum status to false (call=727)<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:30 srv01 crmd: [2622]: info: abort_transition_graph: do_te_invoke:162 - Triggered transition abort (complete=1) : Peer Cancelled<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:30 srv01 crmd: [2622]: info: do_pe_invoke: Query 728: Requesting the current CIB: S_POLICY_ENGINE<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:30 srv01 cib: [2618]: info: cib_process_request: Operation complete: op cib_modify for section nodes (origin=local/crmd/725, version=0.2504.166): ok (rc=0)<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:30 srv01 attrd: [2620]: notice: attrd_local_callback: Sending full refresh (origin=crmd)<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:30 srv01 attrd: [2620]: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-record-events (1413990260)<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:30 srv01 crmd: [2622]: WARN: match_down_event: No match for shutdown action on srv02<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:30 srv01 crmd: [2622]: info: te_update_diff: Stonith/shutdown of srv02 not matched<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:30 srv01 crmd: [2622]: info: abort_transition_graph: te_update_diff:234 - Triggered transition abort (complete=1, tag=node_state, id=srv02, magic=NA, cib=0.2504.167) : Node failure<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:30 srv01 crmd: [2622]: info: do_pe_invoke: Query 729: Requesting the current CIB: S_POLICY_ENGINE<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:30 srv01 cib: [2618]: info: cib_process_request: Operation complete: op cib_modify for section cib (origin=local/crmd/727, version=0.2504.168): ok (rc=0)<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:30 srv01 attrd: [2620]: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-replay-1 (1415025615)<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:30 srv01 attrd: [2620]: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-4-server (1413990250)<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:30 srv01 crmd: [2622]: info: do_pe_invoke_callback: Invoking the PE: query=729, ref=pe_calc-dc-1415259390-10, seq=156, quorate=0<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:30 srv01 attrd: [2620]: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-replay-2 (1415025597)<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:30 srv01 attrd: [2620]: notice: attrd_trigger_update: Sending flush op to all hosts for: probe_complete (true)<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:30 srv01 attrd: [2620]: notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-Proxy (1)<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:30 srv01 attrd: [2620]: notice: attrd_trigger_update: Sending flush op to all hosts for: last-failure-record-1 (1414568098)<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:30 srv01 attrd: [2620]: notice: attrd_trigger_update: Sending flush op to all hosts for: fail-count-replay-1 (1)<o:p></o:p></span></i></p>
<p class="MsoNormal"><i><span lang="EN-US" style="font-size:10.0pt">Nov 06 08:36:30 srv01 pengine: [2621]: notice: unpack_config: On loss of CCM Quorum: Ignore<o:p></o:p></span></i></p>
<p class="MsoNormal"><b><i><span lang="EN-US" style="font-size:10.0pt;color:red">Nov 06 08:36:30 srv01 pengine: [2621]: ERROR: unpack_rsc_op: Hard error - 5-controller_last_failure_0 failed with rc=3: Preventing 5-controller from re-starting on srv01<o:p></o:p></span></i></b></p>
<p class="MsoNormal"><b><i><span lang="EN-US" style="font-size:10.0pt;color:red">Nov 06 08:36:30 srv01 pengine: [2621]: WARN: unpack_rsc_op: Processing failed op 5-controller_last_failure_0 on srv01: unimplemented feature (3)<o:p></o:p></span></i></b></p>
</div>
</body>
</html>