<HTML><HEAD>
<META content="text/html charset=utf-8" http-equiv=Content-Type></HEAD>
<BODY
style="WORD-WRAP: break-word; -webkit-nbsp-mode: space; -webkit-line-break: after-white-space"
dir=ltr>
<DIV dir=ltr>
<DIV style="FONT-SIZE: 10pt; FONT-FAMILY: 'Verdana'; COLOR: #000000">
<DIV>Hello Andrew,</DIV>
<DIV> </DIV>
<DIV>Thank you for your response.</DIV>
<DIV> </DIV>
<DIV>We have a two-node cluster, and we need to upgrade pacemaker at both
nodes.</DIV>
<DIV> </DIV>
<DIV>We ended up applying locally the patch[1], which sends explicit ACK if it
matches the old version of pacemaker.</DIV>
<DIV> </DIV>
<DIV>Thanks,</DIV>
<DIV>Alex.</DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV>[1]</DIV>
<DIV>--- a/pacemaker/pacemaker-1.1.13/crmd/lrm.c</DIV>
<DIV>+++ b/pacemaker/pacemaker-1.1.13/crmd/lrm.c</DIV>
<DIV>@@ -1541,20 +1541,44 @@ do_lrm_invoke(long long action,</DIV>
<DIV>
op->rc = PCMK_OCF_OK;</DIV>
<DIV>
op->op_status = PCMK_LRM_OP_DONE;</DIV>
<DIV>
send_direct_ack(from_host, from_sys, rsc, op, rsc->id);</DIV>
<DIV>
lrmd_free_event(op);</DIV>
<DIV> </DIV>
<DIV>
/* needed?? surely not otherwise the cancel_op_(_key) wouldn't</DIV>
<DIV>
* have failed in the first place</DIV>
<DIV>
*/</DIV>
<DIV>
g_hash_table_remove(lrm_state->pending_ops, op_key);</DIV>
<DIV>
}</DIV>
<DIV>+ else
{</DIV>
<DIV>+
const char *feature_set = NULL;</DIV>
<DIV>+
gboolean need_direct_ack = FALSE;</DIV>
<DIV>+</DIV>
<DIV>+
/*</DIV>
<DIV>+
* For uprading from older versions, we need to send explicit ACK.</DIV>
<DIV>+
* See:</DIV>
<DIV>+
* <A
href="https://github.com/ClusterLabs/pacemaker/commit/abcdaa8893d6071574986af6abc85ae558473735">https://github.com/ClusterLabs/pacemaker/commit/abcdaa8893d6071574986af6abc85ae558473735</A></DIV>
<DIV>+
* <A
href="http://clusterlabs.org/pipermail/developers/2016-May/000219.html">http://clusterlabs.org/pipermail/developers/2016-May/000219.html</A></DIV>
<DIV>+
*/</DIV>
<DIV>+
feature_set = crm_element_value(params, XML_ATTR_CRM_VERSION);</DIV>
<DIV>+
need_direct_ack = safe_str_eq(feature_set, "3.0.5");</DIV>
<DIV>+
crm_notice("PE requested op %s (call=%s) be cancelled in_progress==TRUE
feature_set=%s need_direct_ack=%d",</DIV>
<DIV>+
op_key, call_id ? call_id : "NA", feature_set, need_direct_ack);</DIV>
<DIV>+
if (need_direct_ack) {</DIV>
<DIV>+
lrmd_event_data_t *op = construct_op(lrm_state, input->xml, rsc->id,
op_task);</DIV>
<DIV>+</DIV>
<DIV>+
CRM_ASSERT(op != NULL);</DIV>
<DIV>+
op->rc = PCMK_OCF_OK;</DIV>
<DIV>+
op->op_status = PCMK_LRM_OP_DONE;</DIV>
<DIV>+
send_direct_ack(from_host, from_sys, rsc, op, rsc->id);</DIV>
<DIV>+
lrmd_free_event(op);</DIV>
<DIV>+
}</DIV>
<DIV>+ }</DIV>
<DIV> </DIV>
<DIV>
free(op_key);</DIV>
<DIV> </DIV>
<DIV> } else if (rsc != NULL
&& safe_str_eq(operation, CRMD_ACTION_DELETE)) {</DIV>
<DIV>
gboolean unregister = TRUE;</DIV>
<DIV> </DIV>
<DIV>#if ENABLE_ACL</DIV>
<DIV>
int cib_rc = delete_rsc_status(lrm_state, rsc->id, cib_dryrun |
cib_sync_call, user_name);</DIV>
<DIV> if
(cib_rc != pcmk_ok) {</DIV>
<DIV>
lrmd_event_data_t *op = NULL;</DIV>
<DIV> </DIV>
<DIV> </DIV>
<DIV
style='FONT-SIZE: small; TEXT-DECORATION: none; FONT-FAMILY: "Calibri"; FONT-WEIGHT: normal; COLOR: #000000; FONT-STYLE: normal; DISPLAY: inline'>
<DIV style="FONT: 10pt tahoma">
<DIV> </DIV>
<DIV style="BACKGROUND: #f5f5f5">
<DIV style="font-color: black"><B>From:</B> <A title=andrew@beekhof.net
href="mailto:andrew@beekhof.net">Andrew Beekhof</A> </DIV>
<DIV><B>Sent:</B> Friday, June 03, 2016 3:36 AM</DIV>
<DIV><B>To:</B> <A title=alex@zadarastorage.com
href="mailto:alex@zadarastorage.com">Alex Lyakas</A> </DIV>
<DIV><B>Cc:</B> <A title=Developers@clusterlabs.org
href="mailto:Developers@clusterlabs.org">Developers@clusterlabs.org</A> ; <A
title=yair@zadarastorage.com href="mailto:yair@zadarastorage.com">Yair
Hershko</A> ; <A title=shyam@zadarastorage.com
href="mailto:shyam@zadarastorage.com">Shyam Kaushik</A> ; <A
title=yaron@zadarastorage.com href="mailto:yaron@zadarastorage.com">Yaron
Presente</A> ; <A title=lev@zadarastorage.com
href="mailto:lev@zadarastorage.com">Lev Vainblat</A> </DIV>
<DIV><B>Subject:</B> Re: commit abcdaa8 breaks compatibility with older
pacemaker</DIV></DIV></DIV>
<DIV> </DIV></DIV>
<DIV
style='FONT-SIZE: small; TEXT-DECORATION: none; FONT-FAMILY: "Calibri"; FONT-WEIGHT: normal; COLOR: #000000; FONT-STYLE: normal; DISPLAY: inline'>
<DIV> </DIV>
<DIV>
<BLOCKQUOTE type="cite">
<DIV>On 24 May 2016, at 2:23 AM, Alex Lyakas <<A
href="mailto:alex@zadarastorage.com">alex@zadarastorage.com</A>>
wrote:</DIV>
<DIV> </DIV>
<DIV>
<DIV>Hello Andrew,<BR><BR>We have a system in the field running a MASTER-SLAVE
resource on two nodes. We are trying to upgrade the pacemaker on these two
nodes. First we upgrade the SLAVE node. Then we move the resource to be MASTER
on the upgraded SLAVE node (“crm node standby” on the old MASTER). This move
involves cancelling a monitor operation on the SLAVE node.<BR><BR>With
commit<BR><A
href="https://github.com/ClusterLabs/pacemaker/commit/abcdaa8893d6071574986af6abc85ae558473735">https://github.com/ClusterLabs/pacemaker/commit/abcdaa8893d6071574986af6abc85ae558473735</A><BR>there
is a change of how the “cancel” action is confirmed.<BR><BR>Previously,
send_direct_ack was always used to confirm the cancel action. But now, the
cancel action is being confirmed not by direct ACK but by parsing the
XML.<BR></DIV></DIV></BLOCKQUOTE>
<DIV> </DIV>
<DIV>Oh, and you’re mixing pacemaker versions.</DIV>
<DIV>I can see how that would be a problem.</DIV>
<DIV> </DIV>
<DIV>Are you seeing this in the process of upgrading the entire cluster is the
plan just to update one?</DIV><BR>
<BLOCKQUOTE type="cite">
<DIV>
<DIV><BR>So the new node receives the cancel action, but doesn’t call
send_direct_ack. As a result on the old node, it sends the cancel
action:<BR>May 23 18:05:49 vsa-000001be-vc-0 crmd: [3089]: info:
te_rsc_command: Initiating action 4: cancel VAM:1_monitor_5000 on
vsa-000001be-vc-1<BR><BR>And after 3 minutes only it moves forward due to
timeout<BR>May 23 18:08:49 vsa-000001be-vc-0 crmd: [3089]: WARN:
action_timer_callback: Timer popped (timeout=120000, abort_level=1000000,
complete=false)<BR>May 23 18:08:49 vsa-000001be-vc-0 crmd: [3089]: ERROR:
print_elem: Aborting transition, action lost: [Action 4]: In-flight (id:
VAM:1_monitor_5000, loc: vsa-000001be-vc-1, priority: 0)<BR>May 23 18:08:49
vsa-000001be-vc-0 crmd: [3089]: info: abort_transition_graph:
action_timer_callback:512 - Triggered transition abort (complete=0) : Action
lost<BR><BR>However, the 3 minute-timeout is unacceptable for our
customers.<BR><BR>What would you recommend to fix this backward compatibility
issue?<BR></DIV></DIV></BLOCKQUOTE>
<DIV> </DIV>
<DIV>Unfortunately, you might need to resort to the detach+upgrade
everything+reattach method of upgrading as described here:</DIV>
<DIV> </DIV>
<DIV> <A
href="http://clusterlabs.org/doc/en-US/Pacemaker/1.1-plugin/html/Pacemaker_Explained/_disconnect_and_reattach.html">http://clusterlabs.org/doc/en-US/Pacemaker/1.1-plugin/html/Pacemaker_Explained/_disconnect_and_reattach.html</A></DIV><BR>
<BLOCKQUOTE type="cite">
<DIV>
<DIV><BR>Only as a test, I called send_direct_ack in case “in_progress==TRUE”
also. This fixed the problem, as the older node received the needed ACK. But I
don’t know what this change might break.<BR></DIV></DIV></BLOCKQUOTE>
<DIV> </DIV>
<DIV>It would probably be fine as a transition plan.</DIV>
<DIV>Ie. first do a rolling update to the patched version, then another to the
unpatched version.</DIV><BR>
<BLOCKQUOTE type="cite">
<DIV>
<DIV><BR>Thanks,<BR>Alex.<BR><BR><BR><BR><BR><BR></DIV></DIV></BLOCKQUOTE></DIV>
<DIV> </DIV></DIV></DIV></DIV></BODY></HTML>