[ClusterLabs] Centos 70->71 update fails with "Application of an update diff failed (rc=-206)"

[Patrick Zwahlen]

Hi,

I'm running a CentOS 7.0 2-nodes cluster providing iSCSI/SAN features. In
order to upgrade to CentOS 7.1, I'm testing the whole process in VMs and it
fails. I've now stripped my config down to a pair of DRBD MS with IPADDR2
(cluster.cfg) attached.

>From a running cluster, here are the steps (I'm upgrading node san2):
- put node san2 in standby
- stop/disable pacemaker on san2
- stop/disable corosync on san2
- update san2 to CentOS 7.1 (pacemaker 1.1.10-32.el7_0.1 ->
1.1.12-22.el7_1.1)
- reboot san2
- enable/start corosync on san2. It looks good, rings are fine in
"corosync-cfgtool-s")
- enable/start pacemaker on san2

I can see the following in the logs:

/var/log/messages (attached, line #57)
=================
Apr 24 16:18:26 san2 crmd[11759]: notice: erase_xpath_callback: Deletion of
"//node_state[@uname='san2.local']/transient_attributes": Application of an
update diff failed (rc=-206)

/var/log/pacemaker.log (attached, starting from line #292)
======================
Apr 24 16:18:26 [11754] san2.local        cib:     info: xml_apply_patchset:
v1 digest mis-match: expected 428c0eb4cd80a4c1ee19b627f6876abd, calculated
ffb5456991bd4ed9e5a7774f49e8259d
Apr 24 16:18:26 [11754] san2.local        cib:     info: __xml_diff_object:
Moved node_state at id (0 -> 6)
Apr 24 16:18:26 [11754] san2.local        cib:     info: __xml_diff_object:
Moved node_state at uname (1 -> 0)
Apr 24 16:18:26 [11759] san2.local       crmd:   notice:
erase_xpath_callback: 	Deletion of
"//node_state[@uname='san2.local']/transient_attributes": Application of an
update diff failed (rc=-206)
Apr 24 16:18:26 [11754] san2.local        cib:     info: send_sync_request:
Requesting re-sync from peer
Apr 24 16:18:26 [11754] san2.local        cib:   notice:
cib_server_process_diff: 	Not applying diff 0.0.14 -> 0.46.15 (sync in
progress)
Apr 24 16:18:26 [11754] san2.local        cib:   notice:
cib_server_process_diff: 	Not applying diff 0.0.15 -> 0.46.16 (sync in
progress)
Apr 24 16:18:26 [11754] san2.local        cib:   notice:
cib_server_process_diff: 	Not applying diff 0.0.16 -> 0.46.17 (sync in
progress)

Google doesn't help me in figuring out what might be wrong.

Config was generated with crmsh-2.1-1.4 is that can have an impact.

Any hint would be highly appreciated.

Cheers, Patrick

NOTE: I have kernel modules (scst/zfs) that require reboots when upgrading,
so I cannot upgrade both nodes while in unmanaged state. I really need to
upgrade one node after the other.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: messages
Type: application/octet-stream
Size: 8963 bytes
Desc: not available
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20150424/24e77a8e/attachment-0006.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: cluster.cfg
Type: application/octet-stream
Size: 1954 bytes
Desc: not available
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20150424/24e77a8e/attachment-0007.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: pacemaker.log
Type: application/octet-stream
Size: 67127 bytes
Desc: not available
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20150424/24e77a8e/attachment-0008.obj>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: smime.p7s
Type: application/pkcs7-signature
Size: 5506 bytes
Desc: not available
URL: <http://lists.clusterlabs.org/pipermail/users/attachments/20150424/24e77a8e/attachment-0002.p7s>