[ClusterLabs] All IP resources deleted once a fenced node rejoins

[Ken Gaillot]

On 01/15/2016 11:08 AM, Ken Gaillot wrote:

>> Jan 13 19:33:00 [4291] orana        cib:     info:
>> cib_process_replace: Replacement 0.4.0 from kamet not applied to
>> 0.74.1: current epoch is greater than the replacement
>> Jan 13 19:33:00 [4291] orana        cib:  warning:
>> cib_process_request: Completed cib_replace operation for section
>> 'all': Update was older than existing configuration (rc=-205,
>> origin=kamet/cibadmin/2, version=0.74.1)

I misread. Looking at it again, the above means that the old
configuration was indeed rejected for section "all". However:

>> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op:
>> Diff: --- 0.74.1 2
>> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op:
>> Diff: +++ 0.75.0 (null)
>> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
>> /cib/configuration/nodes/node[@id='kamet']
>> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
>> /cib/configuration/nodes/node[@id='orana']
>> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
>> /cib/configuration/resources/primitive[@id='fence-uc-orana']/meta_attributes[@id='fence-uc-orana-meta_attributes']
>> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
>> /cib/configuration/resources/primitive[@id='fence-uc-kamet']
>> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
>> /cib/configuration/resources/primitive[@id='C-3']
>> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
>> /cib/configuration/resources/primitive[@id='C-FLT']
>> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
>> /cib/configuration/resources/primitive[@id='C-FLT2']
>> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
>> /cib/configuration/resources/primitive[@id='E-3']
>> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
>> /cib/configuration/resources/primitive[@id='MGMT-FLT']
>> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
>> /cib/configuration/resources/primitive[@id='M-FLT']
>> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
>> /cib/configuration/resources/primitive[@id='M-FLT2']
>> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
>> /cib/configuration/resources/primitive[@id='S-FLT']
>> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
>> /cib/configuration/resources/primitive[@id='S-FLT2']
>> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
>> /cib/configuration/constraints/rsc_colocation[@id='colocation-C-3-foo-master-INFINITY']
>> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
>> /cib/configuration/constraints/rsc_order[@id='order-C-3-foo-master-mandatory']
>> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
>> /cib/configuration/constraints/rsc_colocation[@id='colocation-C-FLT-foo-master-INFINITY']
>> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
>> /cib/configuration/constraints/rsc_order[@id='order-C-FLT-foo-master-mandatory']
>> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
>> /cib/configuration/constraints/rsc_colocation[@id='colocation-C-FLT2-foo-master-INFINITY']
>> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
>> /cib/configuration/constraints/rsc_order[@id='order-C-FLT2-foo-master-mandatory']
>> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
>> /cib/configuration/constraints/rsc_colocation[@id='colocation-E-3-foo-master-INFINITY']
>> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
>> /cib/configuration/constraints/rsc_order[@id='order-E-3-foo-master-mandatory']
>> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
>> /cib/configuration/constraints/rsc_colocation[@id='colocation-MGMT-FLT-foo-master-INFINITY']
>> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
>> /cib/configuration/constraints/rsc_order[@id='order-MGMT-FLT-foo-master-mandatory']
>> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
>> /cib/configuration/constraints/rsc_colocation[@id='colocation-M-FLT-foo-master-INFINITY']
>> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
>> /cib/configuration/constraints/rsc_order[@id='order-M-FLT-foo-master-mandatory']
>> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
>> /cib/configuration/constraints/rsc_colocation[@id='colocation-M-FLT2-foo-master-INFINITY']
>> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
>> /cib/configuration/constraints/rsc_order[@id='order-M-FLT2-foo-master-mandatory']
>> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
>> /cib/configuration/constraints/rsc_colocation[@id='colocation-S-FLT-foo-master-INFINITY']
>> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
>> /cib/configuration/constraints/rsc_order[@id='order-S-FLT-foo-master-mandatory']
>> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
>> /cib/configuration/constraints/rsc_colocation[@id='colocation-S-FLT2-foo-master-INFINITY']
>> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
>> /cib/configuration/constraints/rsc_order[@id='order-S-FLT2-foo-master-mandatory']
>> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
>> /cib/configuration/constraints/rsc_colocation[@id='colocation-fence-uc-orana-foo-master-INFINITY']
>> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
>> /cib/configuration/constraints/rsc_colocation[@id='colocation-fence-uc-kamet-foo-master-INFINITY']
>> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
>> /cib/configuration/constraints/rsc_order[@id='order-fence-uc-kamet-foo-master-mandatory']
>> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: --
>> /cib/configuration/constraints/rsc_order[@id='order-fence-uc-orana-foo-master-mandatory']
>> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: +
>> /cib:  @epoch=75, @num_updates=0
>> Jan 13 19:33:00 [4291] orana        cib:     info: cib_perform_op: +
>> /cib/configuration/resources/primitive[@id='fence-uc-orana']/instance_attributes[@id='fence-uc-orana-instance_attributes']/nvpair[@id='fence-uc-orana-instance_attributes-delay']:
>>  @value=0
>> Jan 13 19:33:00 [4291] orana        cib:     info:
>> cib_process_request: Completed cib_replace operation for section
>> configuration: OK (rc=0, origin=kamet/cibadmin/2, version=0.75.0)

It did use the (separate) above replacement for the section
"configuration", because the CIB version here (0.75.0) is higher than
the current cluster CIB (0.74.1).

So orana behaved as expected. The question is why did this replacement
for the configuration section get sent out. Logs from kamet at this time
might help.

> The above is the problem. You can see all the resources being deleted
> from the CIB ("--" indicates lines being removed from the CIB, and "+"
> indicates lines being added). For some reason, the cluster used a much
> older CIB on kamet to replace the current one used by the cluster.
> 
> I'm not sure why this happened; it may be a bug.
> 
> What version of pacemaker are you using?
> 
> Check the permissions on /var/lib/pacemaker/cib and the files in it on
> both nodes. I'd expect everything to be owned and writeable by the
> hacluster user.