[Pacemaker] Order of resources in a group and crm_diff

Gao,Yan ygao at suse.com
Fri Jun 6 05:21:29 UTC 2014


On 01/29/14 13:44, Andrew Beekhof wrote:
> 
> On 28 Jan 2014, at 10:11 pm, Vladislav Bogdanov <bubble at hoster-ok.com> wrote:
> 
>> Hi all,
>>
>> Just discovered, that when I add resource to a middle of
>> (running) group, it is added to the end.
>>
>> I mean, if I update following (crmsh syntax)
>>
>> group dhcp-server vip-10-5-200-244 dhcpd
>>
>> with
>>
>> group dhcp-server vip-10-5-200-244 vip-10-5-201-244 dhcpd
>>
>> with 'crm configure load update', actual definition becomes
>>
>> group dhcp-server vip-10-5-200-244 dhcpd vip-10-5-201-244
>>
>> Also, strange enough, if I get XML CIB with cibadmin -Q, then edit
>> order of primitives with text editor, crm_diff doesn't show any differences:
>>
>> cib-orig.xml:
>> ...
>>      <group id="dhcp-server">
>>        <primitive id="vip-10-5-200-244" class="ocf" provider="heartbeat" type="IPaddr2">
>>          <instance_attributes id="vip-10-5-200-244-instance_attributes">
>>            <nvpair name="ip" value="10.5.200.244" id="vip-10-5-200-244-instance_attributes-ip"/>
>>            <nvpair name="cidr_netmask" value="32" id="vip-10-5-200-244-instance_attributes-cidr_netmask"/>
>>            <nvpair name="nic" value="vlan1" id="vip-10-5-200-244-instance_attributes-nic"/>
>>          </instance_attributes>
>>          <operations>
>>            <op name="start" interval="0" timeout="20" id="vip-10-5-200-244-start-0"/>
>>            <op name="stop" interval="0" timeout="20" id="vip-10-5-200-244-stop-0"/>
>>            <op name="monitor" interval="30" id="vip-10-5-200-244-monitor-30"/>
>>          </operations>
>>        </primitive>
>>        <primitive id="dhcpd" class="lsb" type="dhcpd">
>>          <operations>
>>            <op name="monitor" interval="10" timeout="15" id="dhcpd-monitor-10"/>
>>            <op name="start" interval="0" timeout="90" id="dhcpd-start-0"/>
>>            <op name="stop" interval="0" timeout="90" id="dhcpd-stop-0"/>
>>          </operations>
>>          <meta_attributes id="dhcpd-meta_attributes">
>>            <nvpair id="dhcpd-meta_attributes-target-role" name="target-role" value="Started"/>
>>          </meta_attributes>
>>        </primitive>
>>        <primitive id="vip-10-5-201-244" class="ocf" provider="heartbeat" type="IPaddr2">
>>          <instance_attributes id="vip-10-5-201-244-instance_attributes">
>>            <nvpair name="ip" value="10.5.201.244" id="vip-10-5-201-244-instance_attributes-ip"/>
>>            <nvpair name="cidr_netmask" value="24" id="vip-10-5-201-244-instance_attributes-cidr_netmask"/>
>>            <nvpair name="nic" value="vlan201" id="vip-10-5-201-244-instance_attributes-nic"/>
>>          </instance_attributes>
>>          <operations>
>>            <op name="start" interval="0" timeout="20" id="vip-10-5-201-244-start-0"/>
>>            <op name="stop" interval="0" timeout="20" id="vip-10-5-201-244-stop-0"/>
>>            <op name="monitor" interval="30" id="vip-10-5-201-244-monitor-30"/>
>>          </operations>
>>        </primitive>
>>      </group>
>> ...
>>
>> cib.xml:
>> ...
>>     <group id="dhcp-server">
>>        <primitive id="vip-10-5-200-244" class="ocf" provider="heartbeat" type="IPaddr2">
>>          <instance_attributes id="vip-10-5-200-244-instance_attributes">
>>            <nvpair name="ip" value="10.5.200.244" id="vip-10-5-200-244-instance_attributes-ip"/>
>>            <nvpair name="cidr_netmask" value="32" id="vip-10-5-200-244-instance_attributes-cidr_netmask"/>
>>            <nvpair name="nic" value="vlan1" id="vip-10-5-200-244-instance_attributes-nic"/>
>>          </instance_attributes>
>>          <operations>
>>            <op name="start" interval="0" timeout="20" id="vip-10-5-200-244-start-0"/>
>>            <op name="stop" interval="0" timeout="20" id="vip-10-5-200-244-stop-0"/>
>>            <op name="monitor" interval="30" id="vip-10-5-200-244-monitor-30"/>
>>          </operations>
>>        </primitive>
>>        <primitive id="vip-10-5-201-244" class="ocf" provider="heartbeat" type="IPaddr2">
>>          <instance_attributes id="vip-10-5-201-244-instance_attributes">
>>            <nvpair name="ip" value="10.5.201.244" id="vip-10-5-201-244-instance_attributes-ip"/>
>>            <nvpair name="cidr_netmask" value="24" id="vip-10-5-201-244-instance_attributes-cidr_netmask"/>
>>            <nvpair name="nic" value="vlan201" id="vip-10-5-201-244-instance_attributes-nic"/>
>>          </instance_attributes>
>>          <operations>
>>            <op name="start" interval="0" timeout="20" id="vip-10-5-201-244-start-0"/>
>>            <op name="stop" interval="0" timeout="20" id="vip-10-5-201-244-stop-0"/>
>>            <op name="monitor" interval="30" id="vip-10-5-201-244-monitor-30"/>
>>          </operations>
>>        </primitive>
>>        <primitive id="dhcpd" class="lsb" type="dhcpd">
>>          <operations>
>>            <op name="monitor" interval="10" timeout="15" id="dhcpd-monitor-10"/>
>>            <op name="start" interval="0" timeout="90" id="dhcpd-start-0"/>
>>            <op name="stop" interval="0" timeout="90" id="dhcpd-stop-0"/>
>>          </operations>
>>          <meta_attributes id="dhcpd-meta_attributes">
>>            <nvpair id="dhcpd-meta_attributes-target-role" name="target-role" value="Started"/>
>>          </meta_attributes>
>>        </primitive>
>>      </group>
>> ...
>>
>> # crm_diff --original cib-orig.xml --new cib.xml
>>
>> shows nothing.
>>
>> And, 'cibadmin --replace --xml-file cib.xml' does nothing:
>>
>> Jan 28 11:01:21 booter-0 cib[2693]:   notice: cib:diff: Diff: --- 0.427.2
>> Jan 28 11:01:21 booter-0 cib[2693]:   notice: cib:diff: Diff: +++ 0.427.19 df366a02885285cc95529f402bfdac12
>> Jan 28 11:01:21 booter-0 cib[2693]:   notice: cib:diff: --           <nvpair id="status-2-shutdown" name="shutdown" value="0"/>
>> Jan 28 11:01:21 booter-0 cib[2693]:   notice: cib:diff: ++ <cib epoch="427" num_updates="19" admin_epoch="0" validate-with="pacemaker-1.2" cib-last-written="Tue Jan 28 10:46:06 2014" update-origin="booter-0" update-client="cibadmin" crm_feature_set="3.0.8" have-quorum="1" dc-uuid="1"/>
> 
> Thats a known deficiency in the v1 diff format (and why we need costly digests to detect ordering changes).
> Happily .12 will have a new and improve diff format that will handle this correctly.
> 
>>
>> But, after I do
>>
>> # crm_shadow --create-empty myShadow
>> shadow[myShadow] # cibadmin -E --force
>> shadow[myShadow] # cibadmin --replace --xml-file cib.xml
>> shadow[myShadow] # crm_shadow --commit myShadow --force
>> Now type Ctrl-D to exit the crm_shadow shell
>> shadow[myShadow] # exit
>>
>> group becomes defined in a proper order.
>>
>> That's why the only suspect is xml-diff algorithm.
>>
>> Andrew, David, could you please look?
> 
> Its also partly how crmsh is using diffs.
> It could be verifying the diff produces the correct result by verifying the above mentioned digest.
> Or it could do a replace for the group instead...
I'm a bit surprised that even a replace cannot successfully reorder
resources in a group. I tried it on 1.1.9 ~ 1.1.11.

On DC:
Jun  6 12:18:51 sles11-1 cib[1814]:   notice: cib_perform_op:
Configuration ordering change detected
Jun  6 12:18:51 sles11-1 cib[1814]:   notice: cib:diff: Diff: --- 0.3835.86
Jun  6 12:18:51 sles11-1 cib[1814]:   notice: cib:diff: Diff: +++
0.3835.1 21300207d1fe995ea0475be3dc60718f


On non-DC:
Jun  6 12:16:50 sles11-2 cib[32053]:  warning: cib_process_diff: Diff
0.3835.81 -> 0.3835.1 from sles11-1 not applied to 0.3835.81: Failed
application of an update diff
Jun  6 12:16:50 sles11-2 cib[32053]:  warning: cib_process_replace:
Replacement 0.3835.1 from sles11-1 not applied to 0.3835.81: current
num_updates is greater than the replacement


I think the crm_shadow way mentioned above works because it bumps
"epoch" itself.


If we replace only the snippet of the group with
cibadmin -R -o resources -x group.xml

, it'll apply the change in DC's cib, while it'll leave the non-DC's cib
out of sync.


1.1.12-rc goes a different way in cib_perform_cib() and works.

So, for 1.1.10/1.1.11, is it supposed to be like:

--- pacemaker.orig/lib/cib/cib_utils.c
+++ pacemaker/lib/cib/cib_utils.c
@@ -565,6 +565,7 @@ cib_perform_op(const char *op, int call_
         } else if (crm_str_eq(new_digest, last_digest, TRUE) == FALSE) {

             crm_notice("Configuration ordering change detected");
+            cib_update_counter(scratch, XML_ATTR_GENERATION, FALSE);
             cib_update_counter(scratch, XML_ATTR_NUMUPDATES, TRUE);

             crm_trace("Old: %s, New: %s", last_digest, new_digest);

?

Regards,
  Yan

> 
>>
>> Thank you,
>> Vladislav
>>
>> _______________________________________________
>> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
>> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
>>
>> Project Home: http://www.clusterlabs.org
>> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
>> Bugs: http://bugs.clusterlabs.org
> 
> 
> 
> _______________________________________________
> Pacemaker mailing list: Pacemaker at oss.clusterlabs.org
> http://oss.clusterlabs.org/mailman/listinfo/pacemaker
> 
> Project Home: http://www.clusterlabs.org
> Getting started: http://www.clusterlabs.org/doc/Cluster_from_Scratch.pdf
> Bugs: http://bugs.clusterlabs.org
> 

-- 
Gao,Yan <ygao at suse.com>
Software Engineer
China Server Team, SUSE.




More information about the Pacemaker mailing list