<div dir="ltr">I am getting rather unexpected behavior when I combine clones, location constraints, and remote nodes in an asymmetric cluster. My cluster is configured to be asymmetric, distinguishing between vmhosts and various sorts of remote nodes. Currently I am running upstream version b6d42ed. I am simplifying my description to avoid confusion, hoping in so doing I don't miss any salient points...<div>
<div><br></div><div>My physical cluster nodes, also the VM hosts, have the attribute "nodetype=vmhost". They also have Infiniband interfaces, which take some time to come up. I don't want my shared file system (which needs IB), or libvirtd (which needs the file system), to come up before IB... So I have this in my configuration:</div>
</div><div><br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><font face="courier new, monospace">primitive p-watch-ib0 ocf:heartbeat:ethmonitor \</font></div></div><div><div><font face="courier new, monospace"> params \</font></div>
</div><div><div><font face="courier new, monospace"> interface="ib0" \</font></div></div><div><div><font face="courier new, monospace"> op monitor timeout="100s" interval="10s"</font></div>
</div><div><div><font face="courier new, monospace">clone c-watch-ib0 p-watch-ib0 \</font></div></div><div><div><font face="courier new, monospace"> meta interleave="true" </font></div></div><div><div><font face="courier new, monospace">#</font></div>
</div><div><div><font face="courier new, monospace">location loc-watch-ib-only-vmhosts c-watch-ib0 \</font></div></div><div><div><font face="courier new, monospace"> rule 0: nodetype eq "vmhost"</font></div></div>
</blockquote><div><br></div><div>Something broke between upstream versions 0a2570a and c68919f -- the c-watch-ib0 clone never starts. I've found that if I run "crm_resource --force-start -r p-watch-ib0" when IB is running, the ethmonitor-ib0 attribute is not set like it used to be. Oh well, I can set it manually. So let's.</div>
<div><br></div><div>We use GPFS for a shared file system, so I have an agent to start it and wait for a file system to mount. It should only run on VM hosts, and only when IB is running. So I have this:</div><div><br></div>
<blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><font face="courier new, monospace">primitive p-fs-gpfs ocf:ccni:gpfs \</font></div></div><div><div><font face="courier new, monospace"> params \</font></div>
</div><div><div><font face="courier new, monospace"> fspath="/gpfs/lb/utility" \</font></div></div><div><div><font face="courier new, monospace"> op monitor timeout="20s" interval="30s" \</font></div>
</div><div><div><font face="courier new, monospace"> op start timeout="180s" \</font></div></div><div><div><font face="courier new, monospace"> op stop timeout="120s"</font></div></div><div><div>
<font face="courier new, monospace">clone c-fs-gpfs p-fs-gpfs \</font></div></div><div><div><font face="courier new, monospace"> meta interleave="true" </font></div></div><div><div><font face="courier new, monospace">location loc-fs-gpfs-needs-ib0 c-fs-gpfs \</font></div>
</div><div><div><font face="courier new, monospace"> rule -inf: not_defined "ethmonitor-ib0" or "ethmonitor-ib0" eq 0</font></div></div><div><div><font face="courier new, monospace">location loc-fs-gpfs-on-vmhosts c-fs-gpfs \</font></div>
</div><div><div><font face="courier new, monospace"> rule 0: nodetype eq "vmhost"</font></div></div></blockquote><div><br></div><div>That all used to start nicely. Now even if I set the ethmonitor-ib0 attribute, it doesn't. However, I can use "crm_resource --force-start -r p-fs-gpfs" on each of my VM hosts, then issue "crm resource cleanup c-fs-gpfs", and all is well. I can use "crm status" to see something like:</div>
<div><br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><font face="courier new, monospace">Last updated: Tue Oct 22 16:35:43 2013</font></div></div><div><div><font face="courier new, monospace">Last change: Tue Oct 22 15:50:52 2013 via crmd on cvmh01</font></div>
</div><div><div><font face="courier new, monospace">Stack: cman</font></div></div><div><div><font face="courier new, monospace">Current DC: cvmh04 - partition with quorum</font></div></div><div><div><font face="courier new, monospace">Version: 1.1.10-19.el6.ccni-b6d42ed</font></div>
</div><div><div><font face="courier new, monospace">8 Nodes configured</font></div></div><div><div><font face="courier new, monospace">92 Resources configured</font></div></div><div><div><font face="courier new, monospace"><br>
</font></div></div><div><div><font face="courier new, monospace"><br></font></div></div><div><div><font face="courier new, monospace">Online: [ cvmh01 cvmh02 cvmh03 cvmh04 ]</font></div></div><div><div><font face="courier new, monospace"><br>
</font></div></div><div><div><font face="courier new, monospace"> fence-cvmh01 (stonith:fence_ipmilan): Started cvmh04 </font></div></div><div><div><font face="courier new, monospace"> fence-cvmh02 (stonith:fence_ipmilan): Started cvmh01 </font></div>
</div><div><div><font face="courier new, monospace"> fence-cvmh03 (stonith:fence_ipmilan): Started cvmh01 </font></div></div><div><div><font face="courier new, monospace"> fence-cvmh04 (stonith:fence_ipmilan): Started cvmh01 </font></div>
</div><div><div><font face="courier new, monospace">Clone Set: c-fs-gpfs [p-fs-gpfs]</font></div></div><div><div><font face="courier new, monospace"> Started: [ cvmh01 cvmh02 cvmh03 cvmh04 ]</font></div></div></blockquote>
<div><div> </div></div><div>which is what I would expect (other than I expect pacemaker to have started these for me, like it used to).</div><div><br></div><div>Now I also have clone resources to NFS-mount another file system, and actually do a bind mount out of the GPFS file system, which behave like the GPFS resource -- they used to just work, now I need to use "crm_resource --force-start" and clean up. That finally lets me start libvirtd, using this configuration:</div>
<div><br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><font face="courier new, monospace">primitive p-libvirtd lsb:libvirtd \</font></div></div><div><div><font face="courier new, monospace"> op monitor interval="30s"</font></div>
</div><div><div><font face="courier new, monospace">clone c-p-libvirtd p-libvirtd \</font></div></div><div><div><font face="courier new, monospace"> meta interleave="true" </font></div></div><div><div><font face="courier new, monospace">order o-libvirtd-after-storage inf: \</font></div>
</div><div><div><font face="courier new, monospace"> ( c-fs-libvirt-VM-xcm c-fs-bind-libvirt-VM-cvmh ) \</font></div></div><div><div><font face="courier new, monospace"> c-p-libvirtd</font></div></div><div><div><font face="courier new, monospace">location loc-libvirtd-on-vmhosts c-p-libvirtd \</font></div>
</div><div><div><font face="courier new, monospace"> rule 0: nodetype eq "vmhost"</font></div></div></blockquote><div><br></div><div>Of course that used to just work, but now, like the other clones, I need to force-start libvirtd on the VM hosts, and clean up. Once I do that, all my VM resources, which are not clones, just start up like they are supposed to! Several of these are configured as remote nodes, and they have services configured to run in them. But now other strange things happen:</div>
<div><br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><font face="courier new, monospace">Last updated: Tue Oct 22 16:46:29 2013</font></div></div><div><div><font face="courier new, monospace">Last change: Tue Oct 22 15:50:52 2013 via crmd on cvmh01</font></div>
</div><div><div><font face="courier new, monospace">Stack: cman</font></div></div><div><div><font face="courier new, monospace">Current DC: cvmh04 - partition with quorum</font></div></div><div><div><font face="courier new, monospace">Version: 1.1.10-19.el6.ccni-b6d42ed</font></div>
</div><div><div><font face="courier new, monospace">8 Nodes configured</font></div></div><div><div><font face="courier new, monospace">92 Resources configured</font></div></div><div><div><font face="courier new, monospace"><br>
</font></div></div><div><div><font face="courier new, monospace"><br></font></div></div><div><div><font face="courier new, monospace">ContainerNode slurmdb02:vm-slurmdb02: UNCLEAN (offline)</font></div></div><div><div><font face="courier new, monospace">Online: [ cvmh01 cvmh02 cvmh03 cvmh04 ]</font></div>
</div><div><div><font face="courier new, monospace">Containers: [ db02:vm-db02 ldap01:vm-ldap01 ldap02:vm-ldap02 ]</font></div></div><div><div><font face="courier new, monospace"><br></font></div></div><div><div><font face="courier new, monospace"> fence-cvmh01 (stonith:fence_ipmilan): Started cvmh04 </font></div>
</div><div><div><font face="courier new, monospace"> fence-cvmh02 (stonith:fence_ipmilan): Started cvmh01 </font></div></div><div><div><font face="courier new, monospace"> fence-cvmh03 (stonith:fence_ipmilan): Started cvmh01 </font></div>
</div><div><div><font face="courier new, monospace"> fence-cvmh04 (stonith:fence_ipmilan): Started cvmh01 </font></div></div><div><div><font face="courier new, monospace"> Clone Set: c-p-libvirtd [p-libvirtd]</font></div>
</div><div><div><font face="courier new, monospace"> p-libvirtd (lsb:libvirtd): FAILED slurmdb02 </font></div></div><div><div><font face="courier new, monospace"> Started: [ cvmh01 cvmh02 cvmh03 cvmh04 ]</font></div>
</div><div><div><font face="courier new, monospace"> Stopped: [ db02 ldap01 ldap02 ]</font></div></div><div><div><font face="courier new, monospace"> Clone Set: c-watch-ib0 [p-watch-ib0]</font></div></div><div><div><font face="courier new, monospace"> p-watch-ib0 (ocf::heartbeat:ethmonitor): FAILED slurmdb02 </font></div>
</div><div><div><font face="courier new, monospace"> Started: [ cvmh01 cvmh02 cvmh03 cvmh04 ]</font></div></div><div><div><font face="courier new, monospace"> Stopped: [ db02 ldap01 ldap02 ]</font></div></div><div>
<div><font face="courier new, monospace"> Clone Set: c-fs-gpfs [p-fs-gpfs]</font></div></div><div><div><font face="courier new, monospace"> p-fs-gpfs (ocf::ccni:gpfs): FAILED slurmdb02 </font></div></div><div><div>
<font face="courier new, monospace"> Started: [ cvmh01 cvmh02 cvmh03 cvmh04 ]</font></div></div><div><div><font face="courier new, monospace"> Stopped: [ db02 ldap01 ldap02 ]</font></div></div><div><div><font face="courier new, monospace"> vm-compute-test (ocf::ccni:xcatVirtualDomain): FAILED [ cvmh04 slurmdb0</font></div>
</div><div><div><font face="courier new, monospace">2 ]</font></div></div><div><div><font face="courier new, monospace"> vm-swbuildsl6 (ocf::ccni:xcatVirtualDomain): FAILED slurmdb02 </font></div></div><div><div><font face="courier new, monospace"> vm-db02 (ocf::ccni:xcatVirtualDomain): Started cvmh01 </font></div>
</div><div><div><font face="courier new, monospace"> vm-ldap01 (ocf::ccni:xcatVirtualDomain): Started cvmh02 </font></div></div><div><div><font face="courier new, monospace"> vm-ldap02 (ocf::ccni:xcatVirtualDomain): Started cvmh03 </font></div>
</div><div><div><font face="courier new, monospace"> p-postgres (ocf::heartbeat:pgsql): FAILED [ db02 slurmdb02 ]</font></div></div><div><div><font face="courier new, monospace"> p-mysql (ocf::heartbeat:mysql): FAILED [ db02 slurmdb02 ]</font></div>
</div><div><div><font face="courier new, monospace"> Clone Set: c-fs-share-config-data [fs-share-config-data]</font></div></div><div><div><font face="courier new, monospace"> fs-share-config-data (ocf::heartbeat:Filesystem): FAILED slurmdb02</font></div>
</div><div><div><font face="courier new, monospace"> </font></div></div><div><div><font face="courier new, monospace"> Stopped: [ cvmh01 cvmh02 cvmh03 cvmh04 db02 ldap01 ldap02 ]</font></div></div><div><div><font face="courier new, monospace"> p-mysql-slurm (ocf::heartbeat:mysql): FAILED slurmdb02 </font></div>
</div><div><div><font face="courier new, monospace"> p-slurmdbd (ocf::ccni:SlurmDBD): FAILED slurmdb02 </font></div></div><div><div><font face="courier new, monospace"> Clone Set: c-ldapagent [s-ldapagent]</font></div>
</div><div><div><font face="courier new, monospace"> s-ldapagent (ocf::ccni:WrapInitScript): FAILED slurmdb02 </font></div></div><div><div><font face="courier new, monospace"> Stopped: [ cvmh01 cvmh02 cvmh03 cvmh04 db02 ldap01 ldap02 ]</font></div>
</div><div><div><font face="courier new, monospace"> Clone Set: c-ldap [s-ldap]</font></div></div><div><div><font face="courier new, monospace"> s-ldap (ocf::ccni:WrapInitScript): FAILED slurmdb02 </font></div>
</div><div><div><font face="courier new, monospace"> Started: [ ldap01 ldap02 ]</font></div></div><div><div><font face="courier new, monospace"> Stopped: [ cvmh01 cvmh02 cvmh03 cvmh04 db02 ]</font></div></div></blockquote>
<div><br></div><div>Now this is unexpected for a couple of reasons. I do have constraints like:</div><div><br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px"><div><div><font face="courier new, monospace">location loc-vm-swbuildsl6 vm-swbuildsl6 \</font></div>
</div><div><div><font face="courier new, monospace"> rule $id="loc-vm-swbuildsl6-rule" 0: nodetype eq vmhost</font></div></div><div><div><font face="courier new, monospace">order o-vm-swbuildsl6 inf: c-p-libvirtd vm-swbuildsl6</font></div>
</div></blockquote><div><br></div><div>And it is not the case that slurmdb02 has the vmhost attribute set; using "crm_mon -o -1 -N -A" we see:</div><div><br></div><blockquote style="margin:0px 0px 0px 40px;border:none;padding:0px">
<div><div><font face="courier new, monospace">Node Attributes:</font></div></div><div><div><font face="courier new, monospace">* Node cvmh01:</font></div></div><div><div><font face="courier new, monospace"> + ethmonitor-ib0 : 1 </font></div>
</div><div><div><font face="courier new, monospace"> + nodetype : vmhost </font></div></div><div><div><font face="courier new, monospace">* Node cvmh02:</font></div></div><div><div><font face="courier new, monospace"> + ethmonitor-ib0 : 1 </font></div>
</div><div><div><font face="courier new, monospace"> + nodetype : vmhost </font></div></div><div><div><font face="courier new, monospace">* Node cvmh03:</font></div></div><div><div><font face="courier new, monospace"> + ethmonitor-ib0 : 1 </font></div>
</div><div><div><font face="courier new, monospace"> + nodetype : vmhost </font></div></div><div><div><font face="courier new, monospace">* Node cvmh04:</font></div></div><div><div><font face="courier new, monospace"> + ethmonitor-ib0 : 1 </font></div>
</div><div><div><font face="courier new, monospace"> + nodetype : vmhost </font></div></div><div><div><font face="courier new, monospace">* Node db02:</font></div></div><div><div><font face="courier new, monospace">* Node ldap01:</font></div>
</div><div><div><font face="courier new, monospace">* Node ldap02:</font></div></div><div><div><font face="courier new, monospace">* Node slurmdb02:</font></div></div></blockquote><div><div><br></div></div><div>The results are unexpected to me also because I (perhaps naively) wouldn't expect it to show me the new nodes on the "stopped" lines -- I kind of expected a location rule to limit where clones would even be attempted. For example, with the rule limiting c-p-libvirtd to the vmhosts, I don't really expect to be told that the clones are stopped on the remote VM nodes db02, ldap01, and ldap02 (let alone be started on slurmdb02!).</div>
<div><br></div><div>Until I wrote this note, even the cloned ldap resource c-ldap needed to be started using force-start. Not sure why this time it started on its own... Perhaps this stack trace in the core dump pacemaker left on one of the VM hosts has a clue?</div>
<div><br></div><div><br></div><blockquote style="margin:0 0 0 40px;border:none;padding:0px"><div><div><font face="courier new, monospace">#0 0x00007f121e9ac8e5 in raise () from /lib64/libc.so.6</font></div></div><div><div>
<font face="courier new, monospace">#1 0x00007f121e9ae0c5 in abort () from /lib64/libc.so.6</font></div></div><div><div><font face="courier new, monospace">#2 0x00007f121e9ea7f7 in __libc_message () from /lib64/libc.so.6</font></div>
</div><div><div><font face="courier new, monospace">#3 0x00007f121e9f0126 in malloc_printerr () from /lib64/libc.so.6</font></div></div><div><div><font face="courier new, monospace">#4 0x00007f121e9f05ad in malloc_consolidate () from /lib64/libc.so.6</font></div>
</div><div><div><font face="courier new, monospace">#5 0x00007f121e9f33c5 in _int_malloc () from /lib64/libc.so.6</font></div></div><div><div><font face="courier new, monospace">#6 0x00007f121e9f45e6 in calloc () from /lib64/libc.so.6</font></div>
</div><div><div><font face="courier new, monospace">#7 0x00007f121e9e91ed in open_memstream () from /lib64/libc.so.6</font></div></div><div><div><font face="courier new, monospace">#8 0x00007f121ea5ebdb in __vsyslog_chk () from /lib64/libc.so.6</font></div>
</div><div><div><font face="courier new, monospace">#9 0x00007f121ea5f1b3 in __syslog_chk () from /lib64/libc.so.6</font></div></div><div><div><font face="courier new, monospace">#10 0x00007f121e72b9fb in ?? () from /usr/lib64/libqb.so.0</font></div>
</div><div><div><font face="courier new, monospace">#11 0x00007f121e72a6a2 in qb_log_real_va_ () from /usr/lib64/libqb.so.0</font></div></div><div><div><font face="courier new, monospace">#12 0x00007f121e72a91d in qb_log_real_ () from /usr/lib64/libqb.so.0</font></div>
</div><div><div><font face="courier new, monospace">#13 0x000000000042e994 in te_rsc_command (graph=0x20c7b40, action=0x23b0c90)</font></div></div><div><div><font face="courier new, monospace"> at te_actions.c:412</font></div>
</div><div><div><font face="courier new, monospace">#14 0x0000003a64404019 in initiate_action (graph=0x20c7b40) at graph.c:172</font></div></div><div><div><font face="courier new, monospace">#15 fire_synapse (graph=0x20c7b40) at graph.c:211</font></div>
</div><div><div><font face="courier new, monospace">#16 run_graph (graph=0x20c7b40) at graph.c:366</font></div></div><div><div><font face="courier new, monospace">#17 0x000000000042f8cd in te_graph_trigger (user_data=<value optimized out>)</font></div>
</div><div><div><font face="courier new, monospace"> at te_utils.c:331</font></div></div><div><div><font face="courier new, monospace">#18 0x0000003a6202b283 in crm_trigger_dispatch (source=<value optimized out>, </font></div>
</div><div><div><font face="courier new, monospace"> callback=<value optimized out>, userdata=<value optimized out>)</font></div></div><div><div><font face="courier new, monospace"> at mainloop.c:105</font></div>
</div><div><div><font face="courier new, monospace">#19 0x00000038b3c38f0e in g_main_context_dispatch ()</font></div></div><div><div><font face="courier new, monospace"> from /lib64/libglib-2.0.so.0</font></div></div><div>
<div><font face="courier new, monospace">#20 0x00000038b3c3c938 in ?? () from /lib64/libglib-2.0.so.0</font></div></div><div><div><font face="courier new, monospace">#21 0x00000038b3c3cd55 in g_main_loop_run () from /lib64/libglib-2.0.so.0</font></div>
</div><div><div><font face="courier new, monospace">#22 0x00000000004058ee in crmd_init () at main.c:154</font></div></div><div><div><font face="courier new, monospace">#23 0x0000000000405c2c in main (argc=1, argv=0x7fffdc207528) at main.c:121</font></div>
</div></blockquote><div><br></div><div><br></div><div>Not sure how to take this further. It has been difficult to characterize what exactly is or isn't happening, and hopefully I've not left out some critical detail. Thanks.</div>
<div><br></div><div>/Lindsay</div><div><br></div><div><br></div><div><br></div></div>